COMP 3150Operating Systems

Unit 12 · Self-check

Unit 12 Self-Check Quiz

Twenty-four questions covering the CIA triad, authentication, authorization, DAC/MAC/RBAC, Linux security, privilege escalation, sandboxing, and hardening. Each question can be expanded to reveal a model answer.

24 questions Self-paced Ungraded

Questions

Multiple choice and short answer

Answer each question before expanding. These span all seven lessons of Unit 12.

Question 1

What are the three goals of the CIA triad? Give a concrete threat to each.

Show answer
Confidentiality: data is not disclosed to unauthorised parties. Threat: an attacker reads a file they should not have access to (via a permission bypass, a memory disclosure bug, or a side channel). Integrity: data is not modified without authorisation. Threat: an attacker modifies a system configuration file, or a program modifies its own executable to persist a backdoor. Availability: data and services are accessible when needed. Threat: a denial-of-service attack that exhausts CPU, memory, disk, or network resources, or a ransomware attack that encrypts the data and holds the key. The three goals often conflict: strong confidentiality (heavy encryption, strict access checks) may reduce availability (slower access, more failures), and strong integrity (checksums, verifications) may reduce performance. Security design is the art of balancing them against the threat model.

Question 2

What is the principle of least privilege, and how does it relate to the reference monitor concept?

Show answer
The principle of least privilege states that every process, user, and system component should have only the privileges it needs to perform its function, and no more. The reference monitor is the abstract concept of an access-control mechanism that (a) mediates every access to every object, (b) is tamper-proof, and (c) is small enough to be verified. The principle of least privilege supports the reference monitor by ensuring that even if a component is compromised, the damage it can do is limited by its privilege set. A process that runs with only the privileges it needs cannot, if compromised, use privileges it does not have. Modern mechanisms — capabilities, sandboxing, RBAC, MAC — all implement the principle in different ways.

Question 3

What is the difference between authentication and authorization? Give an example of each in Linux.

Show answer
Authentication answers "who are you?" — it verifies a claimed identity. In Linux, authentication happens at login: the user's password is checked against the hash in /etc/shadow (via PAM). Authorization answers "what are you allowed to do?" — it determines the actions the authenticated identity may perform. In Linux, authorization is enforced by file permissions (mode bits and ACLs), by the sudoers policy, and by MAC systems (SELinux, AppArmor). A user can authenticate successfully and still be denied authorization to read a file, run a command, or access a network port.

Question 4

Why is the password hash stored in /etc/shadow rather than /etc/passwd?

Show answer
Because /etc/passwd must be world-readable: many programs (such as ls when it prints a user name for a file owner) need to look up the UID-to-name mapping. If the password hashes were in /etc/passwd, any user could read them and mount an offline dictionary attack. Moving the hashes to /etc/shadow, which is readable only by root, prevents unprivileged users from reading them. The shadow file also stores additional password-policy fields (minimum age, maximum age, warning period, expiry) that are not needed by ordinary programs. On a modern system, hashes use a strong algorithm with a salt (SHA-512 with $6$); the salt ensures that two users with the same password have different hashes, and that precomputed rainbow tables cannot be used.

Question 5

Compare DAC, MAC, and RBAC. Give an example of each in Linux.

Show answer
DAC (discretionary access control): the owner of a resource decides who may access it. Linux example: file mode bits and POSIX ACLs. The owner can chmod the file to grant or deny access to anyone. MAC (mandatory access control): the system enforces a global policy that even the owner cannot override. Linux examples: SELinux and AppArmor. A process labelled httpd_t cannot read a file labelled shadow_t even if the file's owner grants permission. RBAC (role-based access control): permissions are assigned to roles, and users are assigned to roles. Linux example: sudoers. A user in the webadmin group may be allowed to restart nginx but not to modify user accounts. RBAC is orthogonal to DAC and MAC: a system can enforce all three simultaneously.

Question 6

What are POSIX ACLs, and when would you use them instead of mode bits?

Show answer
A POSIX ACL (access control list) extends the classic mode bits by allowing permissions to be granted to arbitrary users and groups, not just to the owner, one group, and everyone else. An ACL entry is a triple (type, qualifier, permissions), where type is user, group, mask, or other, and qualifier identifies the specific user or group. You would use an ACL when you need to grant a specific user access to a file without creating a group for them, or when you need to grant different permissions to several groups on the same file. For example, a project directory might grant read-write to the dev group, read-only to the qa group, and no access to everyone else — three groups, which the classical mode bits cannot express. getfacl and setfacl manage ACLs; ls -l shows a + after the mode bits when an ACL is present.

Question 7

What is a setuid binary, and why is it dangerous?

Show answer
A setuid binary is an executable whose setuid bit is set and whose owner is a user other than the caller (typically root). When the binary is executed, the process runs with the owner's effective UID rather than the caller's. This allows an unprivileged user to perform a privileged operation — for example, passwd is setuid root so that a user can change their own password, which requires writing to /etc/shadow. The danger is that any bug in the setuid binary — a buffer overflow, a command injection, a race condition, a failure to sanitise the environment or the input — can be exploited by the caller to gain the binary's privileges. A setuid root program that invokes system() or execlp() with caller-controlled input gives the caller a root shell. Modern systems mitigate this with capabilities: instead of setuid root, a binary can be granted only the specific capability it needs (for example, CAP_NET_BIND_SERVICE to bind a low port).

Question 8

What is the reference monitor, and what three properties must it have?

Show answer
The reference monitor is the abstract concept of an access-control mechanism that mediates every access to every object in the system. It must have three properties:
  1. Complete mediation: every access to every object is checked. No access can bypass the monitor.
  2. Tamper-proof: the monitor itself cannot be modified by an untrusted process. In practice, this means it runs in kernel mode, protected from user code.
  3. Verifiable: the monitor is small enough that its correctness can be verified — by inspection, by testing, or by formal methods.
In Linux, the reference monitor is the combination of the permission checks in the kernel (VFS permission checks, capability checks, LSM hooks). It is complete (the kernel mediates all syscalls) and tamper-proof (it runs in kernel mode), but it is not fully verifiable because the kernel is large. MAC systems like SELinux provide a policy layer that can be more easily reasoned about, but they still rely on the kernel's mediation.

Question 9

What is a TOCTOU (time-of-check to time-of-use) vulnerability? Give an example.

Show answer
A TOCTOU vulnerability is a race condition in which a program checks a condition (usually a permission or the state of a resource) and then uses the resource, but between the check and the use, an attacker can change the resource's state so that the check no longer reflects the state at the time of use. Example: a setuid root program that checks whether the caller is allowed to write to a file, and then opens the file with fopen. Between the check and the open, the attacker can replace the file with a symbolic link to /etc/shadow. The check passes (the attacker's own file is writable), but the open follows the symlink and writes to /etc/shadow. The fix is to avoid the check-then-use pattern: open the file first (with O_NOFOLLOW to prevent symlink attacks), then check the opened file descriptor's properties (owner, permissions) with fstat. The check and the use now operate on the same object, so there is no window for an attacker.

Question 10

What is the difference between a capability and a setuid bit?

Show answer
A setuid bit makes a binary run with the entire privilege set of its owner (usually root). The binary can perform any operation root can perform. A capability is a fine-grained privilege: for example, CAP_NET_BIND_SERVICE allows binding to a low-numbered port; CAP_DAC_READ_SEARCH allows bypassing file read-permission checks; CAP_SYS_ADMIN grants a broad set of administrative operations. A binary can be granted one or more capabilities with setcap, and it can perform only the operations those capabilities allow. This is a direct application of the principle of least privilege: a binary that needs to bind a low port gets only CAP_NET_BIND_SERVICE, not the entire root privilege set. Capabilities are strictly safer than setuid root, because a compromise of the binary exposes only the specific capabilities it holds.

Question 11

What are the mechanisms Linux provides for sandboxing? Describe at least four.

Show answer
Linux provides several complementary mechanisms:
  1. Namespaces: isolate a process's view of system resources — mount (file system), PID (process IDs), network (network interfaces), UTS (hostname), IPC (System V IPC and POSIX message queues), user (UID/GID mapping), and cgroup (resource limits). Containers are built on namespaces.
  2. seccomp: a filter that restricts which system calls the process may make. Denied calls cause the process to be killed with SIGSYS (or return an error, depending on the filter).
  3. cgroups: resource limits on CPU, memory, I/O, and device access.
  4. MAC policies (SELinux, AppArmor): restrict which files, network ports, and other resources the process may access, regardless of DAC permissions.
  5. Capability bounding sets: restrict the capabilities a process can ever acquire (for example, even if a binary is setuid root, if the capability is not in the bounding set, the process cannot use it).
Together, these mechanisms allow a process to be confined to exactly the resources it needs. Modern sandboxing tools (bubblewrap, firejail, systemd's Protect* properties) combine several of them.

Question 12

What is the difference between DAC and MAC? Give an example of a DAC failure that MAC would prevent.

Show answer
DAC (discretionary access control) lets the owner of a resource decide who may access it. MAC (mandatory access control) enforces a global policy that even the owner cannot override. Example of a DAC failure: a web server runs as user www-data and serves files from /var/www. The system administrator accidentally leaves a sensitive file (such as a copy of /etc/shadow) in /var/www with world-readable permissions. Under DAC, any process running as www-data can read the file, and a compromise of the web server (for example, through a bug in a CGI script) gives the attacker access to the file. Under MAC (for example, SELinux), the file would be labelled shadow_t or a similar restricted type, and the web server's domain would not be permitted to read files of that type — even if the DAC permissions allowed it. The policy is set by the administrator and cannot be overridden by the file's owner or by the web server process.

Question 13

What is the principle of "defence in depth", and how is it applied in operating-system security?

Show answer
Defence in depth is the principle that a system should not rely on a single security mechanism. If that mechanism fails (because of a bug, a misconfiguration, or a new attack), other mechanisms should still provide protection. In operating-system security, defence in depth is applied by layering:
  • Authentication verifies identity (passwords, tokens, biometrics).
  • DAC enforces the resource owner's wishes (mode bits, ACLs).
  • MAC enforces the system's global policy (SELinux, AppArmor).
  • Capabilities limit what a privileged process can do.
  • Sandboxing (namespaces, seccomp) restricts what a process can see and call.
  • Auditing records what happened so that incidents can be investigated.
  • Kernel hardening (ASLR, W^X, stack canaries) makes exploitation harder.
Each layer alone is imperfect; together they make a successful attack require defeating multiple independent mechanisms.

Question 14

What does the Linux capabilities system replace, and what are three common capabilities?

Show answer
Capabilities replace the all-or-nothing distinction between an unprivileged process and a root process. Traditional UNIX gives a process either the full set of root privileges (if its effective UID is 0) or none (otherwise). Capabilities divide root's privileges into about 40 distinct units, each granting a specific operation. Three common capabilities:
  • CAP_NET_BIND_SERVICE: bind a socket to a port below 1024.
  • CAP_DAC_READ_SEARCH: bypass file read-permission checks and directory search-permission checks.
  • CAP_SYS_ADMIN: a broad set of administrative operations (mount, swapon, access to many device operations); often called "the new root" because granting it gives almost all of root's power.
A binary can be granted one or more capabilities with setcap, and a process can drop unneeded capabilities with capset. This is a direct application of least privilege.

Question 15

Why is auditing important for operating-system security, and what does Linux's auditd record?

Show answer
Auditing is the recording of security-relevant events so that they can be analysed after the fact. It is important because (a) it allows an administrator to detect attacks that have already happened, (b) it provides evidence for forensic investigation, (c) it can serve as a deterrent (users know their actions are recorded), and (d) it is required for compliance with many security standards (PCI DSS, HIPAA, SOX). Linux's auditd records system calls, file accesses, and other security-relevant events. Rules can be configured to log specific events: auditctl -w /etc/passwd -p wa -k passwd_changes logs any write or attribute change to /etc/passwd with the key passwd_changes. The logs can be queried with ausearch and summarised with aureport.

Question 16

What is a "sandbox escape", and why is sandbox design difficult?

Show answer
A sandbox escape is a vulnerability or technique by which a process confined within a sandbox gains access to resources outside it. Sandboxing is difficult because a sandbox is only as strong as its weakest enforcement point. Common escape vectors include:
  • Kernel bugs: if the process can trigger a kernel bug (use-after-free, integer overflow) from within the sandbox, it can execute code in kernel mode and escape.
  • Incomplete mediation: if the sandbox does not intercept every relevant operation — for example, if it filters system calls but not ioctl commands, or if it filters file paths but not file descriptors — a process may reach a resource indirectly.
  • Side channels: a process may infer the contents of memory or the behaviour of other processes by observing cache timings, CPU frequency, or power consumption.
  • Shared resources: if the sandbox shares a resource (a file, a device, an IPC channel) with the outside, the process may exploit the sharing.
Effective sandboxing requires combining multiple mechanisms (namespaces, seccomp, MAC, capabilities), keeping the kernel up to date, and minimising the resources the sandbox exposes. Modern browsers, containers, and mobile operating systems all invest heavily in sandbox design.

Question 17

A program is setuid root and contains a buffer overflow in its parsing of a command-line argument. Describe how an attacker could exploit this to gain root.

Show answer
Because the program is setuid root, the process runs with root's effective UID. The attacker supplies a long command-line argument that overflows a stack buffer, overwriting the saved return address with the address of a code sequence the attacker controls. When the vulnerable function returns, control jumps to the attacker's code, which runs with the process's privileges — root. The attacker's code can do anything root can do: create a new user, modify /etc/shadow, install a rootkit, or spawn a root shell. The classic mitigations are stack canaries (which detect the overflow before the return), non-executable stacks (W^X, which prevent code execution from the stack), ASLR (which makes the address of the attacker's code unpredictable), and, at the design level, avoiding setuid root in the first place (using capabilities instead). None of these mitigations is perfect; the correct approach is defence in depth.

Question 18

What is the purpose of a MAC policy? Why can it not be replaced by careful DAC configuration?

Show answer
The purpose of a MAC policy is to enforce a global, system-wide access-control policy that no individual user or process can override. It cannot be replaced by careful DAC configuration because DAC decisions are made by resource owners, and owners make mistakes. A user may inadvertently create a world-readable file containing sensitive data; a process may run with more permissions than it needs because the administrator cannot anticipate every file it will access; a compromised process running as a privileged user can change the DAC permissions on any file that user owns. MAC closes these gaps by defining, at the system level, which processes may access which resources — regardless of DAC permissions and regardless of who owns the resource. SELinux and AppArmor are the two main MAC implementations in Linux; both are applied to critical system services (web servers, databases, containers) to bound the damage that a compromise can do.

Question 19

What is the principle of "fail-safe defaults", and give an example of its application in operating-system design.

Show answer
The principle of fail-safe defaults states that access decisions should default to denial: a process should be denied access unless it has been explicitly granted. The alternative — "fail open" — is dangerous because any error in the access-control logic results in a permission being granted rather than denied. An example of fail-safe defaults in operating-system design is the default permissions on a newly created file: the umask (022 on most systems) ensures that new files are created with mode 644 (owner can read and write, everyone else can only read) rather than 666 or 777. Another example is SELinux's default policy: a process that is not explicitly permitted to access a file is denied, even if the DAC permissions allow it. Fail-safe defaults reduce the impact of configuration errors and are a foundational principle of secure design.

Question 20

What is the difference between a "vulnerability", an "exploit", and a "threat"?

Show answer
A vulnerability is a weakness in a system that could be used to violate a security policy. A threat is an adversary or event that could exploit a vulnerability. An exploit is the specific technique or code that takes advantage of a vulnerability. For example, a buffer overflow in a setuid program is a vulnerability; a malicious user who wants to gain root is a threat; a specific sequence of bytes that overflows the buffer and injects shellcode is an exploit. The three concepts are related but distinct: a system may have many vulnerabilities and no active threat, or an active threat and no exploitable vulnerabilities. Risk is a function of all three: risk = vulnerability × threat × impact.

Question 21

Why is "security through obscurity" considered a poor design principle?

Show answer
Security through obscurity is the practice of relying on the secrecy of a system's design or implementation as the primary means of protection — for example, using a proprietary algorithm that is not published, or hiding a password in a binary. It is considered poor because:
  • Secrets are hard to keep. Binary analysis, disassembly, and side-channel observation can reveal the design of a hidden algorithm; a disgruntled employee can leak the design; a system can be reverse-engineered.
  • Obscurity prevents review. If the design is secret, it cannot be analysed by the security community, and bugs remain undiscovered until an attacker finds them.
  • Obscurity does not scale. A system that relies on obscurity must keep every instance's secret; a system that relies on a well-analysed public algorithm (AES, SHA-256) can publish the algorithm and keep only the keys secret.
This does not mean obscurity is useless: it can raise the cost of an attack and provide defence in depth. But it must never be the primary defence.

Question 22

What is the difference between white-box and black-box security testing? Which is more thorough?

Show answer
Black-box testing evaluates a system without knowledge of its internal implementation: the tester interacts with the system as an external attacker would, probing inputs and observing outputs. White-box testing evaluates the system with full knowledge of its source code, architecture, and configuration. White-box testing is generally more thorough because the tester can identify vulnerabilities that are not reachable through a particular input sequence (for example, a bug in a rarely used code path), can reason about the interaction between components, and can use static analysis tools that examine the code. Black-box testing is valuable because it captures the attacker's perspective and can find bugs that only emerge in the deployed configuration. Both are used in practice: a comprehensive security assessment typically combines both, along with grey-box testing (partial knowledge).

Question 23

A system administrator finds that a server has been compromised. List the first three things they should do, and explain why.

Show answer
The first three steps should be:
  1. Isolate the system from the network (unplug or block with firewall rules) so that the attacker cannot continue to access it, exfiltrate data, or pivot to other systems. Do not power off the system — the memory contains volatile evidence (process lists, network connections, encryption keys) that will be lost.
  2. Preserve evidence by making a forensic image of the system's memory (with dd or a memory acquisition tool) and its disks (with dd or dcfldd). The image becomes the basis for investigation; the original system may be reimaged for continued operation.
  3. Notify the appropriate parties — management, the security team, and, if required, law enforcement or regulators. The notification timeline may be mandated by law or contract (for example, GDPR requires notification within 72 hours in many cases).
The order matters: isolation stops the attack, evidence preservation keeps the investigation possible, and notification satisfies legal and organisational requirements. A common mistake is to "clean up" first — rebooting, deleting suspicious files, reinstalling software — which destroys evidence and may leave the attacker's persistence mechanisms in place.

Question 24

In one or two sentences, state the central challenge of operating-system security and how the layers of defence address it.

Show answer
The central challenge is that an operating system must mediate every access to every resource while running untrusted code from many sources, and any bug in the mediation logic or in the kernel itself can be exploited to bypass it entirely. The layers of defence — authentication, DAC, MAC, capabilities, sandboxing, auditing, kernel hardening — address this by ensuring that no single failure is sufficient to compromise the system: an attacker who defeats one mechanism still faces the others, and the attacker's activity is recorded for detection and response.

Ready for the graded quiz?

Once you are comfortable with every question above, take the graded Unit 12 quiz in the LMS. It covers the same material with multiple-choice questions and short scenarios requiring a security analysis.

👤