LX0.9 The Linux Security Architecture

The Linux Security Architecture: What Investigators Encounter

Why Security Architecture Matters for Investigation

The security architecture of a Linux system determines two things that directly affect your investigation: what the attacker had to bypass to achieve their objective, and what evidence was generated when they did. An attacker who exploits a privilege escalation vulnerability on a system running SELinux in enforcing mode generates different evidence than the same attacker on a system with SELinux disabled. The SELinux-enforcing system has AVC (Access Vector Cache) denial logs showing every operation the attacker attempted that was blocked — a detailed record of the attacker’s trial-and-error process. The SELinux-disabled system has none of this.

Understanding the security architecture also tells you what the attacker could and could not do without additional exploitation. If AppArmor confined the web server process, the attacker who compromised the web application could only access files and network connections permitted by the AppArmor profile — unless they escaped the confinement. Evidence of AppArmor escape attempts appears in the kernel log and the AppArmor audit log.

Discretionary Access Control: Traditional Unix Permissions

The foundational security model is discretionary access control (DAC) — the familiar user/group/other permission bits on every file and directory. DAC is “discretionary” because the file owner can change the permissions at will. From an investigation perspective, the permission bits are evidence:

A file owned by root with permissions rwxr-xr-x that is located in /tmp and was created during the compromise window is suspicious — a root-owned binary in a world-writable directory suggests privilege escalation was achieved before the file was written. A file owned by www-data (the web server user) with permissions rwxrwxrwx in the web root is doubly suspicious — it has execute permissions that web content should not have, and it is world-writable.

The SUID (Set User ID) and SGID (Set Group ID) bits are the most forensically significant permission bits. A binary with the SUID bit set runs with the permissions of the file owner (typically root), regardless of who executes it. SUID binaries are the primary privilege escalation vector on Linux — if a SUID root binary has a vulnerability, any user can exploit it to gain root access. The investigation technique: enumerate all SUID binaries with find / -perm -4000 -type f 2>/dev/null and compare against a known-good baseline. Any SUID binary that does not appear on the baseline was added by someone — possibly the attacker, who may have copied a vulnerable binary onto the system or set the SUID bit on an existing binary to create a backdoor.

The sticky bit on directories (permission t in the last position, typically set on /tmp) prevents users from deleting files owned by other users. If the sticky bit is missing from /tmp, any user can delete any file — including log files and evidence that the investigator has not yet collected.

Mandatory Access Control: SELinux and AppArmor

Mandatory access control (MAC) frameworks enforce security policies that even the root user cannot override through normal operations. Unlike DAC where the file owner controls access, MAC policies are defined by the system administrator and enforced by the kernel. An attacker who gains root access on a system with properly configured MAC faces additional constraints — root is not all-powerful when MAC is in enforcing mode.

SELinux (RHEL, CentOS, Fedora, Amazon Linux) assigns a security context (label) to every process, file, port, and system object. The SELinux policy defines which contexts can access which other contexts and through which operations. When a process attempts an operation that the policy does not permit, SELinux blocks the operation and logs an AVC denial.

For investigation, SELinux AVC denials are evidence of attacker activity. If the compromised web server process (running in the httpd_t context) attempted to read /etc/shadow (labeled shadow_t), SELinux would block the read and log: type=AVC msg=audit(1711601862.447:1284): avc: denied { read } for pid=4521 comm="httpd" name="shadow" dev="sda1" ino=262178 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:shadow_t:s0 tclass=file. This single log entry tells you: the web server process (PID 4521) tried to read the shadow password file and was blocked. That is evidence of post-exploitation credential harvesting.

Query SELinux denials: ausearch -m avc -ts today (today’s denials), ausearch -m avc -ts "03/28/2026" -te "03/29/2026" (specific date range). Look for: denials from the compromised process’s context, denials related to sensitive files, and transitions between SELinux domains (which indicate the attacker attempted to change the process’s security context to gain additional permissions).

An attacker who discovers SELinux is blocking their activity may attempt to disable it: setenforce 0 switches from enforcing to permissive mode. This is logged: type=MAC_STATUS msg=audit(1711601900.123:1285): enforcing=0 old_enforcing=1. If you find this entry in the audit log during the compromise window, the attacker deliberately disabled SELinux — and everything they did after that point was no longer constrained or logged by the MAC framework.

AppArmor (Ubuntu, Debian, SUSE) uses path-based profiles rather than labels. Each profile defines which files a program can access, which network operations it can perform, and which capabilities it can use. AppArmor profiles for common services (nginx, mysql, sshd) are shipped by the distribution and can be customized.

For investigation, AppArmor generates audit log entries when a confined process violates its profile. Check journalctl -k | grep apparmor or dmesg | grep apparmor for AppArmor events. A violation from the web server profile during the compromise window indicates the attacker attempted an operation outside the web server’s normal behavior — a strong indicator of post-exploitation activity.

Linux Capabilities: Fine-Grained Root Privileges

Traditional Unix has a binary privilege model: you are either root (UID 0, all privileges) or you are not. Linux capabilities split root’s privileges into approximately 40 individual capabilities, each governing a specific operation. A process can have a subset of root’s capabilities without having full root access.

For investigation, capabilities explain why a non-root process was able to perform a privileged operation. The CAP_NET_RAW capability allows raw network access (packet sniffing). CAP_SYS_ADMIN is the “catchall” capability that grants most root-like operations — including mounting filesystems, loading kernel modules, and modifying namespaces. CAP_SYS_PTRACE allows attaching to other processes (used by debuggers and also by attackers for process injection). CAP_DAC_READ_SEARCH bypasses file permission checks for reading — an attacker with this capability can read any file on the system regardless of its permission bits.

Check process capabilities with: cat /proc/[pid]/status | grep Cap (raw capability bitmasks) or getpcaps [pid] (human-readable capability list). If a compromised container process has CAP_SYS_ADMIN, the attacker can escape the container to the host — this is one of the primary container escape vectors investigated in LX9.

Namespaces: Container Isolation and Its Forensic Implications

Linux namespaces isolate system resources for processes. Each namespace type isolates a different resource: PID namespace (processes), network namespace (network interfaces and routing), mount namespace (filesystem mounts), user namespace (UIDs and GIDs), UTS namespace (hostname), IPC namespace (inter-process communication), and cgroup namespace (resource limits).

Containers use namespaces as their primary isolation mechanism. A process inside a Docker container runs in its own PID namespace — ps inside the container shows only the container’s processes, not the host’s processes. A process inside the container has its own network namespace — ss inside the container shows only the container’s network connections.

For investigation, this means: commands run inside a container show container-scoped results, not host-scoped results. If you kubectl exec into a compromised pod and run ps aux, you see the pod’s processes. You do not see the host’s processes or other containers’ processes. To see the full picture — all processes on the host including all containers — you must run commands on the host, outside the container namespace.

The investigation technique for containers: always collect evidence from both inside the container namespace (using docker exec or kubectl exec) AND from the host (using SSH to the node). Cross-reference the two views. A process that appears on the host but not inside the container may be a container escape — the attacker’s process has broken out of the PID namespace.

The Audit Subsystem: The Investigator’s Best Friend (When Configured)

The Linux audit subsystem (auditd) is the most powerful forensic evidence source on Linux and the most underdeployed. When properly configured, auditd records every system call made by every process on the system — file access, process execution, network operations, module loading, privilege changes, and user authentication. The level of detail exceeds what Windows Event Log provides by default.

The audit subsystem consists of three components: the kernel audit framework (records events), the audit daemon auditd (writes events to /var/log/audit/audit.log), and the audit rule set (defines which events to record). The default installation on most distributions enables auditd but configures minimal rules — typically only authentication events and audit configuration changes.

The investigation-critical rules that should be deployed on every Linux server proactively:

Process execution monitoring: -a always,exit -F arch=b64 -S execve -k process_exec — records every command executed on the system. This is the Linux equivalent of Windows Event ID 4688 (Process Creation). Without this rule, you have no record of what commands the attacker ran unless they used an interactive shell with history enabled.

File modification monitoring: -w /etc/passwd -p wa -k identity_change and -w /etc/shadow -p wa -k identity_change — records any modification to the authentication databases. -w /etc/sudoers -p wa -k priv_change — records sudo configuration changes. -w /etc/ssh/sshd_config -p wa -k ssh_config — records SSH configuration changes.

Persistence monitoring: -w /var/spool/cron -p wa -k cron_change — records cron job modifications. -w /etc/systemd/system -p wa -k systemd_change — records new or modified systemd services. -w /root/.ssh/authorized_keys -p wa -k ssh_key_change — records SSH key deployment.

Module loading: -w /sbin/insmod -p x -k module_load and -w /sbin/modprobe -p x -k module_load — records kernel module loading (rootkit detection).

The auditd configuration is stored in /etc/audit/audit.rules (persistent) and can be queried with auditctl -l (current running rules). During an investigation, the first thing to check is whether auditd was running and what rules were configured. If comprehensive rules were deployed, the audit log contains the complete record of the attacker’s activity. If only default rules were deployed, the audit log contains authentication events and little else. This determination shapes the entire investigation — with comprehensive audit logs, you can reconstruct every command the attacker ran. Without them, you rely on the fragmentary evidence in bash history, filesystem timestamps, and log files.

Beyond This Investigation

Security architecture knowledge is applied directly in the investigation scenarios. LX6 (Privilege Escalation) examines SUID exploitation and capability abuse. LX7 (Persistence) examines how attackers create persistence mechanisms that survive MAC enforcement. LX9 (Container Compromise) examines namespace escapes and capability-based container breakouts. LX15 (Detection Engineering) deploys auditd rules based on the investigation findings — converting the reactive evidence discovery into proactive monitoring.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Security posture assessment — determines your evidence landscape

# MAC framework status
getenforce 2>/dev/null && echo "SELinux active" || true
aa-status 2>/dev/null && echo "AppArmor active" || true

# Audit daemon status and rule count
systemctl is-active auditd 2>/dev/null
auditctl -l 2>/dev/null | wc -l
# 3 rules = minimal (auth only). 20+ rules = comprehensive. 0 = not running.

# SUID binary count (compare against known-good baseline)
find / -perm -4000 -type f 2>/dev/null | wc -l
# Typical Ubuntu server: 15-25 SUID binaries. Significantly more = suspicious.

# SELinux denial check (if SELinux is active)
ausearch -m avc -ts recent 2>/dev/null | head -20
# AVC denials during the compromise window = attacker hitting SELinux boundaries

Worked artifact — Security posture assessment:

Run this assessment at the start of every investigation to understand the evidence landscape.

System: [hostname] Distribution: [ID from os-release]

MAC framework: ☐ SELinux enforcing ☐ SELinux permissive ☐ SELinux disabled ☐ AppArmor enforcing ☐ AppArmor complain ☐ None Audit daemon: ☐ auditd running (rules: ___) ☐ auditd installed but minimal rules ☐ auditd not installed SUID binaries: Count: ___ (compare against baseline) Capabilities: Notable: ___ Firewall: ☐ iptables ☐ nftables ☐ firewalld ☐ ufw ☐ None

Evidence implications: More active security layers = more evidence generated by attacker actions. Disabled/absent layers = dark zones where attacker activity is not recorded.

Decision points: interpreting security architecture for investigation

SELinux enforcing + auditd comprehensive rules: richest evidence environment. Check AVC denials for attacker activity, check audit.log for every command executed.

AppArmor + auditd minimal: AppArmor violations in kernel log, but limited command execution evidence. Rely on bash history and /proc.

No MAC + no auditd: minimal evidence environment. Rely entirely on filesystem timestamps, bash history, log files, and /proc. This is the most common situation on production servers.

Troubleshooting: security architecture evidence gaps

SELinux was changed from enforcing to permissive during the compromise. Check ausearch -m MAC_STATUS for the timestamp. Everything after that timestamp was not constrained by SELinux — the attacker deliberately disabled it.

auditd is running but has no execve rules. Only authentication events are recorded. Command execution evidence is limited to bash history and /proc. Recommend deploying comprehensive rules for future incidents.

Cannot determine if SUID binaries are legitimate. Compare against the package manifest: dpkg -S /path/to/binary (Debian) or rpm -qf /path/to/binary (RHEL). If the binary is not owned by any package, it was placed there manually.