TR1.4 Linux Evidence Volatility

Figure TR1.4 — Linux evidence volatility layers. /proc state vanishes on process exit. Container layers vanish on restart. Memory requires AVML or LiME before reboot. Logs persist longest but rotate.

/proc: the virtual filesystem that exists only while the system runs

/proc is the single most valuable evidence source for Linux triage. Unlike regular files stored on disk, /proc is a virtual filesystem generated by the kernel in real time. Every running process gets a directory at /proc/PID/ containing the process's complete runtime state — its command line arguments, executable path, open file descriptors, memory map, network sockets, and environment variables. None of this is written to disk. All of it vanishes the moment the process terminates or the system reboots.

The triage responder reads /proc the same way the Windows analyst reads the process table — but /proc offers something Windows doesn't provide natively. When an attacker deletes their binary from /tmp/backdoor to cover their tracks, /proc/PID/exe still contains a symbolic link to the deleted file. The kernel retains the binary in memory as long as the process keeps running. Running cp /proc/PID/exe /evidence/recovered_binary recovers the attacker's tool even after they wiped it from disk.

On Windows, recovering a deleted-but-running executable requires full memory analysis. On Linux, /proc gives you a direct file copy without the overhead of a memory dump.

The /proc/PID/environ file deserves its own mention. This file contains every environment variable set when the process launched, including variables the attacker may have passed on the command line — API keys, remote server addresses, or encoded payloads stored in custom environment variables. Attackers who launch tools with PAYLOAD=base64... ./tool leave those variables readable in /proc until the process exits. This evidence is invisible to any disk forensic technique.

The /proc/PID/maps file shows every shared library loaded into the process's address space. This is critical for detecting LD_PRELOAD rootkits — a technique where the attacker injects a malicious shared library that intercepts standard C library calls. The rootkit hooks functions like readdir() to hide files and getdents() to hide processes from ps output. Because the rootkit operates at the shared library level, standard commands like ps, ls, and netstat return sanitised results that conceal the attacker's presence.

But the kernel's view in /proc/PID/maps shows every loaded library, including the injected one. If /proc/PID/maps for every process contains a reference to /usr/lib/libkeyutils-1.6.so but that file doesn't exist on disk — or its hash doesn't match the package manager's record — the analyst has found the rootkit.

# Tier 1: capture /proc state for all running processes
mkdir -p /evidence/proc_state
for pid in /proc/[0-9]*; do
    p=$(basename "$pid")
    mkdir -p /evidence/proc_state/"$p"
    cat "$pid"/cmdline > /evidence/proc_state/"$p"/cmdline 2>/dev/null
    cat "$pid"/environ > /evidence/proc_state/"$p"/environ 2>/dev/null
    ls -la "$pid"/exe > /evidence/proc_state/"$p"/exe_link 2>/dev/null
    cat "$pid"/status > /evidence/proc_state/"$p"/status 2>/dev/null
    ls -la "$pid"/fd/ > /evidence/proc_state/"$p"/fd_list 2>/dev/null
    cat "$pid"/maps > /evidence/proc_state/"$p"/maps 2>/dev/null
done
# Network connections with owning process
ss -tnp > /evidence/network_connections.txt
# Loaded kernel modules — rootkit detection baseline
lsmod > /evidence/kernel_modules.txt

This script iterates through every running process in 10–15 seconds on a typical server. The 2>/dev/null redirections are deliberate — some kernel threads in /proc don't expose all files, and permission-denied errors for processes owned by other users would otherwise interrupt the loop. The analyst runs the script as root to ensure full visibility across all process namespaces.

The lsmod output provides the kernel module baseline. If the investigation team later finds a module in the memory dump that doesn't appear in lsmod, that's evidence of a hiding rootkit — the module has unlinked itself from the kernel's module list but still occupies memory. Comparing lsmod output against /proc/modules (which the kernel populates independently) can also reveal discrepancies introduced by rootkits that hook one interface but not the other.

The network connections captured by ss -tnp show what the system is talking to right now:

State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
ESTAB   0       0       10.50.2.18:443       185.220.101.34:49122  users:(("nginx",pid=14231,fd=12))
ESTAB   0       0       10.50.2.18:5432      10.50.2.18:38214      users:(("postgres",pid=8421,fd=8))
ESTAB   0       0       10.50.2.18:22        10.50.1.50:52841      users:(("sshd",pid=19432,fd=4))
LISTEN  0       128     0.0.0.0:443          0.0.0.0:*             users:(("nginx",pid=14231,fd=6))
LISTEN  0       128     0.0.0.0:22           0.0.0.0:*             users:(("sshd",pid=1204,fd=3))

The first row reveals the compromise. Nginx (PID 14231) is holding an established outbound connection to the attacker's external IP on port 443. Nginx should only accept inbound connections from clients — it should never initiate outbound HTTPS sessions to external addresses. The attacker's modified Nginx configuration is proxying C2 traffic through the legitimate web server, disguising the malicious traffic as normal web responses.

The -p flag maps the connection to the owning process, and the PID maps directly to the container where Nginx runs. This connection disappears the moment the container restarts.

Container evidence: the most fragile layer

Container evidence occupies its own volatility tier between Tier 1 (network state) and Tier 2 (memory). When a container restarts — triggered by the orchestrator, a crash, or a manual restart — the entire writable layer is destroyed and rebuilt from the base image. The attacker's modified Nginx configuration, any tools dropped into the container filesystem, and any log entries written inside the container are gone. This isn't a partial loss like a log rotation. It's a total replacement of the filesystem state.

The capture sequence for containers runs before any other action on a containerised system. docker inspect captures the running configuration — environment variables, mounted volumes, network settings, port bindings, and the restart policy that determines when evidence destruction happens automatically. docker diff shows every file the attacker added, modified, or deleted relative to the base image.

This is the forensic evidence that proves exactly what the attacker changed. docker export creates a tar archive of the container's entire filesystem including the writable layer modifications, preserving the evidence for offline analysis even if the container restarts minutes later.

For Kubernetes environments, the equivalent commands are kubectl describe pod, kubectl logs, and kubectl cp to extract specific files from the running pod. The urgency is higher in Kubernetes because the cluster scheduler may replace the pod at any time based on health checks, resource pressure, or rolling update policies. If the compromised pod fails a liveness probe, Kubernetes terminates it and spins up a clean replacement automatically — destroying every artifact the attacker left behind without any human intervention.

The Docker restart policy matters for triage planning. A container with restart: always restarts automatically after any exit, replacing the writable layer. A container with restart: on-failure only restarts on non-zero exit codes. A container with restart: no stays down after a crash. The attacker in the NE scenario chose Nginx precisely because its restart policy is always — if the investigation team accidentally triggers a restart during evidence collection, the C2 configuration is wiped clean.

Memory acquisition on Linux: AVML and LiME

Physical memory on a Linux server contains the same categories of exclusive evidence as Windows — decrypted payloads, cached credentials, active network buffers, and injected code that exists nowhere on disk. Two tools dominate Linux memory acquisition, and the choice between them depends on the target system's kernel configuration.

LiME (Linux Memory Extractor) is the established tool — a loadable kernel module that dumps physical memory to a file or over a TCP socket. LiME's strength is comprehensive coverage: because it runs in kernel space, it can access all physical memory including regions protected from userland access. The challenge is compilation. LiME must be built against the target system's exact kernel headers. If the target runs kernel 5.15.0-97-generic and you compiled LiME against 5.15.0-94-generic, the module won't load.

Pre-compiling LiME modules for every kernel version running in the environment is the preparation step that most organisations skip — and then discover the gap at 03:00 during an active incident.

AVML (Acquire Volatile Memory for Linux), developed by Microsoft, takes a different approach. AVML is a static Rust binary that runs entirely in userland — no kernel module compilation required. It acquires memory through three sources, tried in order: /dev/crash (page-aligned read-only physical memory), /proc/kcore (virtual ELF coredump of kernel memory), and /dev/mem (direct read-only physical memory access). Because AVML ships as a single static binary, the analyst copies it to the target system and runs it immediately without matching kernel versions or installing dependencies.

The tradeoff is a real operational constraint. AVML cannot acquire memory when the kernel's kernel_lockdown feature is enabled. On modern distributions with UEFI Secure Boot, kernel_lockdown activates automatically at boot — it blocks access to /dev/mem, /dev/crash, and /proc/kcore to prevent kernel image extraction. Ubuntu 22.04 with Secure Boot enabled will block AVML entirely.

In that scenario, the analyst either disables Secure Boot and reboots (destroying volatile evidence — defeating the purpose) or falls back to LiME (which can load as a signed kernel module if properly prepared). This is why incident readiness planning must test memory acquisition tools against the exact boot configuration of production systems, not just a lab VM with Secure Boot disabled.

Tier 3: logs and the rotation schedule

Linux log rotation follows cron schedules configured in /etc/logrotate.d/ or managed by systemd's journald. The critical triage log is auth.log (Debian/Ubuntu) or secure (RHEL/CentOS), which records every SSH authentication attempt, sudo invocation, and user session event. The attacker's initial SSH connection generates log entries that tell the full story:

May 18 03:14:07 web-prod-01 sshd[19432]: Accepted publickey for deploy from
    185.220.101.34 port 52841 ssh2: RSA SHA256:kF7x9pQm3...
May 18 03:14:07 web-prod-01 sshd[19432]: pam_unix(sshd:session): session
    opened for user deploy(uid=1001) by (uid=0)
May 18 03:14:12 web-prod-01 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ;
    USER=root ; COMMAND=/usr/bin/docker exec -it nginx-proxy /bin/sh
May 18 03:14:15 web-prod-01 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ;
    USER=root ; COMMAND=/usr/bin/docker cp ./nginx.conf nginx-proxy:/etc/nginx/
May 18 03:14:22 web-prod-01 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ;
    USER=root ; COMMAND=/usr/bin/docker exec nginx-proxy nginx -s reload

Five log entries tell the complete lateral movement story. The attacker authenticated as deploy using a stolen SSH key from the compromised Azure VM's IP address. Within seconds, they used sudo to execute commands inside the Nginx container — first opening a shell, then copying a modified nginx.conf, then reloading Nginx to activate the C2 proxy configuration. Each sudo entry records the exact command, the working directory, and the target user.

This is the evidence chain that connects the Azure VM compromise (TR1.2) to the container modification, establishing the attacker's lateral movement path across environments.

The rotation risk is real. On a default Ubuntu 22.04 configuration, auth.log rotates weekly with 4 retained copies — roughly 4 weeks of history. But on a busy server processing hundreds of SSH connections daily, the log may reach its configured size limit before the weekly rotation triggers, causing an early rotation that shortens the evidence window unexpectedly. The triage action is to copy auth.log and all rotated copies (auth.log.1, auth.log.2.gz) to the evidence folder immediately.

On systems using journald with persistent storage, journalctl --since "2026-05-18" --until "2026-05-19" -o json > /evidence/journal_export.json captures the systemd journal for the incident timeframe in a structured format that's easier to parse than raw syslog.

Two additional log sources deserve triage priority on servers running containers. Docker's own logs (/var/lib/docker/containers/CONTAINER_ID/*.log) capture stdout and stderr from inside each container — including Nginx access and error logs that record the C2 proxy connections. And bash_history for every interactive user account records the exact commands typed during the SSH session, providing an independent record of attacker activity that corroborates the sudo entries in auth.log.

The triage report documents what the analyst captured, what they found, and — critically — what they couldn't capture. The memory acquisition gap (AVML blocked by kernel_lockdown, LiME not pre-compiled) is documented explicitly so the incident response team knows what evidence was missed and why. Omitting the gap from the report creates a false impression that memory was either acquired or not needed. Documenting the gap creates accountability and drives the readiness improvement that prevents the same gap in the next incident.