In this section
LX0.3 The Volatility Problem
The Volatility Problem: Why Collection Order Determines Investigation Success
Evidence that disappears
On Windows, most forensic evidence is persistent. The registry survives reboots. Prefetch files survive reboots. The Event Log survives reboots. The MFT survives reboots. An investigator who images a Windows system three days after a compromise will recover most of the evidence — the same artifacts that existed during the incident still exist on the disk.
On Linux, critical evidence categories are ephemeral by design. They exist only while the system is running, only while a process is active, or only until the next log rotation cycle. An investigator who waits three days to begin collection may find that the most important evidence no longer exists — not because the attacker destroyed it, but because the system's normal operations did.
# Demonstrate the volatility of /proc — run this on a test system
# Start a background process
sleep 300 &
BGPID=$!
echo "Process started: PID $BGPID"
# Read its evidence from /proc while it's alive
cat /proc/$BGPID/cmdline | tr '\0' ' ' # Output: sleep 300
readlink /proc/$BGPID/exe # Output: /usr/bin/sleep
cat /proc/$BGPID/status | head -5 # Name, state, PID, UID
# Now kill the process
kill $BGPID
sleep 1
# Try to read the same evidence — it's gone
cat /proc/$BGPID/cmdline 2>&1 # Output: No such file or directory
# The entire /proc/$BGPID directory was destroyed when the process exited
# This is why volatile collection must happen BEFORE containment actions# Check your evidence retention windows — run this immediately on any
# compromised system to understand how much history you have
# Log rotation policy
echo "=== LOG ROTATION POLICY ==="
cat /etc/logrotate.d/rsyslog 2>/dev/null || cat /etc/logrotate.d/syslog 2>/dev/null
# Look for: rotate N (number of copies), weekly/daily, compress
# weekly + rotate 4 = 28 days of history
# Journal retention
echo "=== JOURNAL RETENTION ==="
journalctl --disk-usage
# Output: Archived and active journals take up 48.0M on disk
grep -E "SystemMaxUse|MaxRetentionSec" /etc/systemd/journald.conf
# SystemMaxUse=50M → journal limited to 50MB (may be only days on busy servers)
# Tmpfiles cleanup
echo "=== TMPFILES CLEANUP ==="
grep -r "tmp" /usr/lib/tmpfiles.d/tmp.conf 2>/dev/null
# Look for: d /tmp 1777 root root 10d
# 10d = files older than 10 days are deleted from /tmp daily
# Check if /tmp is a tmpfs (RAM-backed — lost on reboot)
mount | grep "/tmp"
# Output contains "tmpfs" → /tmp is RAM-backed, lost on reboot
# No output or shows a disk partition → /tmp survives reboot
# Check if /dev/shm exists and what's in it RIGHT NOW
echo "=== /dev/shm CONTENTS ==="
ls -laR /dev/shm/
# Any files here are volatile and must be collected immediatelyMyth: "We should immediately shut down a compromised server to preserve evidence."
Reality: Shutting down a Linux server destroys all volatile evidence — running processes, memory contents, network connections, /dev/shm contents, and current login state. The correct first step is to collect volatile evidence, then decide whether to image live or power off. Immediate shutdown is the worst possible first response for a Linux investigation because the evidence destroyed by shutdown (memory, processes, connections) is often the most valuable for understanding what the attacker was doing at the moment of discovery. The only exception: active ransomware encryption where every second of uptime means more files encrypted.
Decision points: what determines collection urgency
The urgency of volatile collection depends on three factors:
Is the attacker currently active? If the attacker is logged in (visible in who or w), their processes and connections are live evidence that could disappear at any moment — if they detect the investigation and disconnect, their session state is lost. Collection urgency: immediate.
Is a reboot imminent? If the IT team is about to reboot, if a kernel panic is occurring, or if a cloud auto-healing mechanism is about to terminate the instance, you have minutes to collect volatile data. Collection urgency: critical.
Has the attacker deployed anti-forensic triggers? Some malware monitors for investigator activity (new SSH connections, specific commands) and triggers evidence destruction when detected. If you suspect this, memory acquisition should be the absolute first action — before any commands that the malware might intercept.
Troubleshooting: evidence already lost
The server was rebooted before you were called. All volatile evidence (memory, /proc, /dev/shm, network state, utmp) is permanently gone. Focus on persistent evidence: log files, filesystem timestamps, configuration files, disk image. The journal may retain events from before the reboot. wtmp survives reboots and shows the login history including the session that was active when the reboot occurred.
Log rotation ran overnight and deleted the oldest log file. The oldest rotated copy is gone. Check if the events exist in the systemd journal (separate retention). Check if logs were forwarded to a SIEM or remote syslog server. Check wtmp for authentication events that overlap the lost log period.
/dev/shm was collected but is empty. The attacker may have cleaned up before disconnecting, or the staging files may have been in /tmp instead. Check /tmp (may or may not be tmpfs). Check for evidence of cleanup in bash history or auditd.
Memory acquisition failed because LiME was not pre-compiled. Document the gap. Proceed to /proc enumeration as the best available volatile evidence. The /proc data is not as complete as a full memory dump (you cannot detect kernel rootkits, recover encryption keys, or analyze process memory) but it captures running processes, network connections, and open files.
Try it yourself
Check the log retention on a Linux system you manage.
Check the log retention on a Linux system you manage. Run the retention assessment commands from the block above. Calculate: based on the rotation policy and journal settings, how many days of log history do you have? If a compromise began 30 days ago, would you have the authentication evidence from day one? If not, where else would you look? Complete the retention assessment template for one of your systems — you are building the muscle memory for the first 5 minutes of every future investigation.
Beyond this investigation
The volatility hierarchy in this subsection defines the collection sequence you will follow in LX1 (Evidence Collection and Triage). Every scenario module (LX4–LX13) assumes evidence was collected in this order — volatile evidence first, persistent evidence second. When you reach LX12 (Memory Forensics), you will work with memory dumps that were acquired as the first step of collection, before any other investigation activity modified the system state.
Check your understanding:
1. A server was rebooted by the IT team before the security team was notified. Which evidence sources are now permanently lost, and which survived the reboot? 2. Log rotation deleted auth.log.4.gz overnight. What is the maximum age of authentication events you can recover from the remaining log files (assuming weekly rotation with 4 copies)? 3. Why is /dev/shm a preferred staging location for attackers, and what must an investigator do to collect evidence from it before it is lost? 4. You need to investigate a compromised production web server that cannot be taken offline. What collection approach do you use and in what order?
You have a 32GB memory dump from a compromised server. Volatility3 analysis will take approximately 4 hours. The IR lead needs a preliminary finding in 30 minutes. Do you skip memory analysis?
No — run targeted Volatility3 plugins that produce results in minutes, not the full analysis that takes hours. linux.pslist (process listing — 2 minutes) identifies suspicious processes immediately. linux.sockstat (network connections — 3 minutes) identifies active C2. linux.bash (bash history from memory — 1 minute) may capture commands the attacker executed even if they cleared .bash_history on disk. These 3 plugins produce a preliminary finding in under 10 minutes: 'Memory analysis identified [N] suspicious processes, [N] external network connections, and the following attacker commands.' The full analysis (loaded modules, rootkit detection, file extraction) continues while the preliminary finding reaches the IR lead.
Get weekly detection and investigation techniques
KQL queries, detection rules, and investigation methods — the same depth as this course, delivered every Tuesday.
No spam. Unsubscribe anytime. ~2,000 security practitioners.