In this section

LX1.2 Live Response — Volatile Data Collection

3-4 hours · Module 1 · Free

Operational Objective

The Manual Collection Question: UAC is not always available. You SSH into WEBSRV-NGE01 and the SOC lead needs answers in 10 minutes — is the attacker still connected, what processes are they running, where are they sending data? You need a command sequence that captures volatile evidence in the correct order, reads from kernel data structures that rootkits cannot hide from, and produces output that you can immediately analyze. Every minute you spend on the wrong command is evidence that degrades or disappears.

Deliverable: A seven-step live response command sequence that captures volatile evidence from running processes through filesystem contents, including /proc direct reads that bypass userspace rootkits. The ability to detect hidden processes and hidden network connections by comparing userspace tool output against kernel data structures.

⏱ Estimated completion: 35 minutes

Live Response Commands: Capturing What Disappears

When you have shell access to a compromised Linux system and need to collect volatile evidence manually — either because UAC is not available, because you need immediate answers before running UAC, or because you need to verify UAC's output — the following command sequence captures the most critical volatile data in the correct order.

The principle: run the commands that reveal the attacker's current activity first. Process state and network connections tell you what the attacker is doing right now. Everything else tells you what the attacker did in the past. Current activity is the most volatile and the most immediately actionable.

Figure LX1.2 — The seven-step live response sequence ordered by volatility. Red-bordered steps capture the most transient evidence. The /proc direct read technique at Steps 3-4 bypasses userspace rootkits to reveal hidden processes and hidden connections.

Step 1: Record the Timestamp

# Record current time — UTC for consistency
date -u
# Also record the system's timezone configuration
cat /etc/timezone 2>/dev/null || timedatectl show --property=Timezone --value
# Record uptime — how long since last reboot
uptime

# Who is logged in right now?
who
# More detail — what are they running?
w
# Last 20 login sessions (from wtmp — binary, survives log tampering)
last -20
# Failed login attempts (from btmp)
sudo lastb -20

# Process tree — shows parent/child relationships
ps auxf

# Detailed process information with all fields
ps -eo pid,ppid,user,stat,%cpu,%mem,vsz,rss,tty,start_time,time,comm,args --sort=-%cpu

# For each process — read directly from /proc (bypasses rootkits)
for pid in /proc/[0-9]*/; do
  p=$(basename "$pid")
  echo "=== PID $p ==="
  echo "CMD: $(cat /proc/$p/cmdline 2>/dev/null | tr '\0' ' ')"
  echo "EXE: $(readlink -f /proc/$p/exe 2>/dev/null)"
  echo "CWD: $(readlink -f /proc/$p/cwd 2>/dev/null)"
  echo "USER: $(stat -c '%U' /proc/$p 2>/dev/null)"
  echo "---"
done

# All TCP connections — listening and established
ss -tlnp   # TCP listeners with process names
ss -tnp    # Established TCP connections with process names

# All UDP listeners
ss -ulnp

# Direct kernel read (bypasses rootkits)
cat /proc/net/tcp
cat /proc/net/tcp6
cat /proc/net/udp

# DNS resolver configuration
cat /etc/resolv.conf

# Routing table
ip route

# ARP cache — recently communicated hosts
ip neigh

# Firewall rules
sudo iptables-save 2>/dev/null
sudo nft list ruleset 2>/dev/null

# All open files — network, disk, and special
sudo lsof -i        # Network connections with file details
sudo lsof +D /tmp   # Open files in /tmp
sudo lsof +D /dev/shm  # Open files in /dev/shm

# Deleted files still held open by processes
sudo lsof +L1

# Currently loaded modules
lsmod

# Module details for suspicious modules
for mod in $(lsmod | awk 'NR>1 {print $1}'); do
  modinfo "$mod" 2>/dev/null | grep -E "^filename|^description|^author|^vermagic"
done

# Compare against known-good module list (if available)
# Modules loaded after the compromise timestamp are suspicious

# Contents of /tmp — attacker staging area
ls -la /tmp/
ls -laR /tmp/  # Recursive — check subdirectories

# Contents of /dev/shm — RAM-backed staging area
ls -la /dev/shm/
ls -laR /dev/shm/

# Recently modified files across the system (last 24 hours)
find / -mtime -1 -type f -not -path "/proc/*" -not -path "/sys/*" 2>/dev/null | head -100

# Files with SUID bit set (potential priv escalation)
find / -perm -4000 -type f 2>/dev/null

# World-writable directories (attacker staging locations)
find / -type d -perm -0002 -not -path "/proc/*" -not -path "/sys/*" 2>/dev/null

Expand for Deeper Context

Before any collection command, record the current system time in UTC. Every artifact you collect will be timestamped, and you need a reference point to determine whether the timestamps on the compromised system are accurate (the attacker may have changed the system clock).

If the system time is significantly different from your forensic workstation's time, note the offset. All subsequent timestamp analysis must account for clock skew. An offset of more than a few seconds suggests the attacker may have modified the system clock, or NTP has drifted — either way, it changes how you interpret every timestamp in the investigation.

Step 2: Currently Logged-In Users

The who output tells you whether the attacker is currently connected. If you see a session from an unexpected IP address, that is the attacker's live session. The w command adds what each session is currently running — if the attacker is running wget, curl, or python, you can see it in real time.

The last -20 command reads from /var/log/wtmp, a binary file that records login and logout events. Because it is binary, it is harder for an attacker to tamper with than plaintext log files (they cannot simply append or delete lines with sed — they need specialized tools). If the attacker has been logging in and out over several days, last shows the full session history including source IPs, session durations, and timestamps.

Step 3: Running Processes

The ps auxf output shows the process tree — critical for identifying suspicious parent/child relationships. A web server process (nginx, php-fpm, apache2) spawning a shell (/bin/bash, /bin/sh) is a strong indicator of web shell execution. A kernel thread name ([kworker/...]) appearing as a child of a user process is a disguised malicious process.

The /proc direct read loop is the most important live response command in your toolkit. It reads process information directly from the kernel's /proc filesystem, bypassing any userspace hooks that a rootkit might have installed. If ps shows 150 processes but the /proc loop finds 153, three processes are hidden by a rootkit.

The EXE field (readlink -f /proc/$p/exe) reveals the actual binary being executed, even if the attacker renamed the process or deleted the binary from disk. A process showing EXE: /dev/shm/.cache/worker (deleted) tells you: the binary was stored in RAM-backed /dev/shm, the attacker deleted it after launching it, and the binary is still in memory and can be recovered.

Step 4: Network Connections

The ss -tnp output shows every established TCP connection with the process that owns it. An established connection from a web server process to an external IP on port 4444, 8443, or 9001 is likely a reverse shell. A connection from any process to a known mining pool IP on port 3333, 4444, or 14444 is a cryptominer.

The /proc/net/tcp direct read is the kernel-level truth. Each line represents a TCP connection. The fields are hex-encoded (local address, remote address, state, PID). This data is harder to read than ss output but cannot be hidden by a userspace rootkit. LX12 covers the decoding in detail during memory forensics analysis, but during live response, the key technique is comparing the number of connections in /proc/net/tcp against the number shown by ss — a discrepancy indicates hidden connections.

The ARP cache (ip neigh) reveals hosts the compromised system has recently communicated with on the local network. If the attacker pivoted from this system to another internal host, the target IP will be in the ARP cache even if the attacker has cleaned up all other evidence of the connection.

Step 5: Open Files and File Descriptors

The lsof +L1 command is forensically significant. When a process opens a file and the file is then deleted from the filesystem, the file remains accessible through the process's file descriptor until the process closes it. An attacker who deletes their malware binary after launching it leaves the binary accessible through /proc/[pid]/fd/ — you can copy it out for analysis: cp /proc/[pid]/fd/[fd_number] /mnt/usb/recovered_binary.

This is one of the most powerful evidence recovery techniques in Linux IR. The attacker believes the file is gone. The filesystem confirms it is deleted. But the kernel still has it open, and you can extract a perfect copy.

Step 6: Loaded Kernel Modules

Loadable kernel modules (LKMs) are the mechanism for kernel-level rootkits on Linux. A rootkit module hooks system calls to hide processes, files, and network connections from userspace tools. If lsmod shows a module you do not recognize — especially one without a description or with a generic name — it warrants investigation. Note that a sophisticated rootkit may hide itself from lsmod as well, which is why memory forensics (LX12) is the definitive method for detecting kernel-level rootkits.

Step 7: Volatile Filesystem Contents

The find / -mtime -1 command identifies every file modified in the last 24 hours — a rapid scan for attacker activity. If the compromise occurred within the last day, this command surfaces the attacker's files alongside legitimate system changes. The SUID file list is a baseline for privilege escalation investigation in LX6. The world-writable directory list identifies additional staging locations beyond /tmp and /dev/shm.

Worked artifact — Live response output log template:

Record the output of each step during live response. This log becomes your primary evidence record when UAC was not available.

Case: INC-2026-XXXX System: [hostname] IP: [address]

Step 1 — Timestamp: - System time (UTC): ___ Workstation time (UTC): ___ Offset: ___ - Timezone config: ___ Uptime: ___

Step 2 — Active sessions: - Sessions from expected IPs: ___ (list) - Sessions from unexpected IPs: ___ (list) ← attacker sessions - Current activity (from w): ___

Step 3 — Processes: - Total via ps: ___ Total via /proc loop: ___ - Discrepancy: ☐ None ☐ Yes — ___ hidden processes detected - Suspicious parent/child: ___ (e.g., apache2 → /bin/sh) - Deleted binaries (/proc/PID/exe → "(deleted)"): ___

Step 4 — Network: - Connections via ss: ___ Connections via /proc/net/tcp: ___ - Discrepancy: ☐ None ☐ Yes — ___ hidden connections detected - Suspicious outbound: ___ (dest IP, port, process)

Step 5 — Open files: - Deleted files held open (lsof +L1): ___ - Recovered binaries: ☐ None ☐ Yes — recovered to ___

Step 6 — Kernel modules: Unrecognized modules: ___

Step 7 — Filesystem: Suspicious files in /tmp: ___ In /dev/shm: ___ SUID anomalies: ___

Decision points: when to use manual live response vs UAC

UAC is available and the attacker has disconnected: Run UAC. Manual live response adds no value when the automated tool is available and the attacker is no longer modifying the system in real time.

UAC is available but the attacker is still connected: Run the manual Steps 2-4 first (users, processes, network) to capture the attacker's current session and activity. Then run UAC for the comprehensive collection. The manual steps take 2 minutes and capture what the attacker is doing right now. UAC takes 3-8 minutes and captures everything else.

UAC is not available (container, locked-down environment, BusyBox): Manual live response is your only option. Run all seven steps in order. Pipe output to a file: script /tmp/live-response-$(date +%Y%m%d).log starts recording everything displayed on screen to a log file. Type exit when finished to stop recording.

Multiple systems, one analyst: Prioritize manual Steps 3-4 (processes and network) on all systems first to identify which systems have active attacker sessions. Then run full collection (UAC or manual) on the systems with active sessions before moving to systems where the attacker has disconnected.

Troubleshooting: common live response issues

ss is not installed (minimal/container environment). Fall back to cat /proc/net/tcp and cat /proc/net/tcp6 for the kernel-level data. Use netstat -tlnp as an alternative if available. If neither ss nor netstat exists, the /proc/net/ direct reads are your only option — they provide the same data in hex-encoded format.

lsof is not installed. Use /proc/[pid]/fd/ directly: ls -la /proc/[pid]/fd/ shows all open file descriptors for a process. To find deleted files held open: find /proc/*/fd -ls 2>/dev/null | grep "(deleted)". This achieves the same result as lsof +L1 using only /proc.

The /proc loop produces "Permission denied" errors. You are not running as root. Use sudo for the entire loop, or accept that you will only see your own processes. Non-root live response misses other users' processes, root-owned connections, and kernel module details — document this limitation in your evidence log.

Process tree shows nothing suspicious but you suspect compromise. The attacker may be using a rootkit that hides from both ps and the /proc loop (kernel-level hooking). At this point, live response cannot detect the attacker's processes. You need memory forensics (LX12) to examine the kernel's internal process list directly. Collect a memory dump with LiME (LX1.6) and analyze offline.

Output is too large to review on screen. Redirect all output to a file on external media: script -q /mnt/usb/live-response.log. The script command records all terminal output, including command prompts and results, to a file. Review the file on your forensic workstation rather than on the compromised system.

Beyond this investigation: LX4-LX10 use these exact live response commands in real investigation scenarios. LX12 (Memory Forensics) extends volatile data collection into full memory acquisition and analysis with Volatility 3.

Myth: "Running forensic commands on a live system contaminates the evidence and makes it inadmissible."

Reality: Live response is standard forensic practice and is accepted in legal proceedings worldwide. The key is documentation: record what commands you ran, when, and what they modified. The alternative — not collecting volatile evidence — results in permanent evidence loss. Courts and tribunals understand that some evidence modification is unavoidable during collection. What they require is that the modification is documented and the investigator can explain what changed. The SANS DFIR methodology, Mandiant's IR practice, and CrowdStrike's services team all perform live response as standard procedure. Not collecting volatile evidence because of theoretical contamination concerns is the error — not the collection itself.

Try it yourself

Exercise

On a test Linux VM (not a production system), run the full live response sequence from Steps 1-7. Save the output to a file: script /tmp/live-response-$(date +%Y%m%d).log. This records everything displayed on screen. After the collection, review the output: can you identify every running process? Can you identify every network connection? Are there any SUID files you do not recognize? Compare the process count from ps auxf | wc -l against ls /proc/[0-9]* -d | wc -l — are they the same? If they differ, investigate the discrepancy. This is the same review process you will perform on evidence from a compromised system.

Beyond this investigation

The live response commands in this subsection are the foundation for the "how to extract" step of the six-step investigation method. In LX4, you will run these commands on BASTION-NGE01 to identify the attacker's SSH session and post-compromise processes. In LX5, you will use the process tree to trace the web shell execution chain on WEBSRV-NGE01. In LX8, you will use the network connection commands to identify mining pool communications. In LX12, you will compare live response output against memory forensics output to detect rootkit-hidden processes and connections — the discrepancy technique described in Steps 3 and 4 becomes the foundation for rootkit confirmation.

Check your understanding:

1. Why should you read /proc/[pid]/exe directly instead of relying on the process name shown by ps? 2. You run ss -tnp and see 45 established connections. You run wc -l /proc/net/tcp and see 48 entries (minus the header). What does this discrepancy suggest, and what is your next step? 3. An attacker's binary at /dev/shm/.hidden/payload was deleted, but the process is still running as PID 9921. How do you recover the binary for analysis? 4. What does lsof +L1 show, and why is it forensically significant? 5. You are performing live response on a minimal Alpine Linux container. Neither ss nor lsof is installed. How do you collect network connection and open file evidence?

Decision point

You are investigating a Linux server and discover evidence of both a cryptominer (resource abuse) and an SSH key theft (lateral movement preparation). The cryptominer is consuming 95% CPU and impacting production. Which do you address first?

Address the lateral movement first. The cryptominer is visible, noisy, and contained to this server — it is causing performance impact but not spreading. The SSH key theft is silent, potentially already exploited, and may have given the attacker access to additional servers. Contain the lateral movement risk: rotate the stolen SSH keys, check the target servers for unauthorized access, and apply network restrictions. Then address the cryptominer: kill the process, remove the binary and persistence mechanisms. Prioritizing the noisy but contained threat over the silent but spreading threat is the most common Linux IR prioritization mistake.

Unlock the Full Course See Full Course Agenda

Get weekly detection and investigation techniques

KQL queries, detection rules, and investigation methods — the same depth as this course, delivered every Tuesday.

No spam. Unsubscribe anytime. ~2,000 security practitioners.

← Previous Next →