LX1.2 Live Response — Volatile Data Collection

Live Response Commands: Capturing What Disappears

The Live Response Sequence

When you have shell access to a compromised Linux system and need to collect volatile evidence manually — either because UAC is not available, because you need immediate answers before running UAC, or because you need to verify UAC’s output — the following command sequence captures the most critical volatile data in the correct order.

The principle: run the commands that reveal the attacker’s current activity first. Process state and network connections tell you what the attacker is doing right now. Everything else tells you what the attacker did in the past. Current activity is the most volatile and the most immediately actionable.

Step 1: Record the Timestamp

Before any collection command, record the current system time in UTC. Every artifact you collect will be timestamped, and you need a reference point to determine whether the timestamps on the compromised system are accurate (the attacker may have changed the system clock).

1
2
3
4
5
6

# Record current time — UTC for consistency
date -u
# Also record the system's timezone configuration
cat /etc/timezone 2>/dev/null || timedatectl show --property=Timezone --value
# Record uptime — how long since last reboot
uptime

If the system time is significantly different from your forensic workstation’s time, note the offset. All subsequent timestamp analysis must account for clock skew.

Step 2: Currently Logged-In Users

1
2
3
4
5
6
7
8

# Who is logged in right now?
who
# More detail — what are they running?
w
# Last 20 login sessions (from wtmp — binary, survives log tampering)
last -20
# Failed login attempts (from btmp)
sudo lastb -20

The who output tells you whether the attacker is currently connected. If you see a session from an unexpected IP address, that is the attacker’s live session. The w command adds what each session is currently running — if the attacker is running wget, curl, or python, you can see it in real time.

Step 3: Running Processes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Process tree — shows parent/child relationships
ps auxf

# Detailed process information with all fields
ps -eo pid,ppid,user,stat,%cpu,%mem,vsz,rss,tty,start_time,time,comm,args --sort=-%cpu

# For each process — read directly from /proc (bypasses rootkits)
for pid in /proc/[0-9]*/; do
  p=$(basename "$pid")
  echo "=== PID $p ==="
  echo "CMD: $(cat /proc/$p/cmdline 2>/dev/null | tr '\0' ' ')"
  echo "EXE: $(readlink -f /proc/$p/exe 2>/dev/null)"
  echo "CWD: $(readlink -f /proc/$p/cwd 2>/dev/null)"
  echo "USER: $(stat -c '%U' /proc/$p 2>/dev/null)"
  echo "---"
done

The ps auxf output shows the process tree — critical for identifying suspicious parent/child relationships. A web server process (nginx, php-fpm, apache2) spawning a shell (/bin/bash, /bin/sh) is a strong indicator of web shell execution. A kernel thread name ([kworker/...]) appearing as a child of a user process is a disguised malicious process.

The /proc direct read loop is the most important live response command in your toolkit. It reads process information directly from the kernel’s /proc filesystem, bypassing any userspace hooks that a rootkit might have installed. If ps shows 150 processes but the /proc loop finds 153, three processes are hidden by a rootkit.

The EXE field (readlink -f /proc/$p/exe) reveals the actual binary being executed, even if the attacker renamed the process or deleted the binary from disk. A process showing EXE: /dev/shm/.cache/worker (deleted) tells you: the binary was stored in RAM-backed /dev/shm, the attacker deleted it after launching it, and the binary is still in memory and can be recovered.

Step 4: Network Connections

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# All TCP connections — listening and established
ss -tlnp   # TCP listeners with process names
ss -tnp    # Established TCP connections with process names

# All UDP listeners
ss -ulnp

# Direct kernel read (bypasses rootkits)
cat /proc/net/tcp
cat /proc/net/tcp6
cat /proc/net/udp

# DNS resolver configuration
cat /etc/resolv.conf

# Routing table
ip route

# ARP cache — recently communicated hosts
ip neigh

# Firewall rules
sudo iptables-save 2>/dev/null
sudo nft list ruleset 2>/dev/null

The ss -tnp output shows every established TCP connection with the process that owns it. An established connection from a web server process to an external IP on port 4444, 8443, or 9001 is likely a reverse shell. A connection from any process to a known mining pool IP on port 3333, 4444, or 14444 is a cryptominer.

The /proc/net/tcp direct read is the kernel-level truth. Each line represents a TCP connection. The fields are hex-encoded (local address, remote address, state, PID). This data is harder to read than ss output but cannot be hidden by a userspace rootkit. LX12 covers the decoding in detail during memory forensics analysis, but during live response, the key technique is comparing the number of connections in /proc/net/tcp against the number shown by ss — a discrepancy indicates hidden connections.

Step 5: Open Files and File Descriptors

1
2
3
4
5
6
7

# All open files — network, disk, and special
sudo lsof -i        # Network connections with file details
sudo lsof +D /tmp   # Open files in /tmp
sudo lsof +D /dev/shm  # Open files in /dev/shm

# Deleted files still held open by processes
sudo lsof +L1

The lsof +L1 command is forensically significant. When a process opens a file and the file is then deleted from the filesystem, the file remains accessible through the process’s file descriptor until the process closes it. An attacker who deletes their malware binary after launching it leaves the binary accessible through /proc/[pid]/fd/ — you can copy it out for analysis: cp /proc/[pid]/fd/[fd_number] /mnt/usb/recovered_binary.

Step 6: Loaded Kernel Modules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Currently loaded modules
lsmod

# Module details for suspicious modules
for mod in $(lsmod | awk 'NR>1 {print $1}'); do
  modinfo "$mod" 2>/dev/null | grep -E "^filename|^description|^author|^vermagic"
done

# Compare against known-good module list (if available)
# Modules loaded after the compromise timestamp are suspicious

Loadable kernel modules (LKMs) are the mechanism for kernel-level rootkits on Linux. A rootkit module hooks system calls to hide processes, files, and network connections from userspace tools. If lsmod shows a module you do not recognize — especially one without a description or with a generic name — it warrants investigation. Note that a sophisticated rootkit may hide itself from lsmod as well, which is why memory forensics (LX12) is the definitive method for detecting kernel-level rootkits.

Step 7: Volatile Filesystem Contents

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Contents of /tmp — attacker staging area
ls -la /tmp/
ls -laR /tmp/  # Recursive — check subdirectories

# Contents of /dev/shm — RAM-backed staging area
ls -la /dev/shm/
ls -laR /dev/shm/

# Recently modified files across the system (last 24 hours)
find / -mtime -1 -type f -not -path "/proc/*" -not -path "/sys/*" 2>/dev/null | head -100

# Files with SUID bit set (potential priv escalation)
find / -perm -4000 -type f 2>/dev/null

# World-writable directories (attacker staging locations)
find / -type d -perm -0002 -not -path "/proc/*" -not -path "/sys/*" 2>/dev/null

The find / -mtime -1 command identifies every file modified in the last 24 hours — a rapid scan for attacker activity. If the compromise occurred within the last day, this command surfaces the attacker’s files alongside legitimate system changes. The SUID file list is a baseline for privilege escalation investigation in LX6. The world-writable directory list identifies additional staging locations beyond /tmp and /dev/shm.

Beyond This Investigation

The live response commands in this subsection are the foundation for the “how to extract” step of the six-step investigation method. In LX4, you will run these commands on BASTION-NGE01 to identify the attacker’s SSH session and post-compromise processes. In LX5, you will use the process tree to trace the web shell execution chain. In LX8, you will use the network connection commands to identify mining pool communications. In LX12, you will compare live response output against memory forensics output to detect rootkit-hidden processes and connections.