In this section
LX1.5 The Collection Sequence
Putting It Together: The Complete Ordered Collection Procedure
The Master Collection Sequence
Every Linux evidence collection follows the same logical order: most volatile evidence first, least volatile last. The specific commands vary by environment, but the sequence does not. What follows is the complete procedure, ordered and annotated.
This is the procedure you will follow — or delegate to a junior analyst to follow — on every Linux incident. Print it. Laminate it. Keep it in your IR toolkit alongside UAC and your forensic workstation. When the call comes at 03:00 and your hands are shaking from adrenaline, you do not want to be reconstructing this from memory.
Phase A: Pre-Collection (2 minutes)
# On your forensic workstation — create the case directory
export CASE="IR-2026-0402"
export TARGET="BASTION-NGE01"
mkdir -p ~/cases/$CASE/$TARGET/{volatile,logs,config,disk,memory,timeline,notes}
# Start your collection log
cat > ~/cases/$CASE/$TARGET/notes/collection_log.md << LOG
# Evidence Collection Log
# Case: $CASE
# Target: $TARGET
# Investigator: $(whoami)
# Collection started: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
# -----------------------------------------------
LOG
# Verify your forensic workstation's time is NTP-synced
timedatectl status | grep "synchronized"# Transfer LiME to the compromised system
# LiME must be compiled for the target's exact kernel version
scp lime-$(uname -r).ko investigator@target:/tmp/
# On the target — load LiME and dump memory to your workstation
ssh investigator@target "sudo insmod /tmp/lime-*.ko 'path=tcp:4444 format=lime'"
# On your workstation — receive the memory dump
nc target 4444 > ~/cases/$CASE/$TARGET/memory/memory.lime
# Hash immediately
sha256sum ~/cases/$CASE/$TARGET/memory/memory.lime > ~/cases/$CASE/$TARGET/memory/memory.lime.sha256
# Remove LiME from the target
ssh investigator@target "sudo rmmod lime"# Transfer UAC to the target
scp -r uac/ investigator@target:/tmp/uac/
# Run UAC — output piped back to your workstation
ssh investigator@target "cd /tmp/uac && sudo ./uac -p ir_triage -o /tmp/uac-output/"
# Copy UAC output to your workstation
scp -r investigator@target:/tmp/uac-output/ ~/cases/$CASE/$TARGET/volatile/uac/
# Hash the UAC output
find ~/cases/$CASE/$TARGET/volatile/uac/ -type f -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/volatile/uac_hashes.sha256# Execute the 7-step live response sequence from subsection 02
# Pipe all output to your workstation
ssh investigator@target "date -u" > ~/cases/$CASE/$TARGET/volatile/timestamp.txt
ssh investigator@target "who; echo '---'; w" > ~/cases/$CASE/$TARGET/volatile/sessions.txt
ssh investigator@target "ps auxf" > ~/cases/$CASE/$TARGET/volatile/processes.txt
ssh investigator@target "ss -tlnp; echo '==='; ss -tnp; echo '==='; cat /proc/net/tcp" > ~/cases/$CASE/$TARGET/volatile/network.txt
ssh investigator@target "sudo lsof +L1" > ~/cases/$CASE/$TARGET/volatile/deleted_open_files.txt
ssh investigator@target "lsmod" > ~/cases/$CASE/$TARGET/volatile/kernel_modules.txt
ssh investigator@target "ls -laR /tmp/ /dev/shm/" > ~/cases/$CASE/$TARGET/volatile/staging_areas.txt
ssh investigator@target "find / -mtime -1 -type f -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null" > ~/cases/$CASE/$TARGET/volatile/recent_files.txt
ssh investigator@target "find / -perm -4000 -type f 2>/dev/null" > ~/cases/$CASE/$TARGET/volatile/suid_files.txt
# /proc deep collection — every process
ssh investigator@target 'for p in /proc/[0-9]*/; do echo "=== PID=$(basename $p) ==="; cat $p/cmdline 2>/dev/null | tr "\0" " "; echo; readlink -f $p/exe 2>/dev/null; echo "CWD: $(readlink -f $p/cwd 2>/dev/null)"; echo; done' > ~/cases/$CASE/$TARGET/volatile/proc_deep.txt
# Hash all volatile evidence
find ~/cases/$CASE/$TARGET/volatile/ -type f -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/volatile/hashes.sha256# Copy suspicious files from staging areas
scp investigator@target:/dev/shm/.cache/worker ~/cases/$CASE/$TARGET/volatile/recovered_binary
scp investigator@target:/tmp/.hidden/payload.sh ~/cases/$CASE/$TARGET/volatile/recovered_script
# Hash recovered files
sha256sum ~/cases/$CASE/$TARGET/volatile/recovered_* >> ~/cases/$CASE/$TARGET/volatile/hashes.sha256# AWS example — adapt for Azure/GCP using commands from subsection 03
# Disk snapshot (no-login collection)
aws ec2 create-snapshot --volume-id vol-0abc123 \
--description "$CASE forensic snapshot $TARGET" \
> ~/cases/$CASE/$TARGET/disk/snapshot_response.json
# CloudTrail events
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue=i-0abc123 \
--start-time "2026-03-01T00:00:00Z" \
--output json > ~/cases/$CASE/$TARGET/logs/cloudtrail.json
# Instance metadata (security group, IAM role, network config)
aws ec2 describe-instances --instance-ids i-0abc123 \
--output json > ~/cases/$CASE/$TARGET/config/instance_metadata.json
# VPC Flow Logs (if available)
# [Commands depend on flow log configuration]# Docker
docker inspect $CONTAINER > ~/cases/$CASE/$TARGET/config/container_inspect.json
docker logs $CONTAINER > ~/cases/$CASE/$TARGET/logs/container_stdout.txt 2>&1
docker diff $CONTAINER > ~/cases/$CASE/$TARGET/volatile/container_diff.txt
docker export $CONTAINER > ~/cases/$CASE/$TARGET/disk/container_filesystem.tar
# Kubernetes
kubectl describe pod $POD -n $NS > ~/cases/$CASE/$TARGET/config/pod_describe.txt
kubectl logs $POD -n $NS --all-containers > ~/cases/$CASE/$TARGET/logs/pod_logs.txt
kubectl get events -n $NS --field-selector involvedObject.name=$POD > ~/cases/$CASE/$TARGET/logs/k8s_events.txt# Log files — complete /var/log directory
scp -r investigator@target:/var/log/ ~/cases/$CASE/$TARGET/logs/var_log/
# System configuration
scp -r investigator@target:/etc/ ~/cases/$CASE/$TARGET/config/etc/
# User artifacts — all home directories
ssh investigator@target "sudo tar czf /tmp/home_dirs.tar.gz /home/ /root/"
scp investigator@target:/tmp/home_dirs.tar.gz ~/cases/$CASE/$TARGET/config/
ssh investigator@target "rm /tmp/home_dirs.tar.gz"
# Cron jobs — all users
scp -r investigator@target:/var/spool/cron/ ~/cases/$CASE/$TARGET/config/cron/
# Systemd custom services
ssh investigator@target "ls /etc/systemd/system/*.service" > ~/cases/$CASE/$TARGET/config/custom_services.txt
scp investigator@target:/etc/systemd/system/*.service ~/cases/$CASE/$TARGET/config/systemd/ 2>/dev/null
# Hash all persistent evidence
find ~/cases/$CASE/$TARGET/logs/ ~/cases/$CASE/$TARGET/config/ -type f -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/persistent_hashes.sha256# Live disk image over SSH (bare-metal or VM with SSH access)
ssh investigator@target "sudo dc3dd if=/dev/sda hash=sha256" | \
dc3dd of=~/cases/$CASE/$TARGET/disk/disk.raw hash=sha256 log=~/cases/$CASE/$TARGET/disk/acquisition.log
# Cloud VM — use the snapshot created in Phase D
# Attach snapshot as volume to forensic VM, then image from there# Complete the collection log
echo "# Collection ended: $(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> ~/cases/$CASE/$TARGET/notes/collection_log.md
# Generate master hash manifest
find ~/cases/$CASE/$TARGET/ -type f -not -name "master_hashes.sha256" -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/master_hashes.sha256
# Verify the evidence directory structure
tree ~/cases/$CASE/$TARGET/ > ~/cases/$CASE/$TARGET/notes/evidence_tree.txtMyth: "You must follow every phase of the collection procedure in order, or the evidence is compromised."
Reality: The sequence represents the optimal order for maximum evidence preservation — most volatile first, least volatile last. But real incidents rarely allow the optimal path. If the attacker is actively exfiltrating data, you may jump directly to Phase C (volatile collection) to capture their activity, skipping Phase B (memory) entirely. If you only have cloud API access and no SSH, you skip Phases B, C, and F and rely on the disk snapshot and audit trail. The sequence is a framework, not a rigid script. What matters is that you document what you collected, what you skipped, and why. An investigation with Phases A, C, D, and H (skipping memory, persistent collection, and disk imaging) is a valid investigation if you document the decisions and their rationale.
Try it yourself
Exercise
Create the case directory structure on your forensic workstation: mkdir -p ~/cases/TEST-001/TEST-VM/{volatile,logs,config,disk,memory,timeline,notes}. Run the Phase C manual collection against a test VM (not production). Time yourself. The first time takes 20–30 minutes. By the third practice run, you will complete it in 10–12 minutes. Speed matters in incident response — practice the procedure before you need it.
Beyond This Investigation
This collection sequence is the starting point for every investigation scenario in this course. LX4 through LX13 all begin with "evidence has been collected from the compromised system." The evidence they reference is the output of this procedure — the UAC triage data, the live response output, the log files, the disk image. When a scenario says "examine the process listing," it means the processes.txt or UAC live_response/process/ output from Phase C. When it says "analyze the authentication logs," it means the auth.log copies from Phase F.
Check your understanding:
1. Why is memory acquisition (Phase B) performed before live volatile collection (Phase C)? 2. The attacker is actively logged in and you see their session in who. Which phases do you prioritize, and which can you defer? 3. You are investigating a cloud VM but do not have SSH access — only AWS console access. Which phases can you still execute? 4. After completing the full collection sequence, how do you verify that no evidence files were modified during transfer to your forensic workstation?
Lab Exercise: LX01 — Evidence Collection
If you have the Linux IR Lab Pack installed, run the generator and practice the full evidence collection workflow: volatile state capture, log collection, filesystem timeline analysis, bash history mapping, auditd correlation, and chain of custody documentation. The generator creates ~3,500 lines of auth.log with attack indicators buried in 7 days of legitimate noise.
Verify: ./verification/Verify-LX01.sh [your-output-dir]
You are investigating a Linux server and discover evidence of both a cryptominer (resource abuse) and an SSH key theft (lateral movement preparation). The cryptominer is consuming 95% CPU and impacting production. Which do you address first?
Address the lateral movement first. The cryptominer is visible, noisy, and contained to this server — it is causing performance impact but not spreading. The SSH key theft is silent, potentially already exploited, and may have given the attacker access to additional servers. Contain the lateral movement risk: rotate the stolen SSH keys, check the target servers for unauthorized access, and apply network restrictions. Then address the cryptominer: kill the process, remove the binary and persistence mechanisms. Prioritizing the noisy but contained threat over the silent but spreading threat is the most common Linux IR prioritization mistake.
Get weekly detection and investigation techniques
KQL queries, detection rules, and investigation methods — the same depth as this course, delivered every Tuesday.
No spam. Unsubscribe anytime. ~2,000 security practitioners.