LX1.5 The Collection Sequence

3-4 hours · Module 1 · Free

Putting It Together: The Complete Ordered Collection Procedure

Learning objective: Assemble the individual collection techniques from the preceding subsections into a single, ordered, end-to-end procedure that you can execute on any compromised Linux system. Understand the decision points — when to use the full sequence, when to abbreviate, and how the procedure changes across bare-metal, cloud, and container environments.

The Master Collection Sequence

Every Linux evidence collection follows the same logical order: most volatile evidence first, least volatile last. The specific commands vary by environment, but the sequence does not. What follows is the complete procedure, ordered and annotated.

This is the procedure you will follow — or delegate to a junior analyst to follow — on every Linux incident. Print it. Laminate it. Keep it in your IR toolkit alongside UAC and your forensic workstation. When the call comes at 03:00 and your hands are shaking from adrenaline, you do not want to be reconstructing this from memory.

Phase A: Pre-Collection (2 minutes)

Before touching the compromised system, prepare your collection infrastructure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# On your forensic workstation — create the case directory
export CASE="IR-2026-0402"
export TARGET="BASTION-NGE01"
mkdir -p ~/cases/$CASE/$TARGET/{volatile,logs,config,disk,memory,timeline,notes}

# Start your collection log
cat > ~/cases/$CASE/$TARGET/notes/collection_log.md << LOG
# Evidence Collection Log
# Case: $CASE
# Target: $TARGET
# Investigator: $(whoami)
# Collection started: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
# -----------------------------------------------
LOG

# Verify your forensic workstation's time is NTP-synced
timedatectl status | grep "synchronized"

Decision point 1: Can you access the system? If SSH access is available, proceed to Phase B. If the server is in a cloud environment and you have API access but not SSH, skip to Phase D (Cloud API Collection). If the system is a container, skip to Phase E (Container Collection). If the system is physical and you have no remote access, arrange physical access and skip to Phase F (Disk Imaging).

Phase B: Memory Acquisition (5–15 minutes)

Memory is the most volatile evidence. Acquire it before running any other commands on the system. Once you run ps, ss, or any other command, you have modified the memory state — processes are created, memory is allocated, caches are updated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Transfer LiME to the compromised system
# LiME must be compiled for the target's exact kernel version
scp lime-$(uname -r).ko investigator@target:/tmp/

# On the target — load LiME and dump memory to your workstation
ssh investigator@target "sudo insmod /tmp/lime-*.ko 'path=tcp:4444 format=lime'"

# On your workstation — receive the memory dump
nc target 4444 > ~/cases/$CASE/$TARGET/memory/memory.lime

# Hash immediately
sha256sum ~/cases/$CASE/$TARGET/memory/memory.lime > ~/cases/$CASE/$TARGET/memory/memory.lime.sha256

# Remove LiME from the target
ssh investigator@target "sudo rmmod lime"

If LiME is not available (you do not have a pre-compiled module for the target kernel), skip memory acquisition and proceed to Phase C. Document that memory was not acquired and why. You can still perform a partial memory analysis using /proc/kcore if the kernel exposes it, but this is limited compared to a full LiME dump.

If this is a cloud VM and the cloud provider offers hypervisor-level memory acquisition (rare — most do not expose this to customers), use that instead of LiME. It does not require loading a kernel module on the compromised system.

Phase C: Live Volatile Collection (10–20 minutes)

This is the live response sequence from the previous subsection, executed in order. If UAC is available, run UAC with the ir_triage profile first, then supplement with manual commands for anything UAC does not capture. If UAC is not available, run the manual commands.

Option 1: UAC triage (recommended)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Transfer UAC to the target
scp -r uac/ investigator@target:/tmp/uac/

# Run UAC — output piped back to your workstation
ssh investigator@target "cd /tmp/uac && sudo ./uac -p ir_triage -o /tmp/uac-output/"

# Copy UAC output to your workstation
scp -r investigator@target:/tmp/uac-output/ ~/cases/$CASE/$TARGET/volatile/uac/

# Hash the UAC output
find ~/cases/$CASE/$TARGET/volatile/uac/ -type f -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/volatile/uac_hashes.sha256

Option 2: Manual collection (if UAC unavailable)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
# Execute the 7-step live response sequence from subsection 02
# Pipe all output to your workstation

ssh investigator@target "date -u" > ~/cases/$CASE/$TARGET/volatile/timestamp.txt
ssh investigator@target "who; echo '---'; w" > ~/cases/$CASE/$TARGET/volatile/sessions.txt
ssh investigator@target "ps auxf" > ~/cases/$CASE/$TARGET/volatile/processes.txt
ssh investigator@target "ss -tlnp; echo '==='; ss -tnp; echo '==='; cat /proc/net/tcp" > ~/cases/$CASE/$TARGET/volatile/network.txt
ssh investigator@target "sudo lsof +L1" > ~/cases/$CASE/$TARGET/volatile/deleted_open_files.txt
ssh investigator@target "lsmod" > ~/cases/$CASE/$TARGET/volatile/kernel_modules.txt
ssh investigator@target "ls -laR /tmp/ /dev/shm/" > ~/cases/$CASE/$TARGET/volatile/staging_areas.txt
ssh investigator@target "find / -mtime -1 -type f -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null" > ~/cases/$CASE/$TARGET/volatile/recent_files.txt
ssh investigator@target "find / -perm -4000 -type f 2>/dev/null" > ~/cases/$CASE/$TARGET/volatile/suid_files.txt

# /proc deep collection — every process
ssh investigator@target 'for p in /proc/[0-9]*/; do echo "=== PID=$(basename $p) ==="; cat $p/cmdline 2>/dev/null | tr "\0" " "; echo; readlink -f $p/exe 2>/dev/null; echo "CWD: $(readlink -f $p/cwd 2>/dev/null)"; echo; done' > ~/cases/$CASE/$TARGET/volatile/proc_deep.txt

# Hash all volatile evidence
find ~/cases/$CASE/$TARGET/volatile/ -type f -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/volatile/hashes.sha256

After volatile collection — collect the attacker’s staging files. If you identified suspicious files in /tmp or /dev/shm during the volatile collection, copy them to your workstation now — before they are deleted or the system is rebooted:

1
2
3
4
5
6
# Copy suspicious files from staging areas
scp investigator@target:/dev/shm/.cache/worker ~/cases/$CASE/$TARGET/volatile/recovered_binary
scp investigator@target:/tmp/.hidden/payload.sh ~/cases/$CASE/$TARGET/volatile/recovered_script

# Hash recovered files
sha256sum ~/cases/$CASE/$TARGET/volatile/recovered_* >> ~/cases/$CASE/$TARGET/volatile/hashes.sha256

Phase D: Cloud API Collection (5–10 minutes)

For cloud VMs, collect the cloud-layer evidence that exists outside the VM.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# AWS example — adapt for Azure/GCP using commands from subsection 03

# Disk snapshot (no-login collection)
aws ec2 create-snapshot --volume-id vol-0abc123 \
  --description "$CASE forensic snapshot $TARGET" \
  > ~/cases/$CASE/$TARGET/disk/snapshot_response.json

# CloudTrail events
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-0abc123 \
  --start-time "2026-03-01T00:00:00Z" \
  --output json > ~/cases/$CASE/$TARGET/logs/cloudtrail.json

# Instance metadata (security group, IAM role, network config)
aws ec2 describe-instances --instance-ids i-0abc123 \
  --output json > ~/cases/$CASE/$TARGET/config/instance_metadata.json

# VPC Flow Logs (if available)
# [Commands depend on flow log configuration]

Phase E: Container Collection (3–5 minutes)

For containers, speed is paramount. Collect before the container restarts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Docker
docker inspect $CONTAINER > ~/cases/$CASE/$TARGET/config/container_inspect.json
docker logs $CONTAINER > ~/cases/$CASE/$TARGET/logs/container_stdout.txt 2>&1
docker diff $CONTAINER > ~/cases/$CASE/$TARGET/volatile/container_diff.txt
docker export $CONTAINER > ~/cases/$CASE/$TARGET/disk/container_filesystem.tar

# Kubernetes
kubectl describe pod $POD -n $NS > ~/cases/$CASE/$TARGET/config/pod_describe.txt
kubectl logs $POD -n $NS --all-containers > ~/cases/$CASE/$TARGET/logs/pod_logs.txt
kubectl get events -n $NS --field-selector involvedObject.name=$POD > ~/cases/$CASE/$TARGET/logs/k8s_events.txt

Phase F: Persistent Evidence Collection (15–30 minutes)

After volatile evidence is secured, collect the persistent evidence — log files, configuration, and user artifacts. These survive reboots and are less time-sensitive, but should be collected before log rotation runs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Log files — complete /var/log directory
scp -r investigator@target:/var/log/ ~/cases/$CASE/$TARGET/logs/var_log/

# System configuration
scp -r investigator@target:/etc/ ~/cases/$CASE/$TARGET/config/etc/

# User artifacts — all home directories
ssh investigator@target "sudo tar czf /tmp/home_dirs.tar.gz /home/ /root/" 
scp investigator@target:/tmp/home_dirs.tar.gz ~/cases/$CASE/$TARGET/config/
ssh investigator@target "rm /tmp/home_dirs.tar.gz"

# Cron jobs — all users
scp -r investigator@target:/var/spool/cron/ ~/cases/$CASE/$TARGET/config/cron/

# Systemd custom services
ssh investigator@target "ls /etc/systemd/system/*.service" > ~/cases/$CASE/$TARGET/config/custom_services.txt
scp investigator@target:/etc/systemd/system/*.service ~/cases/$CASE/$TARGET/config/systemd/ 2>/dev/null

# Hash all persistent evidence
find ~/cases/$CASE/$TARGET/logs/ ~/cases/$CASE/$TARGET/config/ -type f -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/persistent_hashes.sha256

Phase G: Disk Image (30–120 minutes)

If a full disk image is required (legal proceedings, comprehensive forensic analysis, or you need to recover deleted files), acquire it last — it is the slowest step and the evidence it captures is the least volatile.

1
2
3
4
5
6
# Live disk image over SSH (bare-metal or VM with SSH access)
ssh investigator@target "sudo dc3dd if=/dev/sda hash=sha256" | \
  dc3dd of=~/cases/$CASE/$TARGET/disk/disk.raw hash=sha256 log=~/cases/$CASE/$TARGET/disk/acquisition.log

# Cloud VM — use the snapshot created in Phase D
# Attach snapshot as volume to forensic VM, then image from there

Phase H: Post-Collection (5 minutes)

1
2
3
4
5
6
7
8
# Complete the collection log
echo "# Collection ended: $(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> ~/cases/$CASE/$TARGET/notes/collection_log.md

# Generate master hash manifest
find ~/cases/$CASE/$TARGET/ -type f -not -name "master_hashes.sha256" -exec sha256sum {} \; > ~/cases/$CASE/$TARGET/master_hashes.sha256

# Verify the evidence directory structure
tree ~/cases/$CASE/$TARGET/ > ~/cases/$CASE/$TARGET/notes/evidence_tree.txt

Adapting the Sequence

Time-critical / attacker active: Skip Phase B (memory) if LiME is not pre-compiled. Run Phase C with UAC ir_triage profile only. Skip Phase G (disk) initially. Get volatile evidence and begin analysis immediately. Return for comprehensive collection later.

Business-critical server / cannot go offline: Full sequence Phases A through F. Skip Phase G — do not image the disk while the server is serving production traffic. Use the cloud disk snapshot (Phase D) as your disk evidence instead.

Legal proceedings anticipated: Full sequence, all phases. Dual hashing (SHA256 + MD5). Witness present during collection. Evidence sealed in tamper-evident bags. Disk image with hardware write blocker if bare-metal.

Container environment: Phases A and E only. Container collection is self-contained — the docker export or kubectl cp commands capture the filesystem, and the container logs capture the application output. Add Phase D if the container runs on a cloud VM and you need the cloud audit trail.

Try it: Create the case directory structure on your forensic workstation: mkdir -p ~/cases/TEST-001/TEST-VM/{volatile,logs,config,disk,memory,timeline,notes}. Run the Phase C manual collection against a test VM (not production). Time yourself. The first time takes 20–30 minutes. By the third practice run, you will complete it in 10–12 minutes. Speed matters in incident response — practice the procedure before you need it.

Beyond This Investigation

This collection sequence is the starting point for every investigation scenario in this course. LX4 through LX13 all begin with “evidence has been collected from the compromised system.” The evidence they reference is the output of this procedure — the UAC triage data, the live response output, the log files, the disk image. When a scenario says “examine the process listing,” it means the processes.txt or UAC live_response/process/ output from Phase C. When it says “analyze the authentication logs,” it means the auth.log copies from Phase F.

Check your understanding:

Why is memory acquisition (Phase B) performed before live volatile collection (Phase C)?
The attacker is actively logged in and you see their session in who. Which phases do you prioritize, and which can you defer?
You are investigating a cloud VM but do not have SSH access — only AWS console access. Which phases can you still execute?
After completing the full collection sequence, how do you verify that no evidence files were modified during transfer to your forensic workstation?

You're reading the free modules of this course

The full course continues with advanced topics, production detection rules, worked investigation scenarios, and deployable artifacts. Premium subscribers get access to all courses.

View Pricing See Full Syllabus

← LX1.4 Evidence Integrity and Chain of Custody LX1.6 Memory Acquisition with LiME →