In this section
LX1.8 Container and Kubernetes Evidence Collection
Container Evidence Collection: Racing the Restart Clock
Why Container Collection Is Different
Container evidence collection violates the fundamental assumption of forensics: that evidence persists until collected. On a bare-metal server, evidence exists on a physical disk that survives reboots, power cycles, and — if you do nothing — indefinitely. On a container, the writable filesystem layer exists only while the container is running. When a Kubernetes pod is terminated and rescheduled, the new pod starts from the pristine base image with no trace of what happened in the previous instance. The attacker's web shell, the modified configuration files, the bash history, the downloaded tools — all erased by a container restart.
This creates an urgency that does not exist in other investigation environments. On a bare-metal server, you can wait hours before beginning collection — the evidence is stable. On a container, you may have minutes before a liveness probe failure, an autoscaler decision, or a manual restart destroys the evidence.
Docker Container Collection — Complete Workflow
# Step 1: Confirm the container is still running
docker ps --filter name=compromised-app
# If the container is not running, skip to "Dead Container Recovery"
# Step 2: Capture container metadata (configuration, env vars, mounts)
docker inspect compromised-app > evidence/docker_inspect.json
# Step 3: Capture container logs (stdout/stderr — application output)
docker logs --timestamps compromised-app > evidence/docker_logs.txt 2>&1
# Step 4: Identify what the attacker changed (diff from base image)
docker diff compromised-app > evidence/docker_diff.txt
# Output: A = added, C = changed, D = deleted
# Every entry is a file the attacker created or modified
# Step 5: Export the complete container filesystem
docker export compromised-app > evidence/container_filesystem.tar
# This captures the merged view — base image + writable layer
# Step 6: Copy specific directories for targeted analysis
docker cp compromised-app:/var/log/ evidence/container_var_log/
docker cp compromised-app:/tmp/ evidence/container_tmp/
docker cp compromised-app:/etc/crontab evidence/container_crontab 2>/dev/null
docker cp compromised-app:/root/.bash_history evidence/container_bash_history 2>/dev/null
# Step 7: Capture the running process state inside the container
docker exec compromised-app ps auxf > evidence/container_processes.txt 2>/dev/null
docker exec compromised-app ss -tlnp > evidence/container_network.txt 2>/dev/null
docker exec compromised-app cat /etc/resolv.conf > evidence/container_dns.txt 2>/dev/null
# Step 8: Check for container escape indicators
docker exec compromised-app ls -la /var/run/docker.sock 2>/dev/null && \
echo "WARNING: Docker socket mounted inside container" > evidence/escape_indicator.txt
docker exec compromised-app cat /proc/1/cgroup > evidence/container_cgroup.txt 2>/dev/null# Pod-level evidence
kubectl describe pod suspicious-pod -n production > evidence/pod_describe.txt
kubectl logs suspicious-pod -n production --all-containers > evidence/pod_logs.txt 2>&1
kubectl logs suspicious-pod -n production --previous > evidence/pod_logs_previous.txt 2>/dev/null
kubectl cp production/suspicious-pod:/var/log/ evidence/pod_var_log/
kubectl cp production/suspicious-pod:/tmp/ evidence/pod_tmp/
# Exec into the pod for live volatile collection
kubectl exec -it suspicious-pod -n production -- ps auxf > evidence/pod_processes.txt 2>/dev/null
kubectl exec -it suspicious-pod -n production -- ss -tlnp > evidence/pod_network.txt 2>/dev/null
kubectl exec -it suspicious-pod -n production -- cat /proc/1/cgroup > evidence/pod_cgroup.txt 2>/dev/null
# Cluster-level evidence
kubectl get events -n production --field-selector involvedObject.name=suspicious-pod --sort-by='.lastTimestamp' > evidence/k8s_events.txt
kubectl get pod suspicious-pod -n production -o yaml > evidence/pod_yaml.txt
# Service account token (was it stolen for RBAC abuse?)
kubectl exec suspicious-pod -n production -- cat /var/run/secrets/kubernetes.io/serviceaccount/token > evidence/sa_token.txt 2>/dev/null
# Kubernetes audit log (if enabled — cluster admin must provide access)
# Location varies: /var/log/kubernetes/audit.log, or cloud provider's log serviceMyth: "Container forensics is impossible because containers are ephemeral — there is nothing to investigate."
Reality: Container ephemerality makes forensics urgent, not impossible. A running container contains the complete filesystem, process state, network connections, and logs — the same evidence as a bare-metal server. The constraint is time, not capability. Additionally, the docker diff command provides evidence that is actually cleaner than bare-metal forensics: because the base image is known-good, every change in the diff is a deviation that must be explained. Containers also produce orchestrator-level evidence (Kubernetes events, audit logs, service account activity) that has no equivalent in bare-metal environments.
Try it yourself
Practice the complete Docker collection workflow.
Practice the complete Docker collection workflow. Run a test container: docker run -d --name forensic-test nginx:latest. Make a change inside it: docker exec forensic-test bash -c "echo 'test-evidence' > /tmp/attacker-file.txt". Now run the full collection sequence from Step 1–8 above. Examine the docker diff output — you should see /tmp/attacker-file.txt as an Added file. Export the filesystem and verify you can find the file in the tar archive: tar tf evidence/container_filesystem.tar | grep attacker. Clean up: docker rm -f forensic-test.
Beyond This Investigation
Container evidence collection is the foundation for LX9 (Container Compromise), which investigates a complete container breach scenario including escape to the host and lateral movement through the Kubernetes API. The collection techniques in this subsection capture the evidence that LX9's analysis examines in depth.
Check your understanding:
1. A compromised container was restarted by Kubernetes 5 minutes ago. What evidence from the previous container instance is still accessible? 2. What does the output of docker diff represent, and why is it more useful for container forensics than a full filesystem listing? 3. You find /var/run/docker.sock mounted inside a compromised container. What can the attacker do with this, and what evidence should you look for on the host? 4. The Kubernetes audit log shows create pod and get secrets API calls using the compromised pod's service account token. What does this indicate about the scope of the compromise?
You arrive at a compromised Linux server. The server is still running and the attacker may still be active. Do you collect volatile evidence first or image the disk?
Volatile evidence first — always. Memory contents, running processes (ps auxf), network connections (ss -tlnp), logged-in users (who), and kernel modules (lsmod) exist only while the system is running. Disk evidence survives a reboot; volatile evidence does not. The collection sequence: LiME memory dump → process state → network state → user sessions → then proceed to disk evidence. If the attacker is active, the volatile evidence captures their current operations — processes, connections, and tools that disappear the moment the system is powered off or the attacker disconnects.
Get weekly detection and investigation techniques
KQL queries, detection rules, and investigation methods — the same depth as this course, delivered every Tuesday.
No spam. Unsubscribe anytime. ~2,000 security practitioners.