In this section
LX1.3 Remote and Cloud-Specific Collection
Remote Collection, Cloud Snapshots, and Container Evidence
Every environment where Linux runs has a different evidence surface. A bare-metal server has a physical disk you can image — but only if you are physically present or have remote console access. A cloud VM has an API that can snapshot the disk without logging in — but the snapshot does not capture memory or running processes. A container has a filesystem diff that shows exactly what changed from the base image — but the container can be destroyed at any moment and the evidence disappears with it.
The investigator must know all three collection methods and choose the right one based on the environment, the urgency, and what evidence is needed.
SSH-Based Remote Collection
# From your forensic workstation — collect volatile data first
# Each command runs on the remote system; output streams to local files
# Collect running processes
ssh investigator@target "ps auxf" > evidence/processes.txt
# Collect network connections (userspace + kernel)
ssh investigator@target "ss -tnp && echo '---' && cat /proc/net/tcp" > evidence/network.txt
# Collect /proc data for all processes (bypasses rootkits)
ssh investigator@target "for p in /proc/[0-9]*/; do echo PID=\$(basename \$p); cat \$p/cmdline 2>/dev/null | tr '\0' ' '; echo; readlink -f \$p/exe 2>/dev/null; echo ---; done" > evidence/proc_data.txt
# Copy critical log files to your workstation
scp investigator@target:/var/log/auth.log* evidence/logs/
scp investigator@target:/var/log/syslog* evidence/logs/
scp -r investigator@target:/var/log/journal/ evidence/logs/journal/
# Copy UAC output (if you ran UAC on the target)
scp -r investigator@target:/tmp/uac-output/ evidence/uac/# Stream entire /var/log as a tarball — preserves timestamps and permissions
ssh investigator@target "sudo tar czf - /var/log/" > evidence/var_log.tar.gz
# Stream /etc for full configuration capture
ssh investigator@target "sudo tar czf - /etc/" > evidence/etc.tar.gz
# Verify the tarball integrity after transfer
tar tzf evidence/var_log.tar.gz | wc -l # Count files — sanity check# Get the instance ID and volume ID
aws ec2 describe-instances \
--filters "Name=tag:Name,Values=WEBSRV-NGE01" \
--query "Reservations[].Instances[].{ID:InstanceId,Vols:BlockDeviceMappings[].Ebs.VolumeId}" \
--output table
# Create a snapshot of the root volume
aws ec2 create-snapshot \
--volume-id vol-0abc123def456789 \
--description "IR-2026-0402 forensic snapshot WEBSRV-NGE01" \
--tag-specifications "ResourceType=snapshot,Tags=[{Key=Case,Value=IR-2026-0402},{Key=Investigator,Value=j.morrison}]"
# Monitor snapshot progress
aws ec2 describe-snapshots --snapshot-ids snap-0abc123 \
--query "Snapshots[].{State:State,Progress:Progress}"
# Collect CloudTrail events for the instance (30 days)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue=i-0abc123 \
--start-time "2026-03-03T00:00:00Z" \
--end-time "2026-04-02T23:59:59Z" \
--output json > evidence/cloudtrail_events.json
# Collect VPC Flow Logs (if enabled)
aws ec2 describe-flow-logs \
--filter "Name=resource-id,Values=eni-0abc123" \
--output json > evidence/flow_log_config.json# Create a snapshot of the OS disk
az snapshot create \
--name "IR-2026-0402-WEBSRV-NGE01-osdisk" \
--resource-group NorthgateEng-RG \
--source "/subscriptions/SUB_ID/resourceGroups/NorthgateEng-RG/providers/Microsoft.Compute/disks/WEBSRV-NGE01-osdisk" \
--tags Case=IR-2026-0402 Investigator=j.morrison
# Collect Azure Activity Log (90 days retention)
az monitor activity-log list \
--resource-id "/subscriptions/SUB_ID/resourceGroups/NorthgateEng-RG/providers/Microsoft.Compute/virtualMachines/WEBSRV-NGE01" \
--start-time "2026-03-03T00:00:00Z" \
--output json > evidence/azure_activity_log.json
# Collect NSG Flow Logs (if enabled)
az network watcher flow-log show \
--resource-group NorthgateEng-RG \
--nsg WEBSRV-NGE01-nsg \
--output json > evidence/nsg_flow_log_config.json# Create a snapshot
gcloud compute disks snapshot WEBSRV-NGE01-disk \
--zone=europe-west2-a \
--snapshot-names=ir-2026-0402-websrv-nge01 \
--description="IR-2026-0402 forensic snapshot"
# Collect Cloud Audit Logs
gcloud logging read \
'resource.type="gce_instance" AND resource.labels.instance_id="123456789"' \
--limit=1000 --format=json > evidence/gcp_audit_log.json# Is the container still running?
docker ps | grep suspicious-container
# Collect container metadata (image, config, network, mounts)
docker inspect suspicious-container > evidence/container_inspect.json
# Collect container logs (stdout/stderr)
docker logs suspicious-container > evidence/container_logs.txt 2>&1
# See what files changed from the base image
docker diff suspicious-container > evidence/container_diff.txt
# Export the full container filesystem
docker export suspicious-container > evidence/container_filesystem.tar
# Copy specific files out of the container
docker cp suspicious-container:/var/log/ evidence/container_logs/
docker cp suspicious-container:/tmp/ evidence/container_tmp/
docker cp suspicious-container:/etc/crontab evidence/container_crontab# Is the pod still running?
kubectl get pods -n production | grep suspicious-pod
# Collect pod metadata (containers, volumes, status, events)
kubectl describe pod suspicious-pod -n production > evidence/pod_describe.txt
# Collect pod logs (all containers in the pod)
kubectl logs suspicious-pod -n production --all-containers > evidence/pod_logs.txt
# Previous pod logs (if pod restarted)
kubectl logs suspicious-pod -n production --previous > evidence/pod_logs_previous.txt 2>/dev/null
# Copy files out of the running pod
kubectl cp production/suspicious-pod:/var/log/ evidence/pod_logs/
kubectl cp production/suspicious-pod:/tmp/ evidence/pod_tmp/
# Collect Kubernetes events for the namespace
kubectl get events -n production --field-selector involvedObject.name=suspicious-pod > evidence/k8s_events.txt
# Collect Kubernetes audit log (if enabled — check with cluster admin)
# Location varies by cluster configurationMyth: "A cloud disk snapshot is not real forensic evidence because you did not use a hardware write blocker."
Reality: The concept of a hardware write blocker applies to physical media — connecting a hard drive to a forensic workstation. Cloud disk snapshots operate at the storage API level, not the physical level. The snapshot is created by the cloud provider's storage service and is immutable once created — it cannot be modified after the fact. Courts and forensic practitioners accept cloud snapshots as valid evidence when the investigator documents the snapshot creation (timestamp, snapshot ID, who created it) and the snapshot's immutability is preserved (no delete or modification operations performed on the snapshot). The chain of custody for a cloud snapshot is the cloud audit trail itself — the CreateSnapshot API call recorded in CloudTrail, Activity Log, or Cloud Audit Logs is the documentation.
Try it yourself
Exercise
If you have AWS CLI configured, practice the snapshot workflow on a test instance: create a snapshot, wait for it to complete, then create a volume from the snapshot and attach it to a forensic VM. Mount it read-only: mount -o ro,noexec /dev/xvdf1 /mnt/evidence/. You have just performed the no-login disk collection workflow for a cloud VM. If you use Docker, practice the container collection workflow: docker run -d --name test-container nginx, then run the docker inspect, docker diff, docker logs, and docker export commands above. Examine the diff output — what files changed from the base nginx image just by the container running normally?
Beyond this investigation
Remote SSH collection is the default for LX4-LX8 scenarios where the target is a traditional server. Cloud API collection is the primary method for LX10 (Cloud VM Compromise), where the investigation explicitly exercises the two-plane principle — correlating VM filesystem evidence with cloud audit trail events to reconstruct the attacker's use of stolen cloud credentials. Container collection is the primary method for LX9 (Container Compromise), where the docker diff technique identifies exactly what the attacker added to the base image. LX11 (Lateral Movement) combines all three — the attacker moves from a container to the host to the cloud API, and the investigator collects evidence from all three environments and builds a unified timeline across them.
Check your understanding:
1. When you SSH into a compromised system to collect evidence, what entries does your connection create in the system's logs, and how do you prevent these from being confused with attacker activity? 2. Why is a cloud disk snapshot forensically cleaner than running dd on the live system? 3. What does docker diff show, and why is it more useful for investigation than listing all files in the container? 4. A Kubernetes pod was terminated 5 minutes ago and a new pod has been scheduled. What evidence from the old pod is still available, and where would you look for it? 5. You are investigating an Azure VM compromise. You have collected a disk snapshot and SSH volatile data. What evidence plane are you still missing, and what commands collect it?
You are investigating a Linux server and discover evidence of both a cryptominer (resource abuse) and an SSH key theft (lateral movement preparation). The cryptominer is consuming 95% CPU and impacting production. Which do you address first?
Address the lateral movement first. The cryptominer is visible, noisy, and contained to this server — it is causing performance impact but not spreading. The SSH key theft is silent, potentially already exploited, and may have given the attacker access to additional servers. Contain the lateral movement risk: rotate the stolen SSH keys, check the target servers for unauthorized access, and apply network restrictions. Then address the cryptominer: kill the process, remove the binary and persistence mechanisms. Prioritizing the noisy but contained threat over the silent but spreading threat is the most common Linux IR prioritization mistake.
Get weekly detection and investigation techniques
KQL queries, detection rules, and investigation methods — the same depth as this course, delivered every Tuesday.
No spam. Unsubscribe anytime. ~2,000 security practitioners.