LX0.8 The Investigation Toolkit

The Investigation Toolkit: What Each Tool Does and When You Use It

Toolkit philosophy: free first, commercial when justified

This course teaches every technique with at least one free or open-source tool. When you get the call at 03:00 that a server is compromised, the tool you need must be available immediately, not waiting for a procurement approval. Free tools are available now, on every forensic workstation, without budget or licensing delays.

Commercial tools are referenced where they provide significant advantages: faster processing, better GUI workflows, cross-platform correlation, or enterprise deployment. But the investigation methodology does not depend on them. Every finding can be reached with free tools. Commercial tools get you there faster when you have them.

Phase 1 tools: evidence collection

UAC (Unix Artifact Collector) — the automated triage tool covered in depth in LX1. Open-source, runs as a shell script, collects over 200 artifact categories. Use it as the first tool on every compromised system.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Deploy and run UAC from external media
# UAC requires no installation — it is a self-contained shell script
cd /mnt/usb/uac
./uac -p ir_triage /mnt/usb/output/
# -p ir_triage    : fast triage profile (3-8 minutes)
#                   captures: volatile data, key logs, user artifacts, bodyfile
# /mnt/usb/output : save output to external media (not the compromised disk)
# Output structure:
#   live_response/   → volatile data (processes, network, open files)
#   logs/            → log file copies
#   system/ + user/  → configuration and user artifacts
#   bodyfile/        → filesystem metadata for timeline generation
#   hash.txt         → SHA256 of every collected file

LiME (Linux Memory Extractor) — kernel module for memory acquisition. Pre-compile for every kernel version in your infrastructure. Covered in depth in LX1.6.

dc3dd — forensic disk imaging tool with simultaneous hashing. Covered in LX1.7.

Standard Linux commands — ps, ss, lsof, lsmod, mount, df, ip, find, stat, cat, grep, journalctl, ausearch, aureport, last, lastb, who, w. These commands are available on every Linux system. For live response, always supplement standard commands with direct /proc reads to bypass potential rootkit hooks.

Commercial alternative: Velociraptor — open-source with enterprise features. Provides agent-based remote collection. You deploy an agent on your infrastructure and can collect artifacts from any managed system through a web console without SSH access. It runs VQL queries that can collect specific artifacts, hunt across multiple systems simultaneously, and monitor for ongoing attacker activity. The key advantage: no SSH access needed. The key prerequisite: agent must be deployed before the incident.

Phase 2 tools: filesystem analysis

The Sleuth Kit (TSK) — the foundational open-source filesystem forensics toolkit. Works with disk images — you point it at a dd or dc3dd image and it reads filesystem structures directly, without mounting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Sleuth Kit core commands for Linux forensics

# List all files (including deleted) from a disk image
fls -r -m "/" /path/to/disk.raw
# -r          : recursive (all directories)
# -m "/"      : output in bodyfile format with "/" as mount point
# Deleted files appear with an asterisk (*) prefix
# Pipe to mactime for timeline: fls -r -m "/" disk.raw | mactime -b -

# Display inode metadata (all four timestamps + permissions)
istat /path/to/disk.raw 12345
# 12345 = inode number (from fls output)
# Shows: file type, permissions, owner, group, size,
#        atime, mtime, ctime, crtime, block addresses

# Extract file content by inode number (even if deleted)
icat /path/to/disk.raw 12345 > recovered_file
# Recovers the file content even if the directory entry was removed
# Works as long as the data blocks have not been overwritten

# Display partition layout
mmls /path/to/disk.raw
# Shows: partition table with start/end sectors and types
# Use the start sector as offset for fls: fls -o 2048 disk.raw

plaso / log2timeline — the unified timeline generation tool. Open-source, maintained by the Google Forensics team. plaso parses hundreds of artifact types from a disk image or evidence collection and produces a unified, time-sorted timeline. The timeline is the single most powerful analysis artifact in any investigation — it shows exactly what happened, in what order, across all evidence sources.

mactime (from Sleuth Kit) — generates a timeline from a bodyfile (filesystem metadata). Simpler and faster than plaso but limited to filesystem metadata only. Use mactime for a quick filesystem timeline during initial triage. Use plaso for the comprehensive timeline.

extundelete and photorec — deleted file recovery. extundelete recovers deleted files from ext3/ext4 by scanning the journal and inode table. photorec carves files from unallocated space based on content signatures — works even when inodes are overwritten.

Phase 3 tools: log analysis

journalctl — the systemd journal query tool. Available on all systemd-based distributions. Queries binary journal files with powerful filtering: by time, service, priority, PID, and boot session. The journal’s binary format makes it harder to tamper with than plaintext logs.

ausearch and aureport — the auditd log query tools. ausearch searches by event type, user, time range, system call, and audit key. aureport generates summary reports. These tools transform the raw audit.log into readable investigation-focused output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# ausearch examples for common investigation queries

# Find all command executions during the compromise window
ausearch -m EXECVE --start "03/28/2026" "03:00:00" --end "03/28/2026" "06:00:00"
# -m EXECVE    : filter to process execution events only
# --start/end  : time window matching the compromise timeline
# Output shows every command executed with full arguments

# Find privilege escalation events
ausearch -m USER_CMD -ua root --start "03/28/2026" "03:00:00"
# -m USER_CMD  : sudo/su command events
# -ua root     : events involving root user

# Generate a summary authentication report
aureport --auth --start "03/28/2026" "03:00:00"
# Shows: count of successful/failed authentications per user
# Quick way to identify brute force patterns

grep, awk, sed — the universal text processing tools. For unstructured log files (Apache, Nginx, application logs), these remain the fastest way to extract investigation-relevant entries. Available on every Linux system and process log files at disk read speed.

Phase 4 tools: memory analysis

Volatility 3 — the primary memory analysis framework for Linux. Reads a memory dump (from LiME) and extracts running processes, network connections, loaded kernel modules, bash history from process memory, open files, and much more. The critical prerequisite: requires an ISF (Intermediate Symbol Format) profile matching the target kernel exactly. Building ISF profiles is covered in LX12.

Phase 5 tools: malware analysis

file, strings, readelf — static analysis basics. file identifies the file type. strings extracts readable strings (C2 addresses, file paths, hardcoded credentials). readelf -a displays ELF binary metadata.

strace and ltrace — dynamic analysis tools. strace traces system calls made by a running process. ltrace traces library calls. Both used in a sandbox to observe malware behavior.

Worked artifact — Forensic workstation setup script:

Run this script on your forensic workstation or analysis VM to install the complete toolkit. Adapt paths and package names for your distribution.

# Forensic workstation setup — Ubuntu 22.04
sudo apt update && sudo apt install -y sleuthkit dc3dd wireshark python3-pip python3-venv git
python3 -m venv ~/tools/vol3 && source ~/tools/vol3/bin/activate && pip install volatility3 && deactivate
python3 -m venv ~/tools/plaso && source ~/tools/plaso/bin/activate && pip install plaso && deactivate
git clone https://github.com/tclahr/uac.git ~/tools/uac
git clone https://github.com/504ensicsLabs/LiME.git ~/tools/lime
echo "Verify: fls -V && dc3dd --version && ~/tools/vol3/bin/vol -h | head -1"

Decision points: when to use which tool

Quick triage (time-constrained, need answers fast): UAC ir_triage + mactime for filesystem timeline. Skip plaso (too slow for initial triage). Skip memory acquisition if LiME is not pre-compiled.

Full investigation (time available, legal proceedings possible): LiME for memory, dc3dd for disk image, UAC full profile, plaso for comprehensive timeline, Volatility 3 for memory analysis. Dual-hash everything.

Multi-system hunt (looking for indicators across infrastructure): Velociraptor for agent-based remote collection and VQL hunting queries. If Velociraptor is not deployed, use SSH-based UAC deployment in parallel across all target systems.

Troubleshooting: common tool issues

Volatility 3 cannot parse the memory dump: Wrong or missing ISF profile. The ISF must match the target kernel exactly. Check: vol -f memory.lime banners.Banners to identify the kernel version in the dump, then build or download the matching ISF.

plaso takes hours to process a disk image: plaso is thorough but slow on large images. For faster results, use mactime on just the bodyfile (filesystem metadata timeline) while plaso runs in the background. The mactime result covers most investigation questions within minutes.

Sleuth Kit fls shows no files: Wrong partition offset. Use mmls disk.raw to find the correct start sector, then fls -o [start_sector] disk.raw.

UAC fails to collect auditd logs: auditd is not installed or not running. UAC collects what exists — if audit.log does not exist, that artifact is skipped. This is normal on systems without auditd and is documented in the UAC output log.

Beyond this investigation

This toolkit maps to the entire course. LX1 uses UAC, LiME, and dc3dd. LX2 uses Sleuth Kit, plaso, mactime, and extundelete. LX3 uses journalctl, ausearch, and grep. LX12 uses Volatility 3. LX13 uses file, strings, strace, and ltrace. Each tool is taught in depth in the module where it is used — this subsection provides the complete map.