LX0.7 The Linux Distribution Landscape

The Distribution Landscape: What Changes Between Ubuntu, RHEL, and Alpine

Why distributions matter for investigation

Linux is not one operating system — it is a kernel with hundreds of distributions built on top of it, each making different decisions about where to put files, how to manage services, and which tools to include. An investigator who knows Ubuntu’s evidence paths but encounters a RHEL server will look for /var/log/auth.log and find it does not exist. The equivalent file is /var/log/secure. The commands are the same (cat, grep, journalctl), but the paths, filenames, and sometimes the log formats are different.

The distribution differences that matter for investigation fall into five categories: log file locations, package management, init system and service management, security frameworks, and filesystem layout conventions. Each category affects where you look for specific evidence types.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# The FIRST command on any compromised Linux system — identify the distro
# Run this before any evidence collection commands
cat /etc/os-release
# Key fields to note:
#   ID=ubuntu          → Debian family (auth.log, apt, AppArmor)
#   ID=rhel            → RHEL family (secure, dnf, SELinux)
#   ID=amzn            → Amazon Linux (RHEL-derived + SSM agent)
#   ID=alpine          → Alpine (minimal, musl, OpenRC, no systemd)
#   VERSION_ID=22.04   → specific version (affects available features)

# Fallback identification if os-release is missing or modified
cat /etc/debian_version 2>/dev/null && echo "→ Debian family"
cat /etc/redhat-release 2>/dev/null && echo "→ RHEL family"
cat /etc/alpine-release 2>/dev/null && echo "→ Alpine"
# If all three are missing, check the package manager:
which apt 2>/dev/null && echo "→ Debian family (has apt)"
which dnf 2>/dev/null && echo "→ RHEL family (has dnf)"
which apk 2>/dev/null && echo "→ Alpine (has apk)"

Distribution Family 1: Debian and Ubuntu

Debian and its derivative Ubuntu are the most common Linux distributions in cloud environments. AWS’s default Linux AMI is Amazon Linux (RHEL-derived), but Ubuntu is the most popular user-selected distribution on AWS, Azure, and GCP. The Northgate Engineering environment in this course runs Ubuntu 22.04 LTS.

Authentication logs: /var/log/auth.log is the primary authentication log. SSH login attempts, sudo usage, su session changes, and PAM events all appear here. The file is managed by rsyslog and rotated by logrotate — typically weekly with 4 rotated copies (auth.log.1, auth.log.2.gz, auth.log.3.gz, auth.log.4.gz). The rotated copies are gzip-compressed after the first rotation. To search across all rotated copies: zgrep "Failed password" /var/log/auth.log*.

System logs: /var/log/syslog is the general-purpose system log. Kernel messages, service start/stop events, cron execution, and application messages that use the syslog facility all appear here. The companion file /var/log/kern.log contains kernel-specific messages (useful for detecting kernel exploits, module loading, and hardware errors). /var/log/dpkg.log records package installation and removal by the dpkg subsystem. /var/log/apt/history.log records higher-level package operations by APT (install, upgrade, remove with timestamps and user).

Package management: apt (front-end) and dpkg (back-end). To check what packages were installed recently: grep "install " /var/log/apt/history.log | tail -20. To verify file integrity against package manifests: debsums -c (reports changed configuration files) or debsums -e (reports changed non-configuration files — more likely to indicate tampering). To check if a specific binary belongs to an installed package: dpkg -S /usr/bin/suspicious_binary. If the binary is not owned by any package, it was placed there manually — potentially by an attacker.

Init system: systemd (since Debian 8 and Ubuntu 15.04). Service unit files in /etc/systemd/system/ (admin-created) and /lib/systemd/system/ (package-installed). Custom services in /etc/systemd/system/ that you do not recognize warrant immediate investigation — the attacker may have created a persistence service.

Security framework: AppArmor is the default mandatory access control framework on Ubuntu. Profile files in /etc/apparmor.d/. The aa-status command shows which profiles are loaded and whether they are in enforce or complain mode.

Default shell: bash. History in ~/.bash_history. The default .bashrc configuration sets HISTSIZE=1000 and HISTFILESIZE=2000.

Distribution Family 2: RHEL, CentOS, Rocky, AlmaLinux, and Fedora

Red Hat Enterprise Linux (RHEL) and its derivatives (CentOS Stream, Rocky Linux, AlmaLinux) dominate enterprise bare-metal deployments, especially in regulated industries. Fedora is the upstream development distribution. Amazon Linux 2 and Amazon Linux 2023 are RHEL-derived and share most evidence paths.

Authentication logs: /var/log/secure is the equivalent of Debian’s /var/log/auth.log. Same content (SSH, sudo, su, PAM), same format, different filename. This is the single most common “wrong path” error when investigators switch between distributions.

System logs: /var/log/messages is the equivalent of Debian’s /var/log/syslog. On RHEL 9 and newer, systemd-journald is the primary logging system — rsyslog is optional and may not be installed. If rsyslog is not present, the traditional /var/log/messages and /var/log/secure files do not exist. All log data is in the systemd journal only. Check: rpm -q rsyslog — if not installed, use journalctl exclusively.

Package management: dnf (RHEL 8+, Fedora) or yum (RHEL 7 and earlier). To verify file integrity: rpm -Va (verifies all installed packages against their RPM manifests — any file that differs shows a code indicating what changed: S for size, 5 for MD5, T for mtime, M for mode, U for user, G for group). The output of rpm -Va is one of the most powerful integrity verification tools on any RHEL-based system.

Security framework: SELinux (Security-Enhanced Linux) is the default mandatory access control framework. It is significantly more restrictive than AppArmor. Check the current mode: getenforce. An attacker who needs to bypass SELinux may run setenforce 0 — check for this command in audit logs: ausearch -m MAC_STATUS.

Firewall: firewalld (RHEL 7+) manages iptables/nftables rules through zones. Check: firewall-cmd --list-all.

Distribution Family 3: Alpine and container-optimized distributions

Alpine Linux is the most common base image for Docker containers due to its small size (~5MB base image vs ~72MB for Ubuntu). Other container-optimized distributions include Google’s Container-Optimized OS (for GKE nodes) and AWS Bottlerocket (for EKS nodes).

What Alpine lacks: Alpine uses musl instead of glibc — some forensic tools compiled for glibc will not run. Alpine uses BusyBox for core utilities — ps does not support auxf, use ps -ef instead. Alpine uses OpenRC instead of systemd — there is no journalctl. Alpine uses apk as the package manager. There is no auth.log or secure file by default — authentication events go to /var/log/messages if syslog is running, or nowhere if syslog is not configured.

What this means for investigation: Container-based Alpine systems may have no persistent log files at all. The container’s stdout/stderr output (collected by the container runtime) may be the only log source. The investigation relies on container runtime logs and orchestrator audit logs rather than artifacts inside the container.

Container-Optimized OS (Google COS) and Bottlerocket (AWS): The root filesystem is read-only and verified at boot. Packages cannot be installed at runtime. SSH access is disabled by default. The evidence model is fundamentally different: the investigation focuses on the container layer and the orchestrator layer, not the node OS.

Amazon Linux — the cloud hybrid

Amazon Linux 2 and Amazon Linux 2023 deserve specific mention because they are the most common Linux distribution on AWS (the default AMI). Both include the AWS SSM Agent (Systems Manager) by default — which creates additional log files in /var/log/amazon/ssm/ and provides an alternative remote access mechanism (SSM Session Manager) that may not appear in SSH logs.

1
2
3
4
5
6
7

# On Amazon Linux — check for SSM-based access in addition to SSH
# SSM sessions do NOT appear in auth.log or secure
cat /var/log/amazon/ssm/amazon-ssm-agent.log | grep -i "session" | tail -10
# SSM access evidence is also in AWS CloudTrail:
#   EventName: StartSession, TerminateSession
# An attacker with compromised AWS IAM credentials can use SSM to
# access instances without any SSH key or password

Worked artifact — Distribution identification triage card:

Print this card and keep it in your IR kit. The first 30 seconds of any investigation should complete this card.

Case: INC-2026-XXXX System: [hostname] IP: [IP]

Distribution ID: _________________ (from cat /etc/os-release | grep ^ID=) Version: _________________ (from VERSION_ID=) Kernel: _________________ (from uname -r)

Evidence path mapping: Auth log: ☐ /var/log/auth.log ☐ /var/log/secure ☐ journal only ☐ none System log: ☐ /var/log/syslog ☐ /var/log/messages ☐ journal only Package mgr: ☐ apt/dpkg ☐ dnf/yum ☐ apk ☐ other: _____ Init system: ☐ systemd ☐ OpenRC ☐ other: _____ MAC framework: ☐ SELinux (mode: _____) ☐ AppArmor ☐ none Cloud agent: ☐ SSM Agent ☐ Azure Agent ☐ none

Notes: _______________________________________________

Decision points: which distribution details matter for your investigation

Not every distribution difference affects every investigation. The decision points that matter most:

RHEL 9+ without rsyslog: If rpm -q rsyslog returns “not installed,” there are no traditional log files. Your entire log analysis shifts to journalctl. This is a fundamental change in investigation methodology — not a minor path difference.

Alpine containers without syslog: If the container has no syslog daemon, evidence inside the container is limited to the filesystem and running processes. Shift focus to the container runtime logs (docker logs, kubectl logs) and the orchestrator audit log.

Amazon Linux with SSM Agent: Always check for SSM-based access in addition to SSH. An attacker who used SSM to connect is invisible in auth.log and secure — the evidence is in CloudTrail and the SSM agent log.

SELinux vs AppArmor: This determines where MAC violation evidence appears. SELinux denials: ausearch -m avc. AppArmor violations: journalctl -k | grep apparmor. Checking the wrong framework returns nothing — not because there are no violations, but because you queried the wrong subsystem.

Troubleshooting: when distribution identification fails

/etc/os-release is missing or modified: The attacker may have modified or deleted the file to confuse investigators. Fall back to distribution-specific files (/etc/debian_version, /etc/redhat-release, /etc/alpine-release). If all are missing, check the package manager: which apt && echo Debian or which dnf && echo RHEL or which apk && echo Alpine. The package manager binary is harder to fake.

Unfamiliar distribution or custom build: Some organizations build custom Linux images with non-standard paths. Check the init system (pidof systemd && echo systemd or pidof openrc-init && echo OpenRC), then check where syslog writes: grep -r "auth\." /etc/rsyslog.d/ /etc/rsyslog.conf 2>/dev/null shows you the actual path the syslog daemon uses for authentication events.

Container with almost nothing installed: Minimal container images may lack basic tools. If cat is not available (BusyBox image without it), try head -1 /etc/os-release. If the shell itself is limited (sh instead of bash), adjust your command syntax accordingly — avoid bash-specific features like process substitution and arrays.

Beyond this investigation

Distribution differences affect every investigation scenario in this course. The Northgate Engineering environment runs Ubuntu 22.04, so the primary evidence paths used in scenarios are Debian-derived. However, LX10 (Cloud VM Compromise) includes an Amazon Linux instance, and LX9 (Container Compromise) involves Alpine-based container images. The distribution reference in this subsection applies across all of them.