Distro support
journalctl and the systemd journal are identical on both families. The
differences are the legacy /var/log/* text-file names and the package
manager. The body uses Debian/Ubuntu paths; the RHEL family (Fedora, RHEL,
CentOS Stream, Rocky, Alma, Oracle) equivalents are in the matrix.
| Log / concept | Debian/Ubuntu | RHEL family |
|---|---|---|
| System log (legacy) | /var/log/syslog |
/var/log/messages |
| Auth / sudo / SSH | /var/log/auth.log |
/var/log/secure |
/var/log/mail.log |
/var/log/maillog |
|
| Web (Apache) | /var/log/apache2/ |
/var/log/httpd/ |
| Cron | /var/log/syslog (CRON tag) |
/var/log/cron |
| systemd journal | journalctl … |
identical |
| logrotate configs | /etc/logrotate.d/ |
same |
| Package install | apt install <pkg> |
dnf install <pkg> |
RHEL-family note: minimal RHEL/Fedora installs may not ship rsyslog, so
the legacy /var/log/* files may be absent — the journal (journalctl) is the
primary source. journalctl -u <unit>, -p err, --since all work the same.
In sk-* scripts, prefer journalctl (portable) and resolve unit names via
svc_name from common.sh. See linux-bash-scripting
and docs/multi-distro/plan.md.
Use when
- Reading service, web, security, or database logs on a Linux server.
- Investigating spikes in errors, attacks, or slow queries.
- Reviewing log rotation or journal storage behavior.
Do not use when
- The task is general incident routing without a clear symptom; use
linux-troubleshooting. - The task is metrics collection or centralized observability setup; use
linux-observability.
Required Inputs
| Artefact | Source | Required? | If absent |
|---|---|---|---|
| Service/path and time window | Incident owner or alert | yes | Stop broad collection and request a bounded target |
| Symptom or event signature | Operator or monitoring evidence | yes | Return an inventory only; do not infer cause |
| Retention and privacy rules | Service owner or policy | for rotation/export | Do not delete, rotate early, or share sensitive records |
Capability Contract
Read/search access to journals and log files is sufficient for analysis. Rotation changes, vacuuming, deletion, permission changes, and service reloads require explicit authority; shared evidence must redact secrets and personal data.
Degraded Mode
Fallback when required access is unavailable: report the exact time/source coverage and mark missing periods unassessed. Never equate absent or rotated logs with absence of an event.
Decision Rules
| Condition | Action | Failure avoided |
|---|---|---|
| Event spans multiple services | Correlate timestamps and request IDs | Single-log attribution error |
| Log growth threatens disk | Preserve incident window, then apply authorised retention | Destroyed evidence |
| Sensitive fields appear | Redact values while retaining timestamp/context | Credential or personal-data exposure |
| Timestamp sources differ | Record timezone/clock offset | False sequence of events |
Workflow
- Confirm service, host, timezone, time window, retention, and read/search boundary.
- Preserve a bounded evidence slice before changing rotation or services.
- Correlate events across the minimum relevant sources and separate facts from inference.
- Route the cause to the owning service; stop when evidence cannot distinguish candidates.
- Apply only authorised rotation/retention changes and validate syntax.
- On failure, recover the prior logrotate configuration and retain collected evidence.
Evidence Produced
| Artefact | Acceptance |
|---|---|
| Time-bounded log extract | Names source, host, timezone, command, and redactions |
| Correlation record | Links the conclusion to timestamps/request IDs and states uncertainty |
| Rotation verification | Includes logrotate debug/forced-test result and post-change disk state |
Quality standards
- Use time-bounded inspection instead of dumping entire logs.
- Prefer concrete log evidence over speculation.
- Keep rotation and retention changes deliberate.
Anti-patterns
- Grepping random logs. Fix: identify service, host, timezone, and timeframe first.
- Treating volume as proof of cause. Fix: correlate content with service behaviour.
- Changing rotation before understanding growth. Fix: measure source and preserve the incident window.
- Publishing raw secrets or tokens. Fix: redact values and retain only useful context.
- Treating rotated logs as proof nothing happened. Fix: mark the missing period unassessed.
- Mixing local and UTC timestamps silently. Fix: normalise or label timezone offsets.
Outputs
| Artefact | Consumer | Acceptance condition |
|---|---|---|
| Log finding | Incident owner | States evidence, time coverage, cause/confidence, and gaps |
| Rotation change record | Operations | Shows authorised diff, validation, rollback, and result |
| Specialist handoff | Owning team | Includes the minimum reproducing records without secrets |
Worked Example
For an Nginx 502 spike from 14:00–14:10 EAT, extract only that access/error interval, correlate upstream errors with PHP-FPM journal entries, and preserve request IDs. If older logs rotated, mark earlier onset unassessed rather than declaring 14:00 the start.
References
This skill is self-contained. Every command below is a standard tool on
both the Debian/Ubuntu and RHEL families (only the legacy /var/log/* paths
differ — see Distro support above). The sk-* scripts in the Optional
fast path section are convenience wrappers — never required.
journalctl
sudo journalctl -u <service> -n 50 --no-pager # last 50 lines
sudo journalctl -u <service> -f # follow live
sudo journalctl -u <service> --since "1 hour ago"
sudo journalctl -p err --since "today" --no-pager # errors only
sudo journalctl -k --since "today" | grep -i oom # kernel OOM events
sudo journalctl --disk-usage # journal size
Nginx Logs
sudo tail -f /var/log/nginx/error.log
sudo tail -f /var/log/nginx/access.log
# HTTP status code distribution:
sudo awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
# Top IPs by request count:
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
# Recent 5xx errors:
sudo grep '" 5' /var/log/nginx/access.log | tail -20
Attack Pattern Detection
# Login brute-force attempts:
sudo grep -E "POST.*(login|wp-login|admin|xmlrpc)" /var/log/nginx/access.log | \
awk '{print $1}' | sort | uniq -c | sort -rn | head
# Scanner activity (high 404 rate per IP):
sudo awk '$9 == 404 {print $1}' /var/log/nginx/access.log | \
sort | uniq -c | sort -rn | head
# Attempts to access sensitive files:
sudo grep -E "\.(env|git|htaccess|sql|bak)" /var/log/nginx/access.log | tail -20
fail2ban Log
sudo tail -f /var/log/fail2ban.log
sudo grep "Ban" /var/log/fail2ban.log | tail -20
sudo grep "$(date '+%Y-%m-%d')" /var/log/fail2ban.log | grep "Ban" | wc -l
Other Key Logs
# PHP errors:
sudo tail -f /var/log/php8.3-fpm.log
# MySQL slow queries:
sudo tail -20 /var/log/mysql/mysql-slow.log 2>/dev/null
mysql -e "SHOW VARIABLES LIKE 'slow_query_log%';" 2>/dev/null
# Apache (port 8080 backend):
sudo tail -f /var/log/apache2/error.log
# Backup cron:
tail -50 ~/backups/mysql/cron.log
logrotate
ls /etc/logrotate.d/ # existing configs
sudo logrotate -f /etc/logrotate.d/nginx # force rotate now
sudo logrotate -f /etc/logrotate.d/apache2
All log file locations: references/log-locations.md
Script availability
No wrapper commands for this skill are shipped in this checkout. Use the native procedures above; planned wrappers remain in the script inventory until their source files are present.