aegis-updates-patching

Scope & Objectives

Assess patch discipline and attack-surface minimization on one Linux server (or its config-as-code repo). Answer, in priority order:

Distro identity & lifecycle stage — which release is running, and is it plausibly within vendor support? Flag uncertainty honestly (Needs-Review with the exact release string), never assert an invented EOL date.
Reboot debt — does uname -r lag the newest installed kernel package? Installed-but-not-running fixes are dormant.
Pending update inventory — how many updates are outstanding, and how many belong to the security pocket / advisory stream.
Automation — is an unattended/automated security-update mechanism installed, configured with security origins, and armed (timer active)?
Restart honesty — uptime evidence for deferred kernel/libc/systemd restarts; maintenance-window posture.
Surface shrinkage — offender services (rpcbind, nfs-common, avahi-daemon, cups, postfix, xinetd, samba) and legacy protocol servers (telnet/rsh/tftp/ftp) in enabled/active/disabled/masked/purged states.
Kernel-module hardening option — uncommon filesystem blacklists (present as options with breakage caveats, not unconditional findings).
Host-level runtime currency — system python/node/php majors plausibly supported; Needs-Review when uncertain.

Out of scope (other modules): application dependency patching → supply-chain/code module; firewall/exposure rules → FW; sshd/PAM/sysctl baseline → BASE.

Operating rules:

Every collection command here is read-only. Mutating commands appear only under Remediation.
Commands needing root are tagged [ROOT]; without root, audit world-readable state plus the config repo.
Distro variance is called out inline — detect first (cat /etc/os-release), never assume Debian values on RHEL or Alpine.

Prerequisites & Vocabulary

Zero-background primer: the terms this module uses, one line each. Deeper plain-language explanations for every class live in the repository GUIDE.md glossary.

EOL: software past vendor support — no further patches will ever exist for it
backporting: vendors shipping fixes inside the original version number, so version numbers lie about vulnerability
reboot debt: newer kernels installed on disk than the one running; those fixes stay dormant until reboot
unattended-upgrades: automation, properly armed, that applies security fixes without a human remembering
attack surface: listening services multiplied by their flaws by who can reach them; removing a daemon shrinks it permanently
enabled vs active: "starts at boot" and "running now" are independent service states
taint flow / source→sink: path untrusted data travels from entry point to dangerous function
finding status: Confirmed > Probable > Needs-Review; evidence rules in templates/finding-report.md

Mental Model

Two clocks tick on every server:

The vulnerability clock: public bugs in installed software accumulate daily regardless of what you do.
The fix clock: a vendor fix only takes effect after download → install → process restart → occasionally reboot.

Patch discipline is the machinery keeping the fix clock synchronized to the vulnerability clock. It decomposes into four auditable layers:

Layer	Question	Failure signature
Inventory	What is installed, what is stale?	Nobody can say how far behind the host is
Automation	Does anything apply security fixes without a human remembering?	Config exists but timer dead, or nothing exists at all
Restart honesty	Are installed fixes actually effective?	Kernel packages installed months ago, uptime 400 days
Surface shrinkage	Is unnecessary software absent or provably off?	rpcbind/cups/avahi running on a headless app node

Corrections that prevent bad audits:

Version numbers lie on enterprise distros. Red Hat and Ubuntu backport fixes into the original version number (openssl may stay 1.1.1 across dozens of CVE fixes on one release). Raw version comparison says "vulnerable" forever. Consult changelogs before judging.
Two kernels exist per host: the one executing (uname -r, a snapshot from boot time) and the newest one on disk (dpkg -l linux-image-* / rpm -q kernel). They diverge between "packages updated" and "machine rebooted". That gap is reboot debt, and it silently re-arms every kernel-side fix you thought you had.
Service state is a pair, not a boolean. is-enabled (starts at boot?) and is-active (running now?) are independent. Disabled-but-active units were started by hand or by another unit's dependency — find out who owns that decision.
Attack surface = listening services × their unpatched flaws × who can reach them. Removing a daemon deletes all its future CVEs permanently — the only control that improves over time without effort.

What To Check

Start with this paste-ready read-only sweep (~20 lines), then drill into each area below:

# --- READ-ONLY patching & attack-surface sweep ---------------------------------
. /etc/os-release; echo "DISTRO: $ID ${VERSION_ID:-unknown} ($PRETTY_NAME)"
echo "RUNNING-KERNEL: $(uname -r)"
command -v dpkg >/dev/null && dpkg -l 'linux-image-[0-9]*' 2>/dev/null | awk '/^ii/{print "INSTALLED-KERNEL:",$2,$3}' | sort -V | tail -n 3
command -v rpm  >/dev/null && rpm -q kernel 2>/dev/null | sort -V | tail -n 3
command -v apt  >/dev/null && { \
  echo "UPGRADABLE-TOTAL: $(apt list --upgradable 2>/dev/null | tail -n +2 | wc -l)"; \
  echo "SECURITY-ESTIMATE: $(apt-get -s dist-upgrade 2>/dev/null | grep '^Inst' | grep -c security)"; }
command -v dnf  >/dev/null && { \
  echo "DNF-UPDATE-LINES(est): $(dnf -q check-update 2>/dev/null | grep -c '\.el[0-9]\|\.fc[0-9]')"; \
  echo "SECURITY-ADVISORIES:"; dnf -q updateinfo list security 2>/dev/null | head -n 10; }
command -v apk  >/dev/null && { echo "APK-OUTDATED:"; apk version -l '<' 2>/dev/null | head -n 10; }
for s in rpcbind nfs-common avahi-daemon cups postfix xinetd smb nmb telnet.socket tftp.socket vsftpd; do
  ac=$(systemctl is-active  "$s" 2>/dev/null); en=$(systemctl is-enabled "$s" 2>/dev/null)
  [ -n "$ac$en" ] && printf '%-14s active=%-10s enabled=%s\n' "$s" "${ac:-unknown}" "${en:-unknown}"
done
echo "UPDATE-TIMERS:"; systemctl list-timers --all 2>/dev/null | grep -iE 'apt|dnf|yum|unattended' || echo "  NONE FOUND"
echo "BOOT-TIME: $(uptime -s)"

Interpretation notes for the sweep: SECURITY-ESTIMATE is a heuristic (simulation lines name the -security suite; package names containing "security" can inflate it). dnf check-update exits 100 when updates exist and prints nothing on clean hosts — do not treat exit≠0 as an error. is-active reports inactive even for unknown units, so treat enabled=unknown rows as "not installed".

1. Distro identity & lifecycle stage

Parse /etc/os-release; never infer from motd or hostname.

. /etc/os-release && printf '%s %s\n' "$ID" "${VERSION_ID:-rolling}"

Distro family	Identification in os-release	Lifecycle reasoning guidance	Hardened state
Debian	`ID=debian`, `VERSION_ID=12`	stable → oldstable → oldoldstable as successors release; security support shrinks each step. If you cannot confirm which stage this codename occupies today → Needs-Review with codename	Current stable/oldstable with security suite enabled
Ubuntu	`ID=ubuntu`	LTS releases follow a ~5-year standard support pattern (longer via add-on plans); interim releases live months. Unconfirmed status → Needs-Review with release name	Supported LTS within standard window
RHEL	`ID=rhel`	Major versions carry decade-scale lifecycles with phased maintenance. Confirm currently-maintained major.minor; else Needs-Review	Maintained major.minor
Rocky / AlmaLinux	`ID=rocky` / `ID=almalinux`	Clone of the corresponding RHEL major's lifecycle; apply the same treatment	Maintained major.minor
Amazon Linux	`ID=amzn` (`VERSION_ID=2` vs `2023`)	AL2 and AL2023 have distinct lifecycles; if unsure which applies → Needs-Review	Currently maintained major
Alpine	`ID=alpine`, branch = VERSION_ID major.minor	Stable branches have limited lifespans tied to kernel support; unconfirmed branch → Needs-Review	Current stable branch

Rule: never assert an EOL date from memory. Qualitative patterns you may cite: Ubuntu LTS ≈ five years standard support; Debian oldoldstable receives reduced/no security support once two successors ship. Anything beyond that goes to Needs-Review carrying the exact release string (e.g., "Ubuntu 18.04 — verify support window").

2. Running kernel vs installed kernels

uname -r                                   # kernel executing NOW
# Debian/Ubuntu — newest installed image:
dpkg -l 'linux-image-[0-9]*' | awk '/^ii/{print $2, $3}' | sort -V | tail -n 3
# RHEL/Rocky/Alma/Amazon Linux:
rpm -q kernel | sort -V | tail -n 3        # AL2023: also rpm -q kernel-core
# Reboot-required verdict (RHEL family, if present):
command -v needs-restarting >/dev/null && [ROOT] needs-restarting -r

If the highest-versioned installed kernel is newer than uname -r → finding "pending reboot": fixes shipped to disk are dormant in memory. On Debian/Ubuntu compare ABI strings (e.g., 5.15.0-91-generic), not meta-package numbers.

3. Pending update inventory (read-only)

Debian/Ubuntu:

apt list --upgradable 2>/dev/null | tail -n +2          # total pending
apt-get -s dist-upgrade | grep '^Inst' | grep -c security   # security-pocket estimate
[ -x /usr/lib/update-notifier/apt-check ] && /usr/lib/update-notifier/apt-check --human-readable
#   → "N packages can be updated. M updates are security updates."
command -v unattended-upgrades >/dev/null && [ROOT] unattended-upgrades --dry-run --debug 2>&1 | tail -n 20

Caveat: apt list uses the cached index. A stale cache undercounts. Do not run apt-get update during a read-only audit; instead check freshness via stat /var/lib/apt/periodic/update-success-stamp.

RHEL-family:

dnf -q check-update                 # exit 100 = updates available; lists them
dnf -q updateinfo list security     # one row per advisory touching installed pkgs
dnf -q updateinfo summary           # counts incl. security bucket

Alpine:

apk version -l '<'                  # installed packages older than repo candidates
apk policy <pkg>                    # pinning/repo detail for a specific package

Live-patching presence (canonical names only — no third-party guesses):

command -v kpatch               >/dev/null && [ROOT] kpatch list
command -v canonical-livepatch  >/dev/null && canonical-livepatch status

4. Automated security updates armed?

Debian/Ubuntu — config files plus timers must BOTH check out:

# /etc/apt/apt.conf.d/20auto-upgrades — both keys must be "1"
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

grep -E 'Update-Package-Lists|Unattended-Upgrade' /etc/apt/apt.conf.d/20auto-upgrades
grep -A6 'Allowed-Origins' /etc/apt/apt.conf.d/50unattended-upgrades | grep -- '-security'
systemctl is-enabled apt-daily.timer apt-daily-upgrade.timer   # download / apply
systemctl list-timers --all | grep -iE 'apt|unattended'

Key fields in 50unattended-upgrades: Allowed-Origins must cover ${distro_id}:${distro_codename}-security (the whole point of the exercise); Automatic-Reboot + Automatic-Reboot-Time tradeoff; Mail notification hook so silence cannot hide regressions.

RHEL-family:

rpm -q dnf-automatic || rpm -q yum-cron                 # EL7 uses yum-cron
grep -E '^apply_updates' /etc/dnf/automatic.conf         # yes = install, not just download
systemctl is-enabled dnf-automatic.timer dnf-automatic-install.timer 2>/dev/null
systemctl list-timers --all | grep -iE 'dnf|yum'

Timer semantics: dnf-automatic.timer honors apply_updates (with no it only downloads); dnf-automatic-install.timer always installs. Exactly one should be enabled, chosen deliberately.

Alpine ships no stock unattended mechanism; if nothing provisions periodic upgrades (e.g., scripts under /etc/periodic/), mark Needs-Review rather than auto-failing.

5. Reboot honesty

uptime -s      # boot timestamp
uptime -p      # pretty duration

Kernel, glibc, systemd, and OpenSSL-linked daemons fully pick up fixes only at restart or reboot. Long uptime + pending kernel packages = deferred remediation, not stability. Plan maintenance windows; on fleets, stagger reboots cohort-by-cohort behind load balancers rather than big-bang.

6. Package minimization & offender-service trichotomy

Context first — installed-package count vs role expectation:

dpkg-query -W -f='${binary:Package}\n' 2>/dev/null | wc -l   # Debian-family
rpm -qa | wc -l                                              # RPM-family
apk info | wc -l                                             # Alpine

Classic offenders on standalone app servers (each needs explicit justification to stay):

Offender	Why it is surface	Verdict unless justified
rpcbind (+ nfs-common)	Portmapper historically abused for reflection/amplification; drags NFS RPC surface	remove/purge
avahi-daemon	mDNS discovery; zero server role	remove/purge
cups	Print server on a headless box; history of RCE-class bugs	remove/purge
postfix / exim4	Mail daemon on hosts that need not originate mail	remove/purge (check sendmail-binary consumers first)
xinetd	Legacy superserver resurrecting old protocols	remove; audit `/etc/xinetd.d/*` if kept
samba (smb, nmb)	File-sharing surface on non-file servers	remove/purge
telnet / rsh / rlogin clients	Cleartext credentials by design	remove/purge (use ssh)
telnetd, rsh-server, tftpd/tftp-server, vsftpd/proftpd	Cleartext legacy protocol servers	INSTANT FINDING if installed or listening

Trichotomy — check BOTH dimensions per suspect; the matrix tells you who started what:

systemctl is-active  rpcbind avahi-daemon cups postfix xinetd smb nmb 2>/dev/null
systemctl is-enabled rpcbind avahi-daemon cups postfix xinetd smb nmb 2>/dev/null

active	enabled	Meaning
yes	yes	Conventional; justify or remove
no	no	Good (purged is better)
yes	no	Started manually or by another unit's dependency — identify the trigger before judging
no	yes	Dead weight that returns at reboot; disable
yes	masked	Someone bypassed the mask — hygiene finding

Orphan sockets: every listener must map to an owning package; ownerless listeners (deleted binaries, hand-copied daemons) are findings. Cross-reference the firewall module's sweep for external reachability.

[ROOT] ss -tulpn | awk 'NR>1{print $5, $7}'          # local addr:port, process info
sudo readlink -f /proc/<PID>/exe                      # resolves deleted/renamed binaries
dpkg -S "$(readlink -f /proc/<PID>/exe)"              # Debian-family owner lookup
rpm -qf "$(readlink -f /proc/<PID>/exe)"              # RPM-family owner lookup

7. Host-level language runtime currency

Record system interpreters and compare qualitatively against known lifecycle patterns (python: annual feature releases, multi-year support tails; node: even-numbered LTS lines; php: annual cycles). When you cannot state the installed major.minor's support standing with confidence, mark Needs-Review with the exact version string — do not invent cutoff dates.

python3 --version 2>/dev/null
node --version 2>/dev/null
php -r 'echo PHP_VERSION, "\n";' 2>/dev/null

Application-level dependency patching (node_modules, virtualenvs, composer vendors, gem bundles) belongs to the supply-chain/code module — reference it, do not duplicate it here.

8. Kernel-module & legacy-tech surface (optional hardening)

Blacklisting uncommon filesystem drivers shrinks kernel attack surface — code that cannot load cannot be exploited. Present as hardening options WITH breakage caveats, never as unconditional findings:

# /etc/modprobe.d/fs-blacklist.conf — FIXED example
# Verify none is load-bearing BEFORE deploying:
#   vfat     → required for EFI System Partition mounts
#   squashfs → used by many container/live-media stacks (snap, some initramfs)
install cramfs    /bin/false
install freevxfs  /bin/false
install jffs2     /bin/false
install hfs       /bin/false
install hfsplus   /bin/false

Legacy protocol servers (telnet server, rsh/rlogin/rexec daemons, plain FTP, tftp) are instant findings wherever installed or listening — cleartext by design; replace with ssh/scp/sftp/https.

9. Deeper scanning (requires tools/network — clearly optional)

Run only where tooling exists and policy permits outbound reads; base forms only, no invented flags:

# RHEL-family OpenSCAP + SCAP Security Guide: enumerate profiles, then evaluate
oscap info /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml
[ROOT] oscap xccdf eval --profile <PROFILE_ID> \
      --results /tmp/oscap-results.xml --report /tmp/oscap-report.html \
      /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml

# Lynis — audit-mode framing: reads system state, writes only its own log/dat files
[ROOT] lynis audit system

# Debian — compare installed set against the Debian security tracker
debsecan --suite <CODENAME>

Where To Look

Evidence collection: tools/sweeps/sweep-patching.sh captures [PATCH-nn] sections verbatim; judge them against this module's rubrics, never against raw output alone.

Authoritative locations, by family; verify existence before parsing:

Location	What it proves
`/etc/os-release`	Distro identity — source it, never regex the hostname
`uname -r` vs `dpkg -l linux-image-*` / `rpm -q kernel`	Divergence = reboot debt
`/var/lib/apt/periodic/update-success-stamp` (mtime)	Last successful index refresh; stale stamp explains suspiciously clean `apt list --upgradable`
`/etc/apt/apt.conf.d/20auto-upgrades`	`APT::Periodic::Update-Package-Lists` and `::Unattended-Upgrade` both `"1"`
`/etc/apt/apt.conf.d/50unattended-upgrades`	`-security` origin present; `Automatic-Reboot{,-Time}`; `Mail` hook
`/var/log/unattended-upgrades/unattended-upgrades.log`	Proof the machinery actually fires (dates of past runs)
`/etc/dnf/automatic.conf` (EL7: `/etc/yum/yum-cron.conf`)	`apply_updates` value
`systemctl list-timers --all`	Armed timers — configs without timers mean automation is dead
`/etc/modprobe.d/*.conf`	Filesystem/module blacklist posture
`/etc/xinetd.d/`, `/etc/inetd.conf`	Legacy superserver entries (telnet, tftp, chargen…)
`[ROOT] ss -tulpn` output	Listener↔owner map feeding minimization review
`python3/node/php --version` outputs	Runtime currency inputs

Patterns & Signatures

Vulnerable/fixed pairs:

# VULNERABLE — automation present but inert (both switches off)
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";

# FIXED
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

// VULNERABLE — origins trimmed, security pocket NOT covered
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
};

// FIXED — security pocket explicitly included
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}-security";
};

# VULNERABLE — dnf-automatic downloads but never installs
apply_updates = no

# FIXED
apply_updates = yes

State signatures:

Signature	Meaning	Action
`uname -r` older than newest installed kernel pkg	Reboot debt; fixes dormant	Finding + window plan
`20auto-upgrades` missing entirely	Automation never provisioned	Finding
Config keys correct but timer absent/disabled	Automation armed on paper only	Finding — verify `list-timers` always
Service `active` while `disabled`	Out-of-band start; unknown owner process	Trace trigger, then decide
Service `masked` yet `active`	Mask deliberately bypassed	Hygiene finding
Listener on :23/:21/:69(udp)/:512–514/:873	telnet/ftp/tftp/rsh-family/rsync exposed	Instant finding (cross-ref FW)
`ss -tulpn` row with no resolvable package	Orphan/hand-copied binary listening	Finding
`/var/log/unattended-upgrades/` empty or absent on Ubuntu	Machinery never ran	Finding

Kernel staleness signature (annotated):

$ uname -r
5.4.0-150-generic                     ← executing (boot-time snapshot)
$ dpkg -l 'linux-image-[0-9]*' | awk '/^ii/{print $3}' | sort -V | tail -n 1
5.4.0-169.186                         ← newest on disk (fixes dormant)
→ gap 150→169 = every kernel fix in between requires a reboot to matter

Taint Tracing Guidance

Treat external reachability as the taint source; propagate along ownership chains:

Source: listener bound to 0.0.0.0/::/external IP in ss -tulpn. Everything owning or feeding it inherits taint.
Owner resolution: PID → /proc/PID/exe (resolve symlink; catches deleted binaries) → dpkg -S / rpm -qf → package@version.
Advisory join: package appears in apt-get -s dist-upgrade Inst lines naming -security, or as a row in dnf updateinfo list security → tainted-by-advisory (vendor fix exists, host lacks it).
Restart-gap sizing: uptime -s bounds how long kernel/libc/systemd classes have been dormant; package install timestamps (stat on /var/lib/dpkg/info/<pkg>.list mtime, rpm -q --qf '%{INSTALLTIME:date}\n' <pkg>) bound when fixes landed on disk.
Boot lineage: /proc/cmdline BOOT_IMAGE= plus grubby --default-kernel (EL family) tie the running kernel to its on-disk counterpart; mismatch vs newest installed = tainted kernel class.

Record chains compactly: host → listener/port → package@version → advisory-stream member → restart-gap. Severity inherits from reachability at the chain's HEAD, not from the number of stale packages overall. An internet-facing tainted listener outranks fifty stale internal libraries.

Exploitation & Reproduction

READ-ONLY demonstrations. Objective: prove exposure exists and would survive contact with an attacker; change nothing.

Demo A — prove pending security updates:

/usr/lib/update-notifier/apt-check --human-readable
# "12 packages can be updated. 4 updates are security updates."
#   → M > 0 means vendor-shipped fixes for published vulns are sitting in the repo,
#     one command away, not applied.
dnf -q updateinfo list security
# each row = advisory ID + affected package: vendor fix exists, host lacks it.

Interpretation discipline: report the SECURITY count, not just totals. "300 upgradable" is noise; "4 security advisories including the internet-facing service's daemon" is the finding.

Demo B — prove stale running kernel:

uname -r                                                  # 5.4.0-150-generic
dpkg -l 'linux-image-[0-9]*' | awk '/^ii/{print $2,$3}' | sort -V | tail -n 2
# newest installed: linux-image-5.4.0-169-generic

Explanation for the report: memory executes the 150 build; fixes shipped in builds 151–169 activate only at next boot. Any vendor-fixed kernel privilege-escalation or remote flaw disclosed meanwhile remains exploitable on this host although the package inventory claims otherwise. This is the canonical silent failure mode of patch programs — green dashboards fed by "updates installed", red reality on the wire.

Demo C — prove automation absent:

stat /etc/apt/apt.conf.d/20auto-upgrades        # No such file or directory
systemctl list-timers --all | grep -i apt       # (no output)
ls /var/log/unattended-upgrades/ 2>&1           # No such file or directory
stat /var/lib/apt/periodic/update-success-stamp 2>/dev/null   # months-old mtime, or absent

Missing config + missing timer + empty logs = nothing fetches or applies security fixes without a human remembering. On dnf systems: automatic.conf absent and no dnf-* timer in list-timers proves the same.

Attacker narrative (qualitative — no invented identifiers): adversaries mass-scan address space, fingerprint service banners, and replay publicly documented exploitation techniques against vulnerabilities vendors fixed weeks earlier. The dominant breach path for internet-facing services is precisely this n-day replay — no novel research required. Hosts without update automation sit in the disclosure-to-patch gap indefinitely, because the disclosure happens upstream whether or not the host ever hears about it. Internal hosts inherit the same risk laterally once any foothold exists anywhere on the flat network.

Remediation

Order: automate FIRST (stops the bleeding), then drain reboot debt in planned windows, then shrink surface, then optional hardening.

Enable unattended security updates

Debian/Ubuntu:

sudo apt-get install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades    # seeds 20auto-upgrades with "1"

/etc/apt/apt.conf.d/20auto-upgrades — FIXED:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

/etc/apt/apt.conf.d/50unattended-upgrades — security origin mandatory; choose reboot behavior consciously; keep a notification hook:

Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}-security";
};
// Option 1 — hands-off: automatic reboots inside a scheduled window
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:30";
// Option 2 — operator-owned reboots: keep "false" and run the window checklist below
// Unattended-Upgrade::Automatic-Reboot "false";
// Notification so silence cannot hide regressions:
Unattended-Upgrade::Mail "root";

Tradeoff note: Automatic-Reboot "true" trades surprise-free uptime for hands-off freshness. On stateful database nodes prefer "false" plus explicit windows.

RHEL-family:

sudo dnf install -y dnf-automatic                       # EL7: sudo yum install yum-cron
sudo sed -i 's/^apply_updates *=.*/apply_updates = yes/' /etc/dnf/automatic.conf
sudo systemctl enable --now dnf-automatic-install.timer

Enable exactly ONE timer: dnf-automatic-install.timer (always installs) or dnf-automatic.timer (honors apply_updates). Enabling both invites double semantics.

Reboot planning checklist

Classify pending changes: kernel → reboot; libc/systemd → practically everything restarts (treat as reboot); others → targeted service restarts (EL: needs-restarting -s lists processes).
Announce and schedule the window; drain traffic (load-balancer removal, replica step-down).
Reboot ONE cohort; verify uname -r now equals the newest installed kernel; verify application health.
Proceed cohort-wise; previous kernels remain in the bootloader as rollback by default.
Fleet staggering one-liner: never simultaneous fleet-wide reboots — overlap-free cohorts sized so any single cohort's failure still meets availability targets.

Remove-unused-services procedure — disable → mask → verify → purge later

Stop + disable: systemctl disable --now NAME — stops it now and removes boot autostart. Trivially reversible.
Mask: systemctl mask NAME — symlinks the unit to /dev/null so NOTHING can activate it: not other units' Requires/Wants, not D-Bus or socket activation, not muscle memory.
Verify across at least one reboot cycle plus a monitoring period: is-active fails, socket closed, application metrics unchanged.
Purge afterwards: sudo apt purge NAME / sudo dnf remove NAME — deletion is the least reversible step; doing it LAST lets the observation period reveal hidden dependents safely.

Why disable+mask precedes purge: masking gives reversible protection against ALL activation paths during observation, while purge is permanent and surfaces dependency surprises at the worst possible time if done first.

Pre-purge blast-radius check (postfix example — many apps shell out to the sendmail binary):

command -v sendmail
grep -rl sendmail /usr/local/bin /opt /etc/cron* 2>/dev/null

If consumers exist, migrate them to an SMTP relay or msmtp-style client BEFORE purging postfix.

Blacklist example file

Deploy the /etc/modprobe.d/fs-blacklist.conf content shown under What To Check §8. Caveats to carry into the change record: vfat is load-bearing on EFI-booted hosts; squashfs breaks snap/container-live-media stacks; test on a canary with out-of-band console access before fleet rollout.

Verification & Validation

Post-fix positives:

systemctl is-enabled apt-daily-upgrade.timer                          # → enabled
systemctl list-timers --all | grep -i apt                             # future NEXT elapse
grep -E 'Update-Package-Lists|Unattended-Upgrade' /etc/apt/apt.conf.d/20auto-upgrades   # both "1"
grep -c -- '-security' /etc/apt/apt.conf.d/50unattended-upgrades      # ≥ 1
grep '^apply_updates' /etc/dnf/automatic.conf                         # → apply_updates = yes

After the first maintenance window — security counters trending zero:

/usr/lib/update-notifier/apt-check --human-readable    # "...0 updates are security updates."
dnf -q updateinfo list security                        # empty output
uname -r                                               # equals newest installed kernel (re-list and compare)

Negative tests (nothing broke):

Legit package installs still work: apt-get -s install curl resolves cleanly; a real install of an approved package succeeds in staging.
Application unaffected by removed services: health endpoints return 200; job queues drain; mail flows if postfix was replaced (sendmail consumers verified pre-purge).
Masked services stay dead across one controlled reboot.

Regression watch:

Automatic reboots caused surprise outages → schedule windows FIRST, announce via the Mail hook, THEN flip Automatic-Reboot.
Removing postfix broke apps calling the sendmail binary → restore service immediately (reinstall postfix or deploy the planned relay swap); the pre-purge grep exists precisely to prevent this.
Blacklisted modules broke an early-boot mount → revert the specific install … /bin/false line, regenerate initramfs if needed.

IaC regression greps (run at repo root):

grep -REn 'APT::Periodic::(Unattended-Upgrade|Update-Package-List)s?"\s*"0"' .
grep -REn '^apply_updates\s*=\s*no' .
grep -REin '(telnetd|rsh-server|tftp-server|xinetd|avahi-daemon|\bcups\b)' playbooks/ docker/ k8s/ 2>/dev/null

Severity Assessment

Weight REACHABILITY over raw counts. Ten stale packages on an isolated host matter less than one stale internet-facing daemon.

Anchor	Example CVSS v3.1 vector	Score	Notes
Internet-facing service with known vendor-fixed vulns unapplied >30 days (qualitative anchor)	`CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H`	9.8 Critical — High floor at minimum	N-day replay is the dominant breach path
Security auto-updates absent on INTERNAL host	`CVSS:3.1/AV:A/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:L`	5.7 Medium	Gap compounds silently until exposure changes
Pending reboot >90 days (kernel fixes dormant)	Same medium-vector family	Medium	Evidence: `uptime -s` vs installed kernels
Minor drift — isolated host, few non-security updates, no exposure	`CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:N/I:L/A:L`	3.5 Low	Housekeeping

Escalation multipliers: externally confirmed exposure (+), stale RUNNING kernel matching a disclosed kernel-flaw class (+), no notification channel so regressions hide (+), multiple anchors combined on one host (+). Downgrade where immutable infrastructure rebuilds continuously (see False Positives).

Common False Positives

Backported patches. Ubuntu/RHEL ship fixed code under UNCHANGED version numbers (openssl may remain 1.1.1 across dozens of CVE fixes on one release). Naive installed_version >= fixed_version logic false-positives forever. Before reporting, consult apt changelog <pkg> or rpm -q --changelog <pkg> for the vendor's fix reference.
Pinned/vendor-managed environments. Appliances and vendored images often update through a separate pipeline; absence of HOST automation is by design. Verify the pipeline exists instead of filing.
Read-only/immutable infra rebuilt continuously. Patching is replaced by redeploy cadence; assess image age and rebuild frequency, not in-place patch state. "No unattended-upgrades" is expected there.
Stale package index. apt list --upgradable trusts the cache; a months-old update-success-stamp makes zeros meaningless. Check freshness before trusting clean output.
Disabled-but-running services. is-enabled alone saying "disabled" does not prove anything is off — only is-active speaks for the running state. Check both, every time.

References

ubuntu.com/security — Ubuntu security overview and LTS lifecycle patterns
access.redhat.com — Red Hat Security Data and product lifecycle (root domain)
debian.org — Debian Security Tracker and release lifecycle (root domain)
alpinelinux.org — Alpine release branch lifecycle (root domain)
CIS Benchmarks — CIS Distribution Independent Linux Benchmark and per-distro benchmarks (patch management, service minimization, filesystem-module blacklisting sections)
CWE-1104: Use of Unmaintained Third Party Components
OWASP Top 10:2021 A06 — Vulnerable and Outdated Components
open-scap.org — OpenSCAP and SCAP Security Guide tooling (root domain)
cisofy.com — Lynis security auditing tool (vendor root domain)

About this skill

Scope & Objectives

Prerequisites & Vocabulary

Mental Model

What To Check

1. Distro identity & lifecycle stage

2. Running kernel vs installed kernels

3. Pending update inventory (read-only)

4. Automated security updates armed?

5. Reboot honesty

6. Package minimization & offender-service trichotomy

7. Host-level language runtime currency

8. Kernel-module & legacy-tech surface (optional hardening)

9. Deeper scanning (requires tools/network — clearly optional)

Where To Look

Patterns & Signatures

Taint Tracing Guidance

Exploitation & Reproduction

Remediation

Enable unattended security updates

Reboot planning checklist

Remove-unused-services procedure — disable → mask → verify → purge later

Blacklist example file

Verification & Validation

Severity Assessment

Common False Positives

References