linux-config-management

Linux Configuration Management

Distro support

Ansible and /etc-tracking are cross-platform; write playbooks that work on both families. Body uses Debian/Ubuntu; the RHEL family (Fedora, RHEL, CentOS Stream, Rocky, Alma, Oracle) equivalents are in the matrix.

Concept	Debian/Ubuntu	RHEL family
Ansible package module	`ansible.builtin.apt`	`ansible.builtin.dnf`
Portable package module	`ansible.builtin.package` (works on both)	same
Mandatory access control	AppArmor	SELinux (enforcing by default)
/etc change tracking	`etckeeper`	`etckeeper` (same)
Service manager	systemd	systemd (identical)
Gather pkg-mgr fact	`ansible_facts.pkg_mgr` = `apt`	= `dnf`

RHEL-family gotchas: prefer ansible.builtin.package (or branch on ansible_facts['os_family'] / pkg_mgr) so one playbook runs on both. RHEL enforces SELinux — Ansible config that writes files into service paths must also manage SELinux contexts/booleans (ansible.posix.seboolean, community.general.sefcontext); the equivalent on Debian is AppArmor profiles. Deep SELinux coverage is in linux-server-hardening (Phase 2).

In sk-* scripts use the common.sh primitives instead of hardcoding apt/dnf. See linux-bash-scripting and docs/multi-distro/plan.md.

Use when

Converting manual server state into repeatable Ansible or git-tracked configuration.
Checking for configuration drift or validating idempotency.
Establishing a safer operating model for /etc and other managed assets.

Do not use when

The task is a one-off emergency fix where full automation is unnecessary.
The task is container or virtualization lifecycle work; use linux-virtualization.

Required inputs

Artefact	Source	Required?	If absent
Inventory and target scope	Operator or automation repository	required	Stop before running a playbook; report the missing host boundary.
Playbook, role, or tracked `/etc` baseline	Version-controlled source of truth	required	Produce an adoption plan, not a drift verdict.
Change authority, maintenance constraints, and secret source	Change owner	required for mutation	Stay read-only and limit work to check mode and diffs.

Workflow

Establish the declared state source: playbooks, inventory, or tracked config.
Inspect the current live state and compare it to the declared baseline.
Run the matching workflow below for adoption, drift checks, or remediation.
Verify idempotency and capture what changed versus what remains unmanaged.
Stop if inventory scope, authority, secrets handling, check-mode risk, or rollback is unresolved.
Recover a partial run by restoring the recorded prior state, limiting the next run to the failed cohort, and rechecking idempotency.

Quality standards

Prefer declarative, reviewable state over manual snowflake fixes.
Make drift visible before changing production.
Leave a clear path for repeatable re-application.

Anti-patterns

Using Ansible as a wrapper for ad-hoc shell. Fix: use idempotent modules and declare the end state.
Applying changes without check mode and diff. Fix: preview supported tasks and explain unavoidable check-mode gaps.
Tracking /etc without exclusions. Fix: exclude secrets, machine identity, caches, and generated state.
Hard-coding apt, apache2, or the sudo group. Fix: use facts, portable modules, and family mappings.
Calling one clean run idempotent. Fix: run again and require zero unintended changes.

Outputs

Artefact	Consumer	Acceptance condition
Declared-state change or drift report	Infrastructure maintainer	Every difference maps to a managed resource, accepted exception, or remediation.
Execution evidence	Reviewer	Includes syntax check, check-mode/diff result, applied task recap, and second-run idempotency result.
Recovery note	On-call operator	Names the rollback source and manual recovery for a failed partial run.

References

Evidence Produced

Artefact	Acceptance condition
Configuration-management evidence	Includes inventory scope, syntax result, check-mode diff, task recap, second-run change count, and accepted drift exceptions.

Capability contract

Read/search access to inventory and automation is required. Execute check mode before mutation when supported. Editing playbooks or changing hosts requires explicit authority; secret values must remain redacted and production fan-out must respect the approved batch size.

Degraded mode

Without host execution, review the automation statically and mark drift and idempotency not assessed. Without write authority, return a patch plan and safe validation sequence.

Decision rules

Choice	Action	Failure or risk avoided
Repeated fleet state	Manage with Ansible roles/playbooks	Snowflake hosts.
Emergency one-host repair	Apply the narrow fix, then back-port declared state	Permanent unmanaged drift.
Unknown live-versus-declared difference	Run check mode and diff first	Overwriting legitimate local state.

Worked example

When Apache is manually tuned on one AlmaLinux host, capture the intended directive in the role, run syntax and check mode against one canary, apply it, confirm httpd health, and run the playbook again expecting no change before widening the batch.

This skill is self-contained. Every command below is a standard tool (ansible, ansible-playbook, git, etckeeper, dpkg, diff). The sk-* scripts in the Optional fast path section are convenience wrappers — never required.

This skill owns the modern sysadmin practice of keeping a server's actual state aligned with a declared state that lives in git. It complements linux-server-provisioning (initial build) with ongoing continuous alignment.

Informed by Linux System Administration for the 2020s: move from "pet servers we hand-tune" to "cattle we rebuild from code." This skill starts that journey without requiring a full immutable-infrastructure rewrite.

It does not own:

Initial provisioning — linux-server-provisioning.
First-boot from YAML — linux-cloud-init.
Application deployment — linux-site-deployment.

When to use

Adopting Ansible for a server that was hand-configured.
Detecting configuration drift between declared state and actual state.
Dry-running a playbook against localhost before committing.
Tracking /etc in git for change auditing.
Deciding "should we rebuild or patch this server?" (this skill is the answer).

When NOT to use

A one-off manual config change on a throwaway machine.
Tasks owned by another skill (backups, firewall, deployment).

Standing rules

Every config change is recorded. If it's not in a git-tracked playbook or a committed /etc snapshot, it didn't happen.
Idempotency is the ultimate test. Every Ansible task must be safe to run twice. Second run = zero changes.
Drift is a first-class alert. Run a drift check on weekly cron; any drift emails the operator. Unexpected drift means either the playbook is incomplete or someone edited the server directly — both are bugs.
No manual edits on production. If you SSH in and nano, you create a snowflake. Make the change in the playbook, test in staging, roll forward.
Secrets never live in plain-text Ansible. Use Ansible Vault or sops — see linux-secrets.
Every playbook has a rollback plan. At minimum, the previous git commit. For risky changes, a pre-run /etc snapshot.

Quick reference — manual commands

Install Ansible

# Preferred on both families: pipx (isolates pinned version)
sudo apt install pipx          # Debian/Ubuntu
sudo dnf install pipx          # RHEL family (Fedora, RHEL, Rocky, Alma, ...)
pipx install --include-deps ansible
pipx ensurepath

# Alternative: distro package (older version but packaged)
sudo apt install ansible       # Debian/Ubuntu
sudo dnf install ansible       # RHEL family

# Verify
ansible --version

Local-only Ansible (managed host is localhost)

# Inventory-less run
ansible-playbook -c local -i 'localhost,' playbook.yml

# Or a minimal inventory file
cat > inventory.ini <<'EOF'
[local]
localhost ansible_connection=local
EOF

# Dry run with diff
ansible-playbook -i inventory.ini -c local playbook.yml --check --diff

# Real run
ansible-playbook -i inventory.ini -c local playbook.yml

# Idempotency test — run twice; second must show 0 changes
ansible-playbook -i inventory.ini -c local playbook.yml
ansible-playbook -i inventory.ini -c local playbook.yml   # MUST show "changed=0"

Check mode tags and facts

# Run only tagged tasks
ansible-playbook playbook.yml --tags ssh --check --diff

# Skip specific tags
ansible-playbook playbook.yml --skip-tags slow --check

# Gather facts only (no changes)
ansible localhost -c local -m setup
ansible localhost -c local -m setup -a 'filter=ansible_distribution*'

Tracking /etc in git (etckeeper)

sudo apt install etckeeper
# On install, etckeeper inits a git repo at /etc/.git and commits
sudo etckeeper vcs log | head
sudo etckeeper vcs status

# After a planned change
sudo etckeeper commit "hardening: set PermitRootLogin no"

# Show what changed between two points
sudo etckeeper vcs diff HEAD~5..HEAD

Manual git approach (without etckeeper):

sudo git -C /etc init
sudo git -C /etc add .
sudo git -C /etc commit -m "baseline snapshot $(date -Iseconds)"

# Later — see uncommitted drift
sudo git -C /etc status
sudo git -C /etc diff

Comparing declared vs actual

# Package list drift
dpkg --get-selections > /tmp/pkgs-actual.txt
diff /root/declared-packages.txt /tmp/pkgs-actual.txt

# Config file drift — diff against a committed baseline
sudo git -C /etc diff HEAD -- ssh/sshd_config
sudo git -C /etc diff HEAD -- sysctl.conf sysctl.d/

# Ansible check mode as drift detection
ansible-playbook -i inventory.ini -c local site.yml --check --diff
# Any "changed: <n>" with n > 0 = drift

Full Ansible idioms (idempotent module usage, creates:/removes:, changed_when:, handlers, roles, templates, Jinja2 filters, 3 complete playbook examples for linux-skills bases) — see references/ansible-patterns.md.

Full drift detection strategy (etckeeper vs raw git, Ansible check mode, AIDE integration, alerting, remediation workflow) — see references/drift-detection.md.

Full idempotency guide (common mistakes, how to fix them, two-run test, CI enforcement) — see references/idempotency-guide.md.

Typical workflows

Workflow: Adopting Ansible on an existing server

# 1. Baseline /etc
sudo apt install etckeeper
sudo etckeeper init
sudo etckeeper commit "baseline for Ansible adoption"

# 2. Write the first playbook — pick something simple (SSH hardening,
#    sysctl, unattended-upgrades). Keep it idempotent.
mkdir -p ~/ansible/playbooks
nano ~/ansible/playbooks/ssh-hardening.yml

# 3. Dry run
ansible-playbook -c local -i 'localhost,' \
    ~/ansible/playbooks/ssh-hardening.yml --check --diff

# 4. If the diff matches intent, apply for real
ansible-playbook -c local -i 'localhost,' \
    ~/ansible/playbooks/ssh-hardening.yml

# 5. Run AGAIN to prove idempotency
ansible-playbook -c local -i 'localhost,' \
    ~/ansible/playbooks/ssh-hardening.yml
# Expected: "changed=0"

# 6. Commit to git and schedule drift check
cd ~/ansible && git init && git add . && git commit -m "initial"

Workflow: Weekly drift check

# Via etckeeper
sudo etckeeper vcs status                     # anything uncommitted?
sudo etckeeper vcs diff                       # what's drifted

# Via Ansible check mode
ansible-playbook -c local -i 'localhost,' ~/ansible/site.yml \
    --check --diff --one-line | tee /tmp/drift.log

# If changed>0, investigate
grep "changed=" /tmp/drift.log

Workflow: "Is my playbook idempotent?"

# First run: expect changes
ansible-playbook -c local -i 'localhost,' playbook.yml
# Output: "changed=7"

# Immediate second run: MUST be zero
ansible-playbook -c local -i 'localhost,' playbook.yml
# Output: "changed=0"

# If the second run shows changes, the playbook is buggy.
# Common causes: see references/idempotency-guide.md

Workflow: Drift remediation

# 1. Drift detected on /etc/ssh/sshd_config
sudo git -C /etc diff HEAD -- ssh/sshd_config

# 2. Decide: is the drift legitimate?
#    - Yes → update the Ansible playbook to match new state, commit
#    - No  → re-run the playbook to restore, commit a note

# 3a. If legitimate, update playbook and re-test
vim ~/ansible/roles/ssh/tasks/main.yml
ansible-playbook -c local -i 'localhost,' ~/ansible/site.yml --check --diff

# 3b. If unwanted, restore from declared state
ansible-playbook -c local -i 'localhost,' ~/ansible/site.yml --tags ssh

# 4. Commit the resolution
sudo etckeeper commit "drift remediation: ssh config restored to baseline"

Troubleshooting / gotchas

Playbook is "working" but second run shows changes. Almost always one of: shell:/command: without creates: or changed_when:, lineinfile with a regex that matches multiple lines, copy with content that renders differently each time (timestamps). See references/idempotency-guide.md.
ansible-playbook --check doesn't catch everything. Some modules (notably command/shell) are conservative in check mode. They report "would run" regardless of actual state. Use creates: / removes: for stronger check-mode accuracy.
etckeeper fails on a large commit. The initial commit can be huge. If git complains, configure http.postBuffer or commit in stages.
local_action with become: yes prompts for a password. Use ansible_become_password from Vault, or run the playbook with --ask-become-pass.
Ansible installed via apt is too old for a module you need. Switch to pipx install ansible for the latest, or enable the Ansible PPA.
Handlers don't fire on a failed task. If a task fails mid-playbook, its notify: handlers are skipped. Use force_handlers: yes in the play if you need them to run anyway.

References

references/ansible-patterns.md — full Ansible reference: idempotent modules, roles, templates, Jinja2, 3 complete playbook examples.
references/drift-detection.md — etckeeper, Ansible check mode, AIDE, cron-scheduled checks, remediation workflow.
references/idempotency-guide.md — common idempotency mistakes, how to fix them, CI enforcement.
Book: Linux System Administration for the 2020s — idempotency, drift, cattle not pets.
Ansible docs: https://docs.ansible.com/

Optional fast path (when sk-* scripts are installed)

Running sudo install-skills-bin linux-config-management installs:

Task	Fast-path script
Compare key configs/packages vs git-tracked state	`sudo sk-drift-check`
Run an Ansible playbook in check mode with clean summary	`sudo sk-ansible-dry-run --playbook <file>`
Initialize and verify /etc tracking, stage + commit	`sudo sk-etc-track [--commit]`

These are optional wrappers around ansible-playbook, etckeeper, and git.

Scripts

This skill installs the following scripts to /usr/local/bin/. To install:

sudo install-skills-bin linux-config-management

Script	Source	Core?	Purpose
sk-drift-check	scripts/sk-drift-check.sh	no	Compare key config files and package list against git-tracked declared state; report drift with diffs.
sk-ansible-dry-run	scripts/sk-ansible-dry-run.sh	no	Run an Ansible playbook in `--check --diff` mode against localhost with a clean summary of would-be changes.
sk-etc-track	scripts/sk-etc-track.sh	no	Initialize git tracking for `/etc`, verify it's clean, optionally auto-stage + commit.

About this skill

Distro support

Use when

Do not use when

Required inputs

Workflow

Quality standards

Anti-patterns

Outputs

References

Evidence Produced

Capability contract

Degraded mode

Decision rules

Worked example

When to use

When NOT to use

Standing rules

Quick reference — manual commands

Install Ansible

Local-only Ansible (managed host is localhost)

Check mode tags and facts

Tracking /etc in git (etckeeper)

Comparing declared vs actual

Typical workflows

Workflow: Adopting Ansible on an existing server

Workflow: Weekly drift check

Workflow: "Is my playbook idempotent?"

Workflow: Drift remediation

Troubleshooting / gotchas

References

Optional fast path (when sk-* scripts are installed)

Scripts