linux-network-admin

Linux Network Administration

Distro support

Two-family skill. Diagnostics (ip, ss, ping, dig, resolvectl, traceroute) are identical on both families. The difference is persistent configuration: Debian/Ubuntu use Netplan; the RHEL family (Fedora, RHEL, CentOS Stream, Rocky, Alma, Oracle) uses NetworkManager / nmcli — Netplan does not exist on RHEL. Full detail: references/networkmanager-reference.md.

Concept	Debian/Ubuntu	RHEL family
Persistent config tool	Netplan (`/etc/netplan/*.yaml`)	NetworkManager (`nmcli`)
Config storage	`/etc/netplan/`	`/etc/NetworkManager/system-connections/`
Set static IP	edit YAML	`nmcli con mod <c> ipv4.addresses …`
Apply	`netplan try` / `netplan apply`	`nmcli con up <c>`
Set DNS	`nameservers:` in YAML	`nmcli con mod <c> ipv4.dns "…"`
Diagnostics (ip/ss/dig)	identical	identical
Time sync daemon	`systemd-timesyncd`	`chronyd` (`chronyc sources`)

Over SSH, RHEL has no netplan try auto-rollback — schedule a safety reset (see the reference). In sk-* scripts gate config tooling on the detected family via common.sh. See references/networkmanager-reference.md and docs/multi-distro/plan.md.

Use when

Managing interfaces, routes, netplan, DNS resolution, NTP, or reachability from the server side.
Diagnosing why a host cannot reach another host, port, or resolver.
Making non-firewall network changes on Ubuntu/Debian servers.

Do not use when

The task is firewall policy or TLS; use linux-firewall-ssl.
The task is authoritative DNS service configuration; use linux-dns-server.

Required inputs

Artefact	Source	Required?	If absent
Symptom, source host/interface, destination/address/port, and expected path	Operator and service design	required	Return a discovery checklist; do not guess topology.
Current addresses, links, routes, resolver, listeners, and persistent config	Target host	required for diagnosis/change	Mark the respective layer `not assessed`.
Console/recovery access and downtime authority	Change owner	required for persistent remote change	Stop before applying Netplan or NetworkManager changes.

Workflow

Inspect current interface, route, DNS, and reachability state first.
Choose the matching workflow below for connectivity, port reachability, VLAN, or time-sync work.
Apply the smallest safe change and validate immediately.
Confirm the network path behaves as expected after the change.
Stop if the owning interface/connection, console or timed rollback, expected path, or change authority is unresolved.
Recover by reverting the persistent connection/config through the console or timed fallback, then restore and verify the prior route and reachability.

Quality standards

Prefer observation before mutation.
Treat persistent network changes as high-risk and validate them carefully.
Distinguish local host issues from remote service issues.

Anti-patterns

Applying persistent network config without identifying the live connection/interface. Fix: map link, address, route, and configuration owner first.
Using only ping. Fix: test DNS, route, neighbour, TCP/UDP port, listener, and application path as relevant.
Applying a remote address/route change without rollback. Fix: use console or timed rollback and preserve the prior config.
Editing generated resolver files directly. Fix: change NetworkManager, systemd-resolved, or the owning configuration source.
Mixing firewall and interface changes. Fix: isolate layers and route firewall work to linux-firewall-ssl.

Outputs

Artefact	Consumer	Acceptance condition
Layered diagnosis or approved config change	Network/system operator	Identifies failing layer with observed link, route, DNS, port, and time evidence.
Recovery-safe change record	On-call operator	Contains before/after config, validation, rollback trigger, and console path.
End-to-end verification	Service owner	Expected name, address, route, port, and application response work from the intended source.

References

references/netplan-reference.md
references/networkmanager-reference.md
references/diagnostics-tree.md
references/appliance-vlan-config.md — appliance-side VLAN config (UniFi, pfSense/OPNsense, MikroTik) — the switch/router/AP counterpart to this skill's own host-side netplan VLAN coverage.

Evidence Produced

Artefact	Acceptance condition
Network-change evidence	Includes before/after link, address, route, resolver state, targeted probes, config validation, rollback readiness, and end-to-end checks.

Capability contract

Diagnosis defaults to read-only. Read/execute access is required for host inspection. Persistent address, route, VLAN, DNS, or time changes require explicit authority and a recovery path; firewall and remote-provider changes require separate scope.

Degraded mode

Without host access, provide a layer-by-layer probe plan and qualify all conclusions. Without console or timed rollback, do not apply remote persistent changes that could sever access.

Decision rules

Choice	Action	Failure or risk avoided
Link/address missing	Fix interface ownership/config first	Misdiagnosing higher layers.
Route and IP work but name fails	Inspect resolver path	Unnecessary route/firewall edits.
Host reaches destination but service port fails	Inspect listener/path, then firewall skill	Treating application refusal as routing failure.

Worked example

For a remote AlmaLinux host that lost a static route after reboot, capture ip and nmcli state, identify the owning connection, prepare a timed rollback, add the persistent route, reactivate safely, then prove the route, destination port, and application response before cancelling rollback.

This skill is self-contained. Diagnostic commands below (ip, ss, dig, ping, mtr, resolvectl) are standard on both families. Persistent config differs by family: netplan on Debian/Ubuntu, nmcli (NetworkManager) on the RHEL family — see references/networkmanager-reference.md. Time sync is systemd-timesyncd or chrony (chronyc) depending on the distro default. The sk-* scripts in the Optional fast path section are convenience wrappers — never required.

This skill owns everything about how a server talks to the network below the firewall and above the application layer: interfaces, addresses, routes, name resolution from the server's perspective, and time sync.

It does not own:

Firewall rules — linux-firewall-ssl.
Authoritative DNS serving (bind9/unbound) — linux-dns-server.
Mail / MX records — linux-mail-server.
Deep packet capture for incident triage — linux-troubleshooting.

Informed by Linux Network Administrator's Guide (translated from legacy ifconfig/route to modern ip/ss) and the Canonical Ubuntu Server Guide (netplan, systemd-networkd, systemd-resolved).

When to use

"The server can't reach the internet" / "can't resolve DNS" / "bad gateway."
Adding a new network interface, bridge, bond, or VLAN.
Changing an IP address or default route on a live server.
Validating netplan YAML before applying it.
Testing whether a port is reachable from the server.
Verifying NTP sync is healthy.
Configuring VLANs on the appliance side (UniFi, pfSense/OPNsense, MikroTik) rather than the host's own NIC — see references/appliance-vlan-config.md.

When NOT to use

Opening or closing firewall ports — use linux-firewall-ssl.
Debugging why a remote service can't reach this server when the firewall is the cause — use linux-firewall-ssl.
Mail delivery problems — use linux-mail-server.

Standing rules

Netplan is the source of truth for configuration. Never edit /etc/network/interfaces on a modern Ubuntu server. All changes go through /etc/netplan/*.yaml.
Always sudo netplan try before sudo netplan apply. try has a 120s revert timeout — if connectivity breaks, it rolls back.
DNS lives in systemd-resolved. /etc/resolv.conf is a symlink — never edit it directly; set resolvers via netplan's nameservers.addresses.
Test DNS from the server itself, not from your laptop. A cached local resolver on your laptop will lie. Query both the server's own resolver and a public one to distinguish.
Modern tools only. ip addr not ifconfig. ss -tulnp not netstat. ip route not route -n. resolvectl not nslookup for systemd-resolved state.
Confirm before applying route changes over SSH. A broken default route = no way back in.
Change-window verification is before/after, matched to the exact change, not a generic re-check. netplan generate (parse), netplan try (apply with revert), then re-read the specific interface/route/DNS state that changed and compare to the pre-change baseline before confirming. This is the same discipline cisco-ios-patterns applies to router/switch changes — capture state, apply the smallest change, re-read and diff, only then persist.

Quick reference — manual commands

Inspect current state

ip -c addr                                    # all interfaces + addresses
ip -c link                                    # link state, MAC, MTU
ip -c route                                   # routing table
ip route get 8.8.8.8                          # which iface will this go out?
ss -tulnp                                     # listening ports with owning process
ss -tan state established                     # active connections
nmcli device status                           # if NetworkManager is in use
resolvectl status                             # DNS resolvers per link
resolvectl query example.com                  # resolve via systemd-resolved

Test connectivity

ping -c 3 8.8.8.8                             # L3 to the public internet
ping -c 3 $(ip route | awk '/default/{print $3; exit}')   # gateway
mtr -c 10 --report example.com                # path + loss + latency
traceroute example.com                        # hop-by-hop (apt install traceroute)

# TCP reachability without nmap:
(echo > /dev/tcp/example.com/443) 2>/dev/null && echo OK || echo FAIL

# UDP reachability (requires nc):
nc -u -z -v example.com 53

Resolve DNS correctly

# Resolver 1: what systemd-resolved returns (what applications actually see)
resolvectl query example.com
resolvectl query -p dns -t A example.com

# Resolver 2: bypass the local resolver — query a public one directly
dig @1.1.1.1 example.com +short
dig @8.8.8.8 example.com +short

# Reverse DNS:
dig -x 8.8.8.8 +short

# If 1 and 2 disagree, your local resolver (systemd-resolved or /etc/hosts)
# is the problem, not upstream DNS.

Edit netplan

sudo nano /etc/netplan/01-linux-skills.yaml
sudo chmod 600 /etc/netplan/*.yaml            # contains secrets for wifi

# Generate backend config without applying:
sudo netplan generate

# Try with 120s auto-revert — use this over SSH:
sudo netplan try

# Apply permanently (use only after `try` succeeds):
sudo netplan apply

Full netplan YAML syntax, 8 worked examples, renderers, policy routing, bonds, bridges, VLANs — see references/netplan-reference.md.

Check NTP sync

# chrony (Ubuntu Server default):
chronyc tracking                              # current offset, stratum, leap
chronyc sources -v                            # peers, reachability
chronyc makestep                              # force immediate step correction

# systemd-timesyncd (Ubuntu Desktop default):
timedatectl show-timesync --all
timedatectl status

Typical workflows

Workflow: "The server can't reach the internet"

Walk the diagnostics tree in references/diagnostics-tree.md. Condensed version:

# 1. Link up?
ip -c link | grep -E "state (UP|DOWN)"

# 2. Address assigned?
ip -c addr show dev <iface>

# 3. Default route present?
ip route | grep default

# 4. Gateway reachable?
ping -c 3 $(ip route | awk '/default/{print $3; exit}')

# 5. External reachable by IP (skips DNS)?
ping -c 3 1.1.1.1

# 6. DNS resolving?
resolvectl query example.com
dig @1.1.1.1 example.com +short   # bypass local resolver to confirm

# 7. If 5 works but 6 doesn't → DNS problem. Check `resolvectl status`
#    and `cat /run/systemd/resolve/resolv.conf`.
# 8. If 4 works but 5 doesn't → gateway or upstream ISP problem.
# 9. If 3 is missing → fix default route via netplan.

Workflow: "Can this server reach that port on that host?"

# Local listen check (is it OUR firewall or NOT OUR box?):
ss -tulnp | grep ':<port>'

# TCP handshake from this server:
(echo > /dev/tcp/db.internal/5432) 2>/dev/null && echo OK || echo FAIL

# Path investigation if it fails:
mtr -c 10 -T -P 5432 --report db.internal    # TCP mtr to the exact port
traceroute -T -p 5432 db.internal

# From the server, check the remote's open ports if you have access:
ssh admin@db.internal 'ss -tulnp | grep 5432'

Workflow: "Adding a new VLAN on a trunk"

# 1. Edit netplan (see references/netplan-reference.md for VLAN stanza)
sudo nano /etc/netplan/01-linux-skills.yaml

# 2. Validate and dry-run
sudo netplan generate                         # parses YAML; errors here = bad file
sudo netplan try                               # apply with 120s revert

# 3. Verify the new interface:
ip -c link show dev vlan100
ip -c addr show dev vlan100

# 4. Confirm to persist (type Enter in the `try` session)

Workflow: "NTP sync is drifting"

chronyc tracking                              # look at System time offset
chronyc sources -v                            # any peers reach 0?
chronyc makestep                              # immediate step (use once, then investigate)

# If all sources are unreachable, check:
ss -tunap | grep 123                          # NTP port 123/udp
ping -c 3 pool.ntp.org
sudo systemctl status chrony
journalctl -u chrony -n 50 --no-pager

Troubleshooting / gotchas

/etc/resolv.conf is a symlink. Never edit it. Changes vanish on next boot. Always configure via netplan nameservers.addresses or systemd-resolved drop-ins.
netplan apply can cut your SSH. Use netplan try first. If you're desperate and have no console, run this to auto-revert: sudo timeout 60 bash -c 'netplan apply; sleep 55; ip link set <iface> down'. Ugly but survives a bad apply.
Two DNS servers both "work" but one is wrong. systemd-resolved picks per-link DNS. If multiple interfaces have nameservers, it may split-query. resolvectl status shows exactly which resolver is consulted for which domain.
ifconfig still "works" on old servers. It's provided by net-tools and frozen. Don't rely on anything it reports for bonded, VLAN, or aliased interfaces — it truncates silently.
Policy routing. If packets from one interface are leaving via another, you need routing-policy rules (see the reference file). The symptom is "ping works, TCP doesn't" or asymmetric routing.
MTU mismatches on tunnels. PMTU discovery is often broken by firewalls that drop ICMP. Symptom: SSH connects but a large file transfer hangs. Lower MTU to 1400 on the tunnel interface.

References

references/netplan-reference.md — complete netplan YAML syntax and 8 worked examples.
references/networkmanager-reference.md — NetworkManager / nmcli persistent config for the RHEL family.
references/diagnostics-tree.md — symptom-driven decision tree for networking problems.
Book: Linux Network Administrator's Guide (Kirch & Dawson, 2nd ed.) — routing, resolver, subnetting fundamentals. Translate ifconfig/route to ip.
Book: Ubuntu Server Guide (Canonical, Focal) — netplan, networkd, resolved chapters.
Man pages: netplan(5), ip(8), ss(8), resolvectl(1), chronyc(1).

Optional fast path (when sk-* scripts are installed)

Running sudo install-skills-bin linux-network-admin installs the following convenience wrappers. They are never required — the manual commands above are always the source of truth.

Task	Fast-path script
One-screen report: interfaces, addresses, default gateway, DNS, ports	`sudo sk-net-status`
Validate netplan YAML, `try` with revert, apply on confirm	`sudo sk-netplan-apply`
Test TCP/UDP port reachability with traceroute on failure	`sudo sk-port-check --target <h> --port <n>`
Forward + reverse DNS lookup against local and public resolvers	`sudo sk-dns-check <domain>`
NTP sync state, offset, peers	`sudo sk-ntp-sync`

Scripts

This skill installs the following scripts to /usr/local/bin/. To install:

sudo install-skills-bin linux-network-admin

Script	Source	Core?	Purpose
sk-net-status	scripts/sk-net-status.sh	no	One-screen report: interfaces, addresses, default gateway, DNS servers, open ports, link state.
sk-netplan-apply	scripts/sk-netplan-apply.sh	no	Validate netplan YAML, run `netplan try` with timeout rollback, confirm, then `apply`.
sk-port-check	scripts/sk-port-check.sh	no	Test TCP/UDP port reachability from this server to a target, with traceroute on failure.
sk-dns-check	scripts/sk-dns-check.sh	no	Forward + reverse DNS lookup against local resolver, systemd-resolved, and a public resolver; flags mismatches.
sk-ntp-sync	scripts/sk-ntp-sync.sh	no	Report chrony/timesyncd state, offset, peers, last successful sync.

About this skill

Distro support

Use when

Do not use when

Required inputs

Workflow

Quality standards

Anti-patterns

Outputs

References

Evidence Produced

Capability contract

Degraded mode

Decision rules

Worked example

When to use

When NOT to use

Standing rules

Quick reference — manual commands

Inspect current state

Test connectivity

Resolve DNS correctly

Edit netplan

Check NTP sync

Typical workflows

Workflow: "The server can't reach the internet"

Workflow: "Can this server reach that port on that host?"

Workflow: "Adding a new VLAN on a trunk"

Workflow: "NTP sync is drifting"

Troubleshooting / gotchas

References

Optional fast path (when sk-* scripts are installed)

Scripts