SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsdebug-node

debug-node

Testing & quality

View repository

Updated 10-16 01:33

About this skill

Debug Hetzner Node via Rescue Console

Overview

When a Hetzner Cloud server is unreachable (SSH hangs, provisioning stuck, cloud-init failure), this skill uses Hetzner's rescue mode to mount the node's filesystem and inspect everything from the outside — no working SSH required.

Usage

/debug-node

When invoked, ask for:

  1. The server name or IP (can be found from hcloud server list)
  2. What symptom they're seeing (SSH timeout, provisioning hang, etc.)

Prerequisites

  • hcloud CLI installed and configured with a valid token
  • The server must exist in Hetzner Cloud
hcloud server list

Leap Micro Filesystem Model

Leap Micro uses a transactional-update system on btrfs. This is the mental model for everything below.

Layer Writable? Persists reboot? Persists Hetzner snapshot?
/usr (snapshot) No (read-only) Yes Yes
/etc via transactional-update shell Yes (new snapshot) Yes (after reboot) Yes
/etc via direct edit on running system Yes (current snapshot overlay) Yes, unless a pending snapshot changed the same path Current-snapshot content only
/var (separate subvolume) Yes Yes Yes

Rule: Ordinary /etc edits persist across a normal reboot. However, after transactional-update creates a pending snapshot, concurrent edits to the current snapshot's /etc are not merged when the pending snapshot changed the same path. Put image-building and package-coupled /etc changes inside transactional-update --continue shell, or reboot into the pending snapshot before editing.

Packer build phases:

  1. Rescue mode: Write qcow2 to disk, reboot
  2. install_packages: Inside transactional-update — changes persist
  3. clean_up: Volatile overlay — /etc changes are lost in the Hetzner snapshot

Step 1: Identify the Server

hcloud server list -o columns=id,name,status,ipv4 | grep <pattern>

Step 2: Enable Rescue Mode & Reboot

hcloud server enable-rescue <SERVER_ID> --type linux64
hcloud server reboot <SERVER_ID>
sleep 30

Save the rescue root password from the output (usually key auth works, but just in case).

Step 3: SSH into Rescue

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@<SERVER_IP>

Step 4: Mount the Filesystem

Leap Micro / MicroOS (btrfs)

# Mount btrfs top-level
mount -o subvolid=5 /dev/sda3 /mnt

# List snapshots — highest number is active
ls /mnt/@/.snapshots/

Layout:

/mnt/@/.snapshots/N/snapshot/    latest active snapshot — /etc lives here
/mnt/@/root/                     /root home (/root/.ssh/authorized_keys)
/mnt/@/var/                      /var (logs, cloud-init state, journal)

Key: /etc is inside the snapshot. /var and /root are separate subvolumes at @/var and @/root.

Unsupported/custom ext4 images

mount /dev/sda1 /mnt

Step 5: Diagnostic Checklist

Set this once and use throughout:

SNAP=/mnt/@/.snapshots/N/snapshot   # replace N with highest snapshot number

5a. Cloud-Init Status

Start here — most provisioning failures trace back to cloud-init.

cat /mnt/@/var/lib/cloud/data/result.json
cat /mnt/@/var/lib/cloud/data/status.json
cat /mnt/@/var/lib/cloud/instance/datasource

# What Terraform actually sent
cat /mnt/@/var/lib/cloud/instance/user-data.txt
zcat /mnt/@/var/lib/cloud/instance/user-data.txt.i 2>/dev/null

# Logs
tail -100 /mnt/@/var/log/cloud-init.log
tail -100 /mnt/@/var/log/cloud-init-output.log

Expected: DataSourceHetzner, no errors. Watch for: Skipping modules — means cloud-init already ran for this instance-id.

Cloud-init facts on Hetzner + Leap Micro:

  • Datasource: DataSourceHetzner (metadata API)
  • Terraform's cloudinit_config → gzip+base64 multipart MIME → user_data
  • disable_root: false prevents cloud-init from disabling root but does NOT unlock a locked account
  • ssh_authorized_keys writes keys to /root/.ssh/authorized_keys

5b. SSH Keys

cat /mnt/@/root/.ssh/authorized_keys

Compare with your local pubkey. If missing, cloud-init failed to inject — check 5a logs.

5c. SSHD Configuration

Config loading order (first match wins):

1. /etc/ssh/sshd_config.d/40-kube-hetzner-authorized-keys-command.conf
2. /etc/ssh/sshd_config.d/50-cloud-init.conf
3. /etc/ssh/sshd_config.d/kube-hetzner.conf           (MaxAuthTries 2)
4. /usr/etc/ssh/sshd_config.d/40-suse-crypto-policies.conf
5. /usr/etc/ssh/sshd_config                            (UsePAM yes)

ls $SNAP/etc/ssh/sshd_config.d/
cat $SNAP/etc/ssh/sshd_config.d/*.conf
cat $SNAP/usr/etc/ssh/sshd_config
ls -la $SNAP/etc/ssh/ssh_host_*

5d. Account Status

grep '^root:' $SNAP/etc/shadow
Pattern Meaning SSH pubkey works?
root:*:... Unlocked, no password Yes
root:!*:... or root:!:... Locked No (PAM rejects with UsePAM yes)

This is fixed in packer (usermod -p '*' root inside transactional-update) with a cloud-init bootcmd safety net. If you see a locked account on a fresh node, the packer snapshot needs rebuilding.

5e. Journal Logs

journalctl -D /mnt/@/var/log/journal/ -u sshd --no-pager | tail -50
journalctl -D /mnt/@/var/log/journal/ -u k3s --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-server --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-agent --no-pager | tail -30

5f. Network

ls $SNAP/etc/NetworkManager/system-connections/
cat $SNAP/etc/NetworkManager/system-connections/*.nmconnection 2>/dev/null

5g. Kubernetes

cat $SNAP/etc/rancher/k3s/config.yaml 2>/dev/null
cat $SNAP/etc/rancher/rke2/config.yaml 2>/dev/null
cat /mnt/@/var/lib/rancher/k3s/server/token 2>/dev/null

5h. SELinux

cat $SNAP/etc/selinux/config
chroot $SNAP rpm -qa | grep -iE 'selinux|k3s|rke2'
grep -i 'avc:.*denied' /mnt/@/var/log/audit/audit.log | tail -50
journalctl -D /mnt/@/var/log/journal/ --no-pager | grep -i 'avc:.*denied' | tail -50

For workload denials, follow docs/selinux.md: collect the AVC lines, workload name/version, k3s/RKE2 distribution, OS image, and udica result before proposing upstream policy changes. Do not globally disable SELinux as the first answer; use per-pool selinux = false only as the last resort for a workload or nodepool that cannot run under policy.

Step 6: Apply a Fix

Edit files in the active snapshot ($SNAP), not in @/ base.

# Unlock root account (if locked)
sed -i 's/^root:!*/root:*/' $SNAP/etc/shadow

# Fix authorized_keys
mkdir -p /mnt/@/root/.ssh
echo "ssh-ed25519 AAAA..." > /mnt/@/root/.ssh/authorized_keys
chmod 700 /mnt/@/root/.ssh && chmod 600 /mnt/@/root/.ssh/authorized_keys

# Regenerate host keys
mount --bind /proc $SNAP/proc && mount --bind /sys $SNAP/sys && mount --bind /dev $SNAP/dev
chroot $SNAP ssh-keygen -A
umount $SNAP/proc $SNAP/sys $SNAP/dev

Note: Rescue-mode edits are immediate fixes. The proper long-term fix belongs in the packer template or cloud-init.

Step 7: Reboot to Normal

umount /mnt 2>/dev/null
exit

hcloud server disable-rescue <SERVER_ID>
hcloud server reboot <SERVER_ID>
sleep 60
ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP> 'echo ok'

Common Diagnoses

Symptom Likely Cause Check Fix
SSH timeout Firewall or network Hetzner firewall rules Open port 22
SSH "Connection refused" sshd not running Journal logs Fix sshd config syntax
SSH key rejected Keys not injected authorized_keys empty Check cloud-init logs
SSH "Too many auth failures" Agent offers too many keys MaxAuthTries 2 Use -o IdentitiesOnly=yes
SSH "unable to authenticate" Root locked, or key mismatch /etc/shadow, authorized_keys Rebuild packer snapshot
Provisioner hangs "Still creating" SSH can't connect All above Fix underlying SSH issue
Cloud-init skips modules Already ran for instance-id cloud-init.log Clean /var/lib/cloud/instance
k3s/rke2 not starting Config or SELinux Journal + audit.log Fix config or policy
Workload denied by SELinux Missing workload policy AVC lines in audit/journal Follow docs/selinux.md; try udica before disabling a pool
Network/subnet destroy hangs Autoscaler-created server outside Terraform state hcloud server list for cluster-name or kh-ci-* leftovers Delete only after control plane is dead, or scale autoscaler min_nodes = 0 first
/etc change vanished after transactional reboot Current and pending snapshots changed the same path Inspect transactional-update status and /var/lib/overlay Reapply in the active snapshot or move image-build changes into the transaction

Debugging SSH Manually

# Verbose with specific key (avoids agent key spray hitting MaxAuthTries 2)
ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP>

# In -vvv output:
# "Offering public key: ..."              → key was offered
# "Server accepts key: ..."               → success path
# "Authentications that can continue: ..."  → key was REJECTED
# "Too many authentication failures"       → agent sent too many keys

Pro Tips

  1. Mount with subvolid=5 — gets the real btrfs root, navigate to @/.snapshots/N/snapshot/
  2. Highest snapshot = active — that's where /etc lives
  3. /var is separate — logs and cloud-init are at /mnt/@/var/, not inside the snapshot
  4. Journal without a running system — journalctl -D /path/to/journal/
  5. Use -o IdentitiesOnly=yes — kube-hetzner sets MaxAuthTries 2
  6. Volatile overlay trap — if rescue shows different content than the live system did, it was running on a volatile overlay that never got committed
  7. After fixing packer, rebuild snapshots — verify build logs show changes inside the transactional-update output
  8. Rescue mode is non-destructive — you're just reading/writing files on the disk
  9. Destroy with the wrapper — for full-cluster teardown, run scripts/destroy.sh from the Terraform root; it retries only the known ingress-LB detach race and prints a read-only orphan report. Use scripts/cleanup.sh only as the forceful fallback; it treats the token's entire HCloud project as cluster-dedicated, so review the dry run first.