docker-troubleshooting

Docker Troubleshooting — Problem diagnosis and debugging

Systematic approach to diagnosing and resolving common Docker issues.

When to Use

ALWAYS use this skill when the user mentions:

"docker troubleshooting", "docker debug", "container issues"
"container startup failure", "exit code", "crash"
"OOM", "disk full", "disk full"
"network unreachable", "network issues"
"docker daemon", "daemon not running"

Diagnostic Framework

1. Check container state    → docker ps -a, docker inspect
2. Check logs               → docker logs --tail 100
3. Check resources          → docker stats, docker system df
4. Check exit code          → docker inspect --format='{{.State.ExitCode}}'
5. Interactive debug        → docker run --rm -it image sh

Container Startup Failure

Exit Code Reference

Code	Meaning	Action
0	Success	—
1	Application error	Check `docker logs`
125	Docker daemon error	Check daemon status
126	Command cannot execute	Check permissions
127	Command not found	Check CMD/ENTRYPOINT path
137	SIGKILL (OOM or docker kill)	Check memory limits
139	SIGSEGV (segfault)	Application bug
143	SIGTERM (graceful stop)	Health check timeout?

# Quick diagnosis
docker ps -a | grep Exited
docker inspect --format='{{.State.ExitCode}} {{.State.Error}}' container
docker logs container

Disk Space Exhaustion

# Check disk usage
docker system df

# What's using space?
docker system df -v   # Verbose breakdown

# Clean up
docker system prune -a     # Remove all unused: images, containers, volumes, networks
docker image prune -a      # Unused images only
docker volume prune        # Unused volumes only

# Log rotation (prevent future issues)
docker run --log-opt max-size=10m --log-opt max-file=3 myapp

OOM (Out of Memory)

# Check if container was OOM-killed
docker inspect --format='{{.State.OOMKilled}}' container

# Check memory usage
docker stats --no-stream

# Solution: increase memory limit
docker update --memory=1g --memory-swap=1g container

Network Connectivity

# Check network configuration
docker network inspect bridge
docker exec container ip addr show
docker exec container ping other-container

# DNS check
docker exec container nslookup other-container
docker exec container cat /etc/resolv.conf

# Host networking check
iptables -t nat -L DOCKER -n   # Port forwarding rules

Permission Issues

# Bind mount permission denied
# Solution 1: Match UID
docker run -u $(id -u):$(id -g) -v $PWD:/app myapp

# Solution 2: Fix permissions in entrypoint
# In entrypoint script: chown -R appuser:appgroup /data

# Docker socket permission
sudo usermod -aG docker $USER   # Add user to docker group
newgrp docker                    # Refresh group (or re-login)

Workflow — Troubleshooting process

Step 1: Check status: docker ps -a → Confirm exit code and OOMKilled Step 2: Check logs: docker logs --tail 100 <container> Get error information Step 3: In-depth inspection: docker inspect → Check network/mounts/resource limits Step 4: Interactive debugging: docker run -it --rm --entrypoint sh <image> Enter troubleshooting Step 5: Resource troubleshooting: docker stats + docker system df Check CPU/memory/disk

Gotchas — Common Pitfalls

docker system prune is destructive: Removes ALL unused objects. Don't run blindly in production. → Recovery: Use docker system df first to preview; add --filter "until=24h" for safe cleanup; never run prune -a without confirmation.
Logs fill disk silently: JSON-file driver has NO rotation by default. A noisy app can fill the disk in hours. → Recovery: docker run --log-opt max-size=10m --log-opt max-file=3 app; existing: truncate -s 0 $(docker inspect -f '{{.LogPath}}' myapp).
OOMKilled=TRUE but no error in logs: SIGKILL can't be logged. → Recovery: Check docker inspect -f '{{.State.OOMKilled}}' myapp; increase memory: docker update --memory 512m myapp.
Exited (0) but service not working: Application started but crashed after health check passed. → Recovery: docker logs --tail 100 myapp; check if app listens on 0.0.0.0 (not 127.0.0.1); verify with docker exec myapp netstat -tlnp.

Boundary — Scope of capabilities (applicable and non-applicable scenarios)

Category	Scenario	Description
✅ Can do	Container startup failure troubleshooting	Exit code analysis + docker logs + inspect
✅ Can do	Disk/log/resource issues	docker system df + prune + log rotation
✅ Can do	OOM/CPU troubleshooting	docker stats + --memory limit + OOMKilled
⚠️ Conditional	Packet-level network analysis	Requires tcpdump/Wireshark + network knowledge
⚠️ Conditional	Kernel-level issues	Requires Linux kernel debugging skills
❌ Out of scope	Fixing application code bugs	Developer fixes code
❌ Out of scope	Production environment deployment	Use `docker-production`
❌ Out of scope	Security vulnerability remediation	Use `docker-security` + `docker-scout`

When NOT to Use This Skill

❌ Skip	✅ Use Instead
Setting up production	`docker-production`
Security issues	`docker-security`
Performance tuning	`docker-run` (resource limits)
Docker basics	`docker-basics`

Security & Stability

docker exec into production containers can expose sensitive data. Prefer log-based debugging.
docker system prune -a removes ALL unused images and volumes — use with caution in production.
Inspecting containers with --privileged debug tools can bypass security controls.
Never expose Docker socket (/var/run/docker.sock) to containers for debugging — it grants root access.
Rotate debug logs — they may contain sensitive information. Use log drivers with TTL.

📚 Official documentation reference

Documentation	Address
Docker Engine troubleshooting	https://docs.docker.com/engine/troubleshoot/
docker logs command	https://docs.docker.com/reference/cli/docker/container/logs/
docker inspect	https://docs.docker.com/reference/cli/docker/inspect/
Docker CLI reference	https://docs.docker.com/reference/cli/docker/
Daemon configuration	https://docs.docker.com/reference/cli/dockerd/
Common issues FAQ	https://docs.docker.com/faq/

🧭 Docker Skills Journey

📍 You are here: docker-troubleshooting — Problem diagnosis

← Previous: docker-production