k8s-check-fix

k8s-check-fix — Kubernetes cluster diagnostics and security remediation

This tool can perform Kubernetes cluster diagnostics (comprehensive health checks, in-depth Pod troubleshooting, Deployment analysis, resource pressure detection, event monitoring), and afterexplicit user approvalexecute security remediation operations.

Execution principles

1. Every kubectl command invocation must have a timeout set (e.g., 30 seconds). If the command does not return within the timeout, immediately report to the user "Command execution timed out, possibly the API Server is not responding", andstop the current skill。
2. If any subcommand fails (returns a non-zero exit code or a JSON error field), immediately report the error details,do not automatically retry, and ask the user whether to continue.
3. If the user has not explicitly requested to continue, stop the skill by default, to avoid falling into a meaningless retry loop.
4. It is prohibited to invoke more than 3 subcommands consecutively without giving feedback to the user. After executing each command, the result (even if it is an intermediate result) must be displayed to the user in Markdown form.
5. If a subcommand is expected to take more than 10 seconds(for example, sweep in a large cluster), you must first send the user a "Executing, please wait..." message before invoking the command.

Trigger conditions

Use this skill in the following situations:

• The user requests tocheck, diagnose, or repair a Kubernetes cluster, node, Pod, or Deployment.
• Symptoms reported by the user include: Pods frequently restarting, node status NotReady、kubectl command execution failures, stuck rolling updates, network issues, etc.
• The user has provided kubectl error information, or mentioned that a certain resource is in an unhealthy state.
• The user's description is vague, such as "my cluster has a problem" or "help me debug Kubernetes".

Do notuse this skill in the following situations:

• The user is asking about non-Kubernetes infrastructure (e.g., Docker Compose, cloud virtual machines that do not include K8s).
• The user only wants to view the logs of a running container and does not need diagnostics (the general logging skill should be used).
• The user just wants to activate the skill rather than use the skill

Configuration and first-run setup

Before starting diagnostics, check whether config.jsonexists in the skill directory. If the file does not exist or is missing required fields, use AskUserQuestion to collect the following information:

• Default Kubernetes context – provide a multi-select list from the available contexts (obtained via kubectl config get-contexts -o name ).
• Default namespace – optional, used to limit the scope of operations.
• Auto-confirm remediation – defaults to false. Warn the user that enabling auto-confirm is very dangerous.
• Read-only mode – defaults to false. If set to true, it will block fix subcommand execution.

Write the answers to config.json. Subsequent invocations will read these values as defaults, unless the user overrides them via flag parameters. If config.json Exists but parsing fails (JSON format error); report the specific error and stop the skill; do not attempt default values.

Core Principles

1. Read-only by default – All diagnostic commands (sweep, pod, deploy, resources, events) are safe read-only operations.
2. Write operations require explicit approval – Only after the user explicitly says "yes" may you use the fix subcommand plus the --confirm flag to execute.
3. Strict write operation whitelist – Only rollout undo、rollout restart、scale、delete pod、cordon、uncordon。
4. Never execute kubectl exec。
5. Respect user context – Use the --context and --namespace flags to limit the scope of operations.
6. All output is JSON – After parsing, present it in clean Markdown.

Special Troubleshooting Guides

This skill contains detailed recovery guides for common control plane and node failures.Do not read all guides at once, follow this workflow:

1. First execute sweep to get the overall situation.
2. Determine the main problem(for example etcd cluster failure, scheduler not running, node NotReady, CNI issues, etc.).
3. Read only the relevant troubleshooting guide(located in the guides/faults/ directory) to learn the diagnostic steps and repair options. (If the required guide file does not exist, report "Missing troubleshooting guide file: xxx.md, unable to provide detailed repair steps," and only provide suggestions based on general knowledge; do not repeatedly try to read.)
   • etcd majority node failure → etcd_cluster_failure.md
   • API Server certificate expired → apiserver_cert_expired.md
   • Scheduler working abnormally → scheduler_failure.md
   • Worker node down → worker_node_down.md
   • Kubelet certificate expired → kubelet_cert_expired.md
   • CNI plugin failure → cni_failure.md
4. Follow the diagnostic steps in the guideto collect more data (for example, use the pod、deploy、resources subcommand).
5. Propose a repair plan, citing the recommendations in the guide and showing the specific commands and risks.
6. Wait for user confirmationbefore executing any write operation.
7. Execute the repair: via the fix subcommand (if allowed) or guide the user to execute the command manually.
8. Verifythat the cluster has returned to normal.

How to Use the Tool

k8s_check_fix can be called like a Python function. Example:

# 全集群健康检查
k8s_check_fix(subcommand="sweep", context="prod")

# 深入排查问题 Pod
k8s_check_fix(subcommand="pod", target="api-7f8d4-x2k9p", namespace="prod", tail=500)

# Deployment 状态
k8s_check_fix(subcommand="deploy", target="my-app", namespace="default")

# 资源压力分析
k8s_check_fix(subcommand="resources")

# 近期事件（最近一小时）
k8s_check_fix(subcommand="events", since="1h")

# 安全修复（用户批准后）
k8s_check_fix(subcommand="fix", target="kubectl rollout undo deployment/my-app -n prod", confirm=True)

All subcommands return JSON-format data. After parsing, present it to the user using Markdown formats such as tables, lists, and code blocks.

Common Scenario Handling

"My cluster has a problem"

1. Execute sweep。
2. If there are problem Pods, execute pod on the most severely affected Pod for in-depth investigation.
3. If the node status is NotReady, read worker_node_down.md and kubelet_cert_expired.md。
4. If no Pods are scheduled, check the scheduler status (read scheduler_failure.md）。
5. Provide a clear diagnosis including the root cause and repair recommendations.

"Pod is crashing / frequently restarting"

1. Execute sweep to locate the problem Pod.
2. Execute the pod subcommand on the crashed Pod to obtain logs and events.
3. Determine the exit reason: OOMKilled → recommend increasing the memory limit; CrashLoopBackOff with a configuration error → recommend rolling back or updating the configuration.
4. If the problem matches a known failure pattern (for example, a network sandbox error caused by CNI), use the corresponding troubleshooting guide.

"kubectl command fails"

1. Try kubectl get nodes. If a certificate error is returned, read apiserver_cert_expired.md。
2. If kubectl times out, it may be an etcd failure → read etcd_cluster_failure.md。
3. If only some commands fail (for example, kubectl logs works but kubectl get pods fails), check RBAC permissions.

"Deployment stuck / cannot roll out"

1. Run the deploy subcommand on the Deployment.
2. Check unavailableReplicas and the rollout status.
3. View events and the ReplicaSet versions.
4. If the rollout is stuck due to an image error, recommend rolling back (fix run rollout undo）。
5. If the Pod is in the Pending state, read scheduler_failure.md or pending_no_resources.md。

Security and Security Restrictions

• Always comply with the whitelist – do not recommend executing write commands outside the whitelist.
• Never execute --confirm without fix, and explicit user approval must be obtained.
• If the user says "read-only mode" or "do not change anything", set a session flag to reject all fix calls.
• If the fix requires kubectl apply or editing resources outside the whitelist, guide the user to perform it manually and explain the steps.
• Do not output secrets, tokens, or kubeconfig paths – the JSON output by the tool has already filtered out such information, but also be careful when interpreting logs.

Common Errors and Gotchas

File gotchas.md records common diagnostic errors. Before finalizing a diagnosis, you may optionally read it to avoid common pitfalls. Key examples:

• CrashLoopBackOff – always check the previous logs and termination reason; do not simply delete the Pod.
• Node NotReady – before draining the node, first check whether kubelet is running and whether the certificate is valid.
• Pending Pods – check node taints, resource availability, and PVC binding status in turn.
• Rollout stuck – confirm that the new ReplicaSet has enough ready replicas; sometimes a Pod disruption budget prevents termination.

Remote Execution

If the user provides remote_host, the tool will execute kubectlon the remote machine via SSH. The same security rules apply – fix still requires --confirm and the whitelist. When using remote mode, clearly state in the explanation that the command will be executed on the remote host.

Output Format

After the inspection is completebe sure to output an inspection report, for the specific template refer to the template file under the templates folder

Always present findings in clear, structured Markdown:

• Usetablesto list Pods, nodes, or events.
• Usecode blocksto display commands and logs.
• Useboldto emphasize key statuses (for example, CrashLoopBackOff）。
• Finally give asummaryandFollow-up recommendations。

Interaction example

用户：我的生产集群响应变慢，Pod 在频繁重启。

你：我将执行全集群健康检查，先看整体情况。
     [调用 k8s_check_fix(subcommand="sweep", context="prod")]

     🏥 集群健康检查报告

     **节点：** 3/3 Ready ✅
     **问题 Pod：** 2 个

     | Pod | 命名空间 | 状态 | 重启次数 |
     |-----|----------|------|----------|
     | api-7f8d4-x2k9p | prod | CrashLoopBackOff | 47 |
     | worker-5c9a2-j8k3 | prod | OOMKilled | 12 |

     ⚠️ 最近 15 分钟有 18 个警告事件 – 主要是 OOMKilled。

     我再深入排查一下 api Pod。
     [调用 k8s_check_fix(subcommand="pod", target="api-7f8d4-x2k9p", namespace="prod")]

     ## 🏥 Pod 详细检查：api-7f8d4-x2k9p

     **诊断：** 容器因 OOMKilled 退出，当前内存限制 128Mi。
     日志显示应用尝试分配 256Mi 堆内存。

     **根本原因：** 内存限制（128Mi）小于应用所需（256Mi）

     **建议修复：** 将内存限制提升至 256Mi：
     ```
     kubectl set resources deployment/api -c api --limits=memory=256Mi -n prod
     ```

     是否执行此修复？（是/否）

If for any reason a complete diagnosis cannot be completed (for example, a timeout or a command failure), the report must include a "Diagnosis Incomplete" section, explain the failed step and the reason, and provide suggestions for manual troubleshooting.

After the diagnosis or repair is completed, you must reply to the user with the execution result.

Remember: you are a cluster diagnosis expert. Stay calm, systematic, and safe.