SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsgke-ai-troubleshooting-jobset-interruption

gke-ai-troubleshooting-jobset-interruption

General & other

View repository

Updated 10-15 21:54

About this skill

GKE JobSet Interruption Troubleshooting

Use this skill to systematically diagnose and resolve JobSet interruptions, restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.

⚠️ Prerequisites

  • JobSet metrics package must be enabled in kube-state-metrics for your cluster (see KSM JobSet Metrics).
  • Cloud Logging and Cloud Monitoring must be enabled for the Google Cloud Project.

🔍 Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

To begin troubleshooting, acquire the following context from the user:

  • Project ID (e.g., customer-ai-project-123)
  • Cluster Name (e.g., tpu-cluster-prod)
  • Workload Name (JobSet Name) (e.g., llama3-70b-training)
  • Workload Namespace (e.g., default)
  • Issue Time (e.g., 2026-05-20T08:15:00Z)

Time Handling Rules

  1. Reject Relative Time: If the user says "X minutes ago" or "just now", stop and request the exact timestamp or specific time window.
  2. Window Calculation: If the user provides a single timestamp T, calculate the query window as [T - 30m] to [T + 30m].
    • Let Start_Time = T - 30m
    • Let End_Time = T + 30m

Step 1: Identify JobSet Restarts and Attempts [Low Risk]

Verify if the JobSet is experiencing restart loops and determine the frequency of restarts.

Option A: Visual Chart (MQL) - restarts

  • Tool to use: monitoring_time_series_chart
  • MQL Query: 
    fetch prometheus_target
| metric 'prometheus.googleapis.com/kube_jobset_restarts/gauge'
| filter resource.cluster_name == '<cluster_name>' && metric.jobset_name == '<workload_name>'
| align next_older(1m)
| every 1m
| group_by [metric.jobset_name], [val: max(value)]

Option B: PromQL Query - restarts

  • Method: Execute via shell command (requires BypassSandbox: true).
  • Command Template: 
    curl -sSf -H "Authorization: Bearer \$(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v1/projects/<project_id>/location/global/prometheus/api/v1/query?query=kube_jobset_restarts%7Bjobset_name%3D%22<workload_name>%22%2Ccluster%3D%22<cluster_name>%22%7D"
  • Logic: A non-zero or increasing value for restarts indicates that the JobSet is being actively restarted by the controller due to worker failure or interruption.
  • Automation: Proceed to Step 2 automatically after reporting findings.

Step 2: Inspect Nodepool Interruptions [Low Risk]

Determine if the JobSet restarts were triggered by physical nodepool-level events (such as spot preemptions, maintenance, or host terminations).

A. Metrics Query (Nodepool Interruption Counts)

Option A: Visual Chart (MQL) - interruptions
  • Tool to use: monitoring_time_series_chart
  • MQL Query: 
    fetch k8s_node_pool
| metric 'kubernetes.io/node_pool/interruption_count'
| filter cluster_name == '<cluster_name>'
| align next_older(10m)
| every 10m
| group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]
Option B: PromQL Query - interruptions
  • Method: Execute via shell command (requires BypassSandbox: true).
  • Command Template: 
    curl -sSf -H "Authorization: Bearer \$(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v1/projects/<project_id>/location/global/prometheus/api/v1/query?query=sum%20by%20%28interruption_type%2C%20interruption_reason%2C%20node_pool_name%2C%20cluster_name%29%20%28avg_over_time%28%7B__name__%3D%22kubernetes.io%2Fnode_pool%2Finterruption_count%22%2C%20monitored_resource%3D%22k8s_node_pool%22%2C%20cluster_name%3D%22<cluster_name>%22%7D%5B10m%5D%29%29"

B. Log Query (Nodepool Life Events)

  • Tool to use: query_logs
  • LQL Log Filter Template: 
    resource.type="gke_nodepool"
AND resource.labels.cluster_name="<cluster_name>"
AND timestamp >= "<Start_Time>"
AND timestamp <= "<End_Time>"
  • Logic:
    • PreemptionEvent: Spot VMs were preempted, or node was scale-down.
    • MaintenanceEvent: Node pool updated or Google scheduled maintenance.
    • TerminationEvent: Serious host failures. Check interruption_reason or logs for host issues.
  • Automation: Proceed to Step 3 automatically.

Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]

Correlate node readiness failures with physical host VMs to see if a single faulty host repeatedly fails coordinator pods.

A. Metrics Query (Node Ready Status Check)

Option A: Visual Chart (MQL) - node status
  • Tool to use: monitoring_time_series_chart
  • MQL Query: 
    fetch k8s_node
| metric 'kubernetes.io/node/status_condition'
| filter cluster_name == '<cluster_name>' && metric.condition == 'Ready' && metric.status == 'False'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]
Option B: PromQL Query - node status
  • Method: Execute via shell command (requires BypassSandbox: true).
  • Command Template: 
    curl -sSf -H "Authorization: Bearer \$(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v1/projects/<project_id>/location/global/prometheus/api/v1/query?query=sum%20by%20%28status%2C%20condition%2C%20node_pool_name%29%20%28%7B__name__%3D%22kubernetes.io%2Fnode%2Fstatus_condition%22%2C%20monitored_resource%3D%22k8s_node%22%2C%20cluster_name%3D%22<cluster_name>%22%2C%20condition%3D%22Ready%22%2C%20status%3D%22False%22%7D%29"

B. Metrics Query (Node-to-Host Metadata Topology Correlation)

  • Tool to use: monitoring_time_series_chart
  • MQL Query: 
    fetch k8s_node
| metric 'kubernetes.io/node/cpu/total_cores'
| filter cluster_name == '<cluster_name>'
| align next_older(1m)
| every 1m
| group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]

C. Log Query (Node Fault Logs)

  • Tool to use: query_logs
  • LQL Log Filter Template: 
    resource.type="k8s_node"
AND resource.labels.cluster_name="<cluster_name>"
AND (textPayload:"host error" OR textPayload:"kernel panic" OR textPayload:"hardware failure" OR textPayload:"NodeNotReady")
AND timestamp >= "<Start_Time>"
AND timestamp <= "<End_Time>"
  • Logic: Identify if specific nodes are unhealthy (Ready=False or Unknown) and correlate them to their GCE physical host ID via metadata.user.gce_topology_host. Check if the same host is repeatedly failing.
  • Automation: Proceed to Step 4 automatically.

Step 4: Inspect Pod and Worker / Container Failures [Low Risk]

Analyze pod status phases and retrieve coordinator worker logs to identify application-level crashes or network deadlocks.

A. Metrics Query (Pod Lifecycle Phases)

Option A: Visual Chart (MQL) - pod phase
  • Tool to use: monitoring_time_series_chart
  • MQL Query: 
    fetch k8s_pod
| metric 'kubernetes.io/pod/status/phase'
| filter cluster_name == '<cluster_name>' && pod_name ==~ '<workload_name>.*'
| align next_older(10m)
| every 10m
| group_by [metric.phase], [val: count()]
Option B: PromQL Query - pod phase
  • Method: Execute via shell command (requires BypassSandbox: true).
  • Command Template: 
    curl -sSf -H "Authorization: Bearer \$(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v1/projects/<project_id>/location/global/prometheus/api/v1/query?query=sum%20by%20%28phase%29%20%28avg_over_time%28%7B__name__%3D%22kube_pod_status_phase%22%2C%20cluster%3D%22<cluster_name>%22%2C%20pod%3D~%22<workload_name>.*%22%7D%5B10m%5D%29%29"

B. Metrics Query (Unschedulable Pod Count)

  • Tool to use: monitoring_time_series_chart
  • MQL Query: 
    fetch k8s_pod
| metric 'kubernetes.io/pod/status/unschedulable'
| filter cluster_name == '<cluster_name>' && pod_name ==~ '<workload_name>.*'
| align next_older(10m)
| every 10m
| group_by [pod_name], [val: max(value)]

C. Log Query (Worker Container Logs)

  • Tool to use: query_logs
  • LQL Log Filter Template: 
    resource.type="k8s_container"
AND resource.labels.cluster_name="<cluster_name>"
AND labels."k8s-pod/jobset_sigs_k8s_io/jobset-name"="<workload_name>"
AND timestamp >= "<Start_Time>"
AND timestamp <= "<End_Time>"
  • Logic:
    1. Check the pod timeline to spot pending or unschedulable pods.
    2. Use the worker container logs to analyze worker 0 in slice 0 (coordinator) for NCCL timeouts, collective communication issues, or MegaScale hangs.
  • Automation: Proceed to Resolution.

🛠️ Resolution Workflow

Resolution 1: Preemption & Autoscaling Optimizations [Low Risk]

If Step 2 showed high preemption counts on Spot VMs:

  • Action: Suggest switching critical long-running training workloads to GKE Reserved/On-Demand VMs or utilizing Compact Placement Policies to minimize defragmentation interruptions.
  • Justification: Eliminates spot-market preemptions and reduces training restarts.

Resolution 2: Quarantine Faulty Host VMs [High Risk]

If Step 3 identified a specific host ID (gce-topology-host) that consistently fails or triggers restarts across multiple attempts:

  • Action: Draft a recommendation to cordon/drain the GKE node, delete the underlying GCE VM instance to trigger instance recreation, and open a support ticket with Google Cloud Support specifying the physical host ID.
  • Justification: GKE auto-repair will recreate the VM instance on healthy physical hardware, preventing infinite restart loops.
  • Automation: Stop and request explicit user confirmation before cordoning/draining any node.

📋 Copypaste Checklist

  • Acquire context and compute [T - 30m, T + 30m] window.
  • Query JobSet restart attempts.
  • Check Nodepool interruptions (spot preemptions vs. hardware terminations).
  • Query node-to-host mapping and check node logs for physical host errors.
  • Inspect pod timeline status and coordinator worker container logs.
  • Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM quarantining.