log-analysis

Skill: log_analysis

Analyze a SLURM job log file for failure root-cause attribution and restart decisions using RestartAgent.

Command: python -m nvidia_resiliency_ext.attribution.restart_agent.cli

---

What it does

1. Reads the log file (UTF-8, falls back to latin-1).
2. Builds deterministic L0 evidence from training progress, failure lines, checkpoint state, and known restart-policy signals.
3. Optionally runs L1 model enrichment to explain the primary failure and related evidence.
4. Applies the restart-agent retry/recovery policy and returns a structured STOP or RESTART decision.

---

CLI

python -m nvidia_resiliency_ext.attribution.restart_agent.cli \
    /path/to/job.log \
    [--job-id JOB_ID] \
    [--cycle-id CYCLE_ID] \
    [--config examples/attribution/restart_agent.json] \
    [--llm-model MODEL] \
    [--llm-base-url URL] \
    [--llm-api-key-file PATH] \
    [--disable-l1] \
    [--summary]

Flag	Default	Description
log_path	required	Path to the job log file
--job-id	unset	Optional job id for request metadata and retry history
--cycle-id	unset	Optional cycle id for retry history
--config	unset	Versioned restart-agent config for configured model routes and retry policy
--llm-model	environment/default config	L1 enrichment model when --config is not used
--llm-base-url	environment/default config	OpenAI-compatible base URL when --config is not used
--llm-api-key-file	environment/default config	File containing the L1 API key
--disable-l1	off	Run deterministic-only analysis
--summary	off	Emit a compact human-readable summary to stderr in addition to JSON stdout

---

Programmatic API

from nvidia_resiliency_ext.attribution.restart_agent.l1 import LlmConfig, LlmEvidenceExtractor
from nvidia_resiliency_ext.attribution.restart_agent.models import RestartAgentRequest
from nvidia_resiliency_ext.attribution.restart_agent.pipeline import RestartAgent

agent = RestartAgent(evidence_extractor=LlmEvidenceExtractor(LlmConfig.from_env()))
run = agent.run(RestartAgentRequest(log_path="/path/to/job.log", job_id="12345"))
result = run.result.to_payload()

---

Output

The CLI prints one JSON object to stdout. Important fields include:

Field	Meaning
decision	RESTART or STOP
decision_basis	Policy reason for the decision
retry_policy	Retry budget and selected rule details
primary_failure	Primary failure evidence, when identified
secondary_failures	Related or cascading failure evidence
evidence_coverage	Which evidence sources were checked/found
justification	Human-readable explanation for the final decision

---

Prerequisites

• LLM_API_KEY set (env var, LLM_API_KEY_FILE, or ~/.llm_api_key) for L1 enrichment
• Use --disable-l1 when you want deterministic-only analysis without model access