tool-failure-handling

Tool Failure Handling

Tools fail. Design the failure first, the success second.

A schema that's clean and a tool that's "fast on the happy path" still produces a broken agent if the failure semantics are wrong. The most common production failure isn't a bug in the tool — it's the agent retrying, compensating, or proceeding under the wrong assumption about whether the call took effect.

When to use this skill

• The user is adding a tool to an agent that calls external systems (APIs, databases, queues).
• The user reports duplicate side effects (charged twice, emailed twice, ticket created twice).
• The user reports the agent hanging on a slow tool, or looping retries forever.
• The user is choosing between framework-level retry policies and per-tool semantics.

Classify the call first

The right retry / recovery policy depends on what the tool does. Classify before designing:

• Read-safe. No side effects. Retry freely with backoff. Default tier.
• Idempotent write. Same call, same effect. Retry safely with the same idempotency key.
• Non-idempotent write. Each call is a new effect. Retry only with idempotency key support, or only after explicit reconciliation.
• Irreversible. Cannot be undone (money sent, message delivered, data deleted). Do not retry automatically; require human confirmation or a propose/commit split.

Mis-classifying a non-idempotent write as read-safe is the most common cause of duplicate side effects.

Decision flow

1. Did the call succeed? → trust only typed acknowledgment (HTTP 200 and a typed status field, or equivalent). HTTP 200 + error body is silent failure — see [[tool-use-schema-design]].
2. Did the call time out before a response? → unknown state. Use the call's idempotency key to reconcile; do not blindly retry.
3. Is the failure transient (429, 503, timeout) or permanent (400, 401, 422)? → transient → retry with backoff. Permanent → surface to the agent; do not retry.
4. Is the call idempotent? → yes → retry safely. No → reconcile or human-in-the-loop.
5. Has the retry budget been exhausted? → escalate, log, return a typed error to the agent. Don't silently give up.

The five rules

1. Idempotency key on every non-read write. Provided by the agent or generated by the tool wrapper. Server-side dedupe by key. Re-runs are safe by construction.
2. Typed errors, not exceptions. Tool returns {status: "rate_limited", retry_after_ms: 1200} or {status: "invalid_argument", field: "amount"} — never a stringified traceback.
3. Timeout every call. Wall-clock cap per tool. No "default to infinite".
4. Bounded retry budget. Per-call limit (e.g. 3) and per-task limit (e.g. 10). Both enforced.
5. Compensation, not silent rollback. If a multi-step tool sequence partially succeeded, run an explicit compensation step (refund the charge, delete the row); don't pretend nothing happened.

Patterns that work

• Idempotency key + server-side dedupe. The cleanest "exactly-once" you'll get.
• Propose / commit. propose_X() returns a token; commit_X(token) is idempotent on the token. Two-phase, safe to retry the commit.
• Saga / compensation. For multi-step writes across systems, define a compensation per step. On failure, run compensations in reverse.
• Circuit breaker. Tool fails N times in a window → break for T seconds; agent gets a typed tool_unavailable and adapts.
• Reconciliation read. After timeout on a write, read the system's state by the idempotency key to learn outcome before retrying.

Anti-patterns to flag immediately

• Blind retry on timeout. Most timeout-then-success patterns produce duplicates.
• Retry on 5xx only, ignore 429. Rate limits are transient; retry with retry-after.
• Retry on 4xx. 400 / 422 / 401 don't fix themselves. Retrying wastes budget and tokens.
• Exception → string → agent. Stringified errors are unreadable to the model; it can't recover. Type the error.
• No timeout. Agents hang. Whole runs hang. Wall-clock cap is mandatory.
• No retry budget. Infinite-retry loops happen. Cap per call and per task.
• Silent fallback. "If the tool fails, return empty" hides errors; downstream proceeds wrong.
• Compensation as try/except: pass. Compensation is an explicit, typed step. It either runs or surfaces.

Questions to ask the user

1. What's the classification of each tool — read-safe, idempotent write, non-idempotent write, irreversible?
2. Does every non-read write have an idempotency key?
3. What's the timeout per call, per task?
4. What's the retry budget per call, per task?
5. How does the agent distinguish transient from permanent errors?
6. What's the compensation strategy for multi-step sequences?
7. Are failures typed, traced, and fed back into the eval?

The hard line

Never auto-retry a non-idempotent write. Reconcile, not retry. Duplicates are debugged at customer-support cost; the prevention is one server-side dedupe.

Why this exists

The cleanest tool schema produces broken agents in production because the failure semantics weren't designed. Idempotency keys, typed errors, bounded retries, compensation — the patterns are well-known in distributed systems and routinely skipped in agent codebases. The cost is paid in duplicate side effects, hung runs, and untraceable production incidents.

References

• references/classification.md — read-safe / idempotent write / non-idempotent write / irreversible — examples and policies.
• references/idempotency-and-reconcile.md — patterns for idempotency keys, server-side dedupe, post-timeout reconciliation.
• references/saga-compensation.md — multi-step writes across systems, compensation design, rollback shapes.

Related skills

• Failure handling is the runtime half of [[tool-use-schema-design]] — read both.
• Bounded retries are part of [[agent-cost-modeling]] and [[latency-budgeting]].
• Typed errors are first-class trace events — [[agent-observability]].
• Irreversible actions go through [[human-in-the-loop]] before retry policy ever applies.
• Tool failures should be tested in [[agent-evaluation-harness]] adversarial sets.
• The same retry surface is an injection attack vector — [[prompt-injection-defense]].