SuperAPI AI Directory

ai-hallucination-slo-doc

General & other

View repository

Updated 10-16 03:40

About this skill

AI Hallucination SLO Doc Skill

Overview

The AI complement to the SaaS SLO doc. Treats hallucination, citation, abstention, and safety violations as first-class SLIs with their own error budgets and rollback rules.

Core Instructions

Step 1: SLI inventory per AI feature

Required SLIs:

  • Factuality SLI — % of responses on a production-sample where judge-LLM marks all claims as supported.
  • Citation accuracy SLI (RAG only) — % of citations matching source spans within tolerance.
  • Abstention precision SLI — % of abstain responses that correctly should have abstained.
  • Abstention recall SLI — % of should-abstain inputs that did abstain.
  • Safety violation SLI — count of outputs hitting content-policy or PII filters per million calls.
  • Latency SLI (already in parent SLO doc, restated for AI clarity).
  • Cost-per-call SLI (cross-link to cost runbook).

Step 2: Measurement procedure

For each SLI state the measurement source and sample method:

  • Factuality and citation: production-sample replayed through the judge-LLM nightly. Sample rate per feature.
  • Abstention: classify production responses by abstain-payload; spot-check by humans monthly to verify abstain correctness.
  • Safety violation: counted at content-filter; alarmed on each event.

Step 3: Set per-feature SLO targets

Per feature × tier:

Tier Factuality Citation Abstention precision Safety violations
Free >= 0.85 n/a or >= 0.85 >= 0.70 0
Pro >= 0.92 >= 0.90 >= 0.80 0
Enterprise >= 0.95 >= 0.95 >= 0.85 0

Step 4: Error budgets

Standard formula: error_budget = (1 - SLO) × calls_in_window. Safety violations: zero-budget; any breach is SEV1.

Step 5: Multi-burn-rate alerts

Alert Burn rate Window Threshold
Fast burn 14× 1 h 2% of monthly budget
Medium burn 6× 6 h 5%
Slow burn 1× 3 d 10%
Safety n/a 0 any

Step 6: Freeze and rollback rules

  • Error budget exhausted: freeze prompt and model changes; eval bumps require executive approval.
  • Citation accuracy drop > 5 pp in 24 h: auto-rollback to last green prompt tag.
  • Safety violation: pause feature; SEV1; postmortem; provider escalation if upstream.

Step 7: Customer-facing AI-quality commitments

Mirror the parent SLO doc pattern. Per tier, define what is contractually committed (likely abstention behaviour and uptime; not numerical factuality, since per-output verifiability remains imperfect). Define how customers report a perceived hallucination (flag button -> ticket -> review).

Step 8: Write the doc

AI_Hallucination_SLO_Doc.md sections: 1) AI SLI Inventory, 2) Measurement Procedure, 3) Per-Feature SLO Targets, 4) Error Budgets, 5) Burn-Rate Alerts, 6) Freeze & Rollback Rules, 7) Customer-Facing AI Commitments, 8) Review Cadence.

Standards

  • Google SRE applied to AI features
  • ISO/IEC 25010 (functional correctness)
  • ISO/IEC 42001 Clause 9 (performance evaluation)