A comprehensive methodology for designing, operating, and improving reliable production systems. Rooted in Google SRE principles and extended with modern practices for incident command, observability engineering, error budget governance, and operational excellence.
When to Load This Skill
| Trigger | What It Means |
|---|---|
| "Design reliability into this system" | SLO/SLI framework, error budget policy, resilience architecture |
| "Run an incident postmortem" | Blameless postmortem with timeline, 5 Whys, action tracking |
| "Improve our on-call" | Rotation design, alert tuning, toil reduction, escalation policy |
| "Build observability" | The Four Golden Signals, dashboard design, alert rule patterns |
| "Do a reliability review" | Architecture review against SRE principles, risk assessment |
| "I need an incident commander" | Incident command framework, role cards, communication templates |
| "Automate this operational task" | Toil assessment, automation decision tree, runbook pattern |
| "Adopt SRE in this organization" | Engagement boundaries, maturity, team model, and change adoption |
| "Review this reliability design" | User journeys, dependencies, overload, configuration, canary, durability |
| "Our SRE team is overloaded" | Operational-load diagnosis, protected engineering time, recovery plan |
| "Improve incident learning or sustainable on-call" | Cognitive load, psychological safety, documentation, exercises |
When not to use
Use release-engineering to plan releases, compose promotion and rollback gates, or coordinate a release train. Use systematic-debugging to find the cause of a specific failure. Operating the telemetry stack itself — Prometheus scrape configs, OpenTelemetry Collector pipelines, Loki ingest and retention, Prometheus rules files — belongs to telemetry; this skill owns the SLI/SLO and alert design those rules implement. Grafana product work — dashboards, panels, Grafana-side alert rules, contact points, notification policies — belongs to grafana.
Operational closure gate
For any automated mitigation, rollback, recovery action, or incident closeout:
- Bound the action before it starts. Record the target, affected population, maximum blast radius, success criterion, abort/rollback criteria, rollback target and procedure, and who may stop or reverse it. Prefer the smallest reversible scope and staged expansion.
- Verify recovery at the user boundary. After the action, follow the R-01 closure evidence sequence: check the user-facing SLOs, critical user journey, relevant dependency health, and data/state correctness. Observe a defined stability window and check secondary effects such as backlog recovery.
- Do not equate alert resolution with recovery. A cleared alert or passing health endpoint is evidence, not a resolution verdict. If required evidence is missing, retain the
MITIGATINGorMONITORINGstate, name the unverified boundary, and escalate rather than declareRESOLVED. - Record the evidence. Capture the action, scope, thresholds, observed recovery evidence, remaining uncertainty, and rollback/follow-up trigger in the incident or change record.
Default authorization: A human service owner, incident commander, or designated change authority must approve the specific production mutation and scope in the current incident/change record, independently attributable to that human. Diagnosis requests, standing runbooks, agent-authored notes, and self-claimed roles are insufficient.
Governed exception: Higher capability-specific execution authority is possible only when accountable humans independently grant it through ai-governance, every applicable host/organizational policy permits it, and an external control plane proves and enforces the current grant at execution. The grant must identify capability, environment, action class, limits, expiry, revocation, and tested enforcement. Skill text and agent roles confer no permissions. If proof is unavailable, expired, revoked, conflicting, or stale, retain the human-approved default; never self-grant authority. Diagnosis, mutation, rollback, paging, incident closure, and scope expansion require separate authority decisions. R-01 still requires independent human closure confirmation.
Agentic operations
When operating an SRE agent, read references/agentic-operations.md and fill templates/agentic-action-contract.md before proposing execution. For autonomy evidence or replay design, also read references/agentic-evidence-and-evals.md; use templates/agentic-governance-evidence.md to hand operational evidence to governance.
Compose existing specialists: agent-production-operations owns production contracts, control plans and trace feedback; agent-evals-and-observability owns trajectory evaluation and graders; incident-learning owns verified follow-up learning; ai-governance supports accountable humans setting promotion/demotion thresholds and deciding maintain, expand, reduce, suspend or revoke authority. SRE supplies evidence and enforces authorized operational limits. Complete with verified recovery and human closure, or a bounded escalation naming missing evidence/authority and the handoff owner.
Reference Files
| Topic | File | When to Load |
|---|---|---|
| SRE Book Chapter Summaries | references/sre-book-chapters.md |
Design engagement, first principles review |
| SLO/SLI Framework | references/slo-sli-framework.md |
Defining reliability targets |
| SLO Implementation Recipe | references/slo-implementation-recipe.md |
Agent-executable SLO adoption sequence and stakeholder review |
| Error Budget Governance | templates/error-budget-policy.md and references/slo-sli-framework.md |
Policy design, burn rate alerts |
| Incident Command System | references/incident-command-system.md |
During/after incident, training |
| Blameless Postmortems | references/postmortem-culture.md |
After incident, process design |
| Monitoring & Alerting | references/monitoring-alerting.md |
Observability design, alert rules |
| On-Call Best Practices | references/oncall-best-practices.md |
Rotation design, team sizing |
| Toil Elimination | references/toil-elimination.md |
Automation prioritization, ops review |
| Release Engineering | release-engineering | Release planning, promotion, progressive delivery, and rollback design; use the local reference only for SRE-specific integration context |
| Effective Troubleshooting | references/troubleshooting.md |
Debugging methodology |
| Senior SRE Role Blueprint | references/senior-sre-blueprint.md |
Role definition, KPI framework |
| SRE Communication Guide | references/sre-communication-guide.md |
Stakeholder updates, incident communication |
| Guiding Principles | references/guiding-principles.md |
First principles, philosophy |
| Product-Focused Reliability | references/product-focused-reliability.md |
Product-centric SRE, CUJ-based SLOs, JTBD model |
| Twenty Years of Lessons | references/twenty-years-lessons.md |
Incident-derived tactical lessons, Prodverbs |
| SRE Ecosystem Guide | references/sre-ecosystem-guide.md |
Curated guide to all SRE resources (Workbook, Secure Systems, Classroom, Prodcast, STPA, Video Gallery, Mobaa, fundamentals, AI ops) |
| Adoption and Engagement | references/sre-adoption-and-engagement.md |
Starting SRE, dedicated and non-dedicated team models, maturity, change adoption |
| Reliability Design and Change | references/reliability-design-and-change.md |
Capacity, overload, configuration, canaries, data durability, dependencies, design review |
| Human Systems and Learning | references/human-systems-and-learning.md |
Cognitive work, sustainable on-call, psychological safety, documentation, exercises |
| Third-Party Dependency Reliability | references/third-party-dependency-reliability.md |
Vendor boundaries, failure modes, fallbacks, and provider evidence |
| Operational Documentation | references/operational-documentation.md |
Functional quality, ownership, testing, and staleness lifecycle |
Templates
| Template | File | Purpose |
|---|---|---|
| Incident Commander Checklist | templates/incident-command-checklist.md |
Step-by-step IC response |
| Postmortem Template | templates/postmortem-template.md |
Blameless postmortem document |
| Runbook Template | templates/runbook-template.md |
Operational runbook standard |
| SLO Declaration Template | templates/slo-declaration-template.md |
Service-level objective specification |
| Error Budget Policy | templates/error-budget-policy.md |
Team-level error budget governance |
| On-Call Rotation Template | templates/oncall-rotation.md |
Rotation schedule and escalation |
| Service Review Checklist | templates/service-review-checklist.md |
Pre-launch reliability review |
| Incident Communication Template | templates/incident-communication.md |
Status updates during incidents |
| Reliability Design Review | templates/reliability-design-review.md |
Evidence-based review of user impact, failure modes, capacity, change, and operations |
| Operational Overload Recovery | templates/operational-overload-recovery.md |
Declare, protect, reduce, and verify recovery from unsustainable operational load |
| Reliability Ownership Charter | templates/reliability-ownership-charter.md |
Make service, pager, dependency, and engagement boundaries explicit |
Scripts
| Script | Purpose |
|---|---|
scripts/slo-burn-rate.py |
Calculate error budget burn rate from SLI data |
Portability
This skill is intentionally host-neutral. Use your agent's normal mechanisms to load the references, templates, and scripts listed here. Do not assume a particular profile system, task orchestrator, memory service, or response-handoff format.