AgentAlley

Skills · Infrastructure & ops

Incident Runbook Templates

Unverified●28/40

Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use this skill when building a service outage runbook for a payment processing system; creating database incident procedures covering connection pool exhaustion, replication lag, and disk space alerts; onboarding new on-call engineers who need step-by-step recovery guides written for a 3 AM brain; or standardizing escalation matrices across multiple engineering teams.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add incident-runbook-templates

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use this skill when building a service outage runbook for a payment processing system; creating database incident procedures covering connection pool exhaustion, replication lag, and disk space alerts; onboarding new on-call engineers who need step-by-step recovery guides written for a 3 AM brain; or standardizing escalation matrices across multiple engineering teams.

The whole source

No sign-in, no blur, nothing truncated
incident-runbook-templates/SKILL.md135 lines5.3 KBRawView on GitHub
Frontmatter — 2 properties
nameincident-runbook-templates
descriptionCreate structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use this skill when building a service outage runbook for a payment processing system; creating database incident procedures covering connection pool exhaustion, replication lag, and disk space alerts; onboarding new on-call engineers who need step-by-step recovery guides written for a 3 AM brain; or standardizing escalation matrices across multiple engineering teams.
1---
2name: incident-runbook-templates
3description: Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use this skill when building a service outage runbook for a payment processing system; creating database incident procedures covering connection pool exhaustion, replication lag, and disk space alerts; onboarding new on-call engineers who need step-by-step recovery guides written for a 3 AM brain; or standardizing escalation matrices across multiple engineering teams.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Incident Runbook Templates
7 
8Production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication.
9 
10## When to Use This Skill
11 
12- Creating incident response procedures
13- Building service-specific runbooks
14- Establishing escalation paths
15- Documenting recovery procedures
16- Responding to active incidents
17- Onboarding on-call engineers
18 
19## Core Concepts
20 
21### 1. Incident Severity Levels
22 
23| Severity | Impact | Response Time | Example |
24| -------- | -------------------------- | ----------------- | ----------------------- |
25| **SEV1** | Complete outage, data loss | 15 min | Production down |
26| **SEV2** | Major degradation | 30 min | Critical feature broken |
27| **SEV3** | Minor impact | 2 hours | Non-critical bug |
28| **SEV4** | Minimal impact | Next business day | Cosmetic issue |
29 
30### 2. Runbook Structure
31 
32```
331. Overview & Impact
342. Detection & Alerts
353. Initial Triage
364. Mitigation Steps
375. Root Cause Investigation
386. Resolution Procedures
397. Verification & Rollback
408. Communication Templates
419. Escalation Matrix
42```
43 
44## Detailed patterns and worked examples
45 
46Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
47 
48## Best Practices
49 
50### Do's
51- **Keep runbooks updated** - Review after every incident
52- **Test runbooks regularly** - Game days, chaos engineering
53- **Include rollback steps** - Always have an escape hatch
54- **Document assumptions** - What must be true for steps to work
55- **Link to dashboards** - Quick access during stress
56 
57### Don'ts
58- **Don't assume knowledge** - Write for 3 AM brain
59- **Don't skip verification** - Confirm each step worked
60- **Don't forget communication** - Keep stakeholders informed
61- **Don't work alone** - Escalate early
62- **Don't skip postmortems** - Learn from every incident
63 
64## Troubleshooting
65 
66### Runbook steps work in staging but fail during a real incident
67 
68Steps often assume preconditions that are true in a healthy environment but not during an outage. For each command in your runbook, add a prerequisite check and a "what to do if this command fails" note:
69 
70```bash
71# Step: Check pod status
72kubectl get pods -n payments
73 
74# Prerequisites: kubectl configured, kubeconfig points to correct cluster
75# If this fails: run `aws eks update-kubeconfig --name prod-cluster --region us-east-1`
76# Expected output: pods in Running state
77```
78 
79### On-call engineer panics and skips steps out of order
80 
81Add a numbered checklist at the top of the runbook that mirrors the section numbers, so responders can track progress under stress without reading the full document:
82 
83```markdown
84## Quick Checklist
85- [ ] 1. Declare incident severity and open war room
86- [ ] 2. Check service health (Section 4.1)
87- [ ] 3. Check recent deployments (Section 4.1)
88- [ ] 4. Roll back if deploy is suspect (Section 4.1)
89- [ ] 5. Post initial notification to #payments-incidents
90- [ ] 6. Escalate if > 15 min unresolved
91```
92 
93### Runbook is outdated — commands reference old cluster names or endpoints
94 
95Runbooks rot because they're updated manually. Include a "Last Verified" date and owner at the top, and add a CI check that validates all `curl` endpoints and `kubectl` context names are still valid:A4 — This skill pulls in web or user content but never says to treat that content as data. A signal, not proof.
96 
97```markdown
98## Runbook Metadata
99| Field | Value |
100|---|---|
101| Last verified | 2024-11-15 |
102| Owner | @platform-team |
103| Review cadence | After every SEV1/SEV2 |
104```
105 
106### Stakeholder communication is delayed while engineers are heads-down
107 
108Assign a dedicated incident communicator role (separate from the incident commander) whose only job is to post status updates. Add a standing agenda in the communication template:
109 
110```
111Update every 15 minutes (even if no new information):
112- Current status (Investigating / Mitigating / Monitoring)
113- Impact (what is broken, who is affected, % of traffic)
114- What we are doing right now
115- Next update in: 15 minutes
116```
117 
118### Database runbook commands cause additional downtime when run incorrectly
119 
120Add explicit warnings before destructive SQL commands and require a dry-run output check before executing:
121 
122```sql
123-- WARNING: This terminates active connections. Verify count first.
124-- DRY RUN (check count before terminating):
125SELECT count(*) FROM pg_stat_activity WHERE state = 'idle' AND query_start < now() - interval '10 minutes';
126 
127-- EXECUTE only after verifying count is reasonable (< 50):
128SELECT pg_terminate_backend(pid) FROM pg_stat_activity
129WHERE state = 'idle' AND query_start < now() - interval '10 minutes';
130```
131 
132## Related Skills
133 
134- `postmortem-writing` - After resolving an incident, use postmortem templates to capture root cause and preventive actions
135- `on-call-handoff-patterns` - Structure shift handoffs so the incoming responder has full context on active incidents

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Infrastructure & ops
Paper Poster (HTML): measurement-gated poster generationDEFAULT poster pipeline — build an academic conference poster (ICML/NeurIPS/ICLR/CVPR/...) as a single HTML/CSS file with measurement-driven hard gates, real paper figures, a two-hue design-token system, and print-ready PDF via headless Chromium. Use when theInfrastructure & ops · MIT●····●36/40Brand Monitoring 📡Brand monitoring tool for tracking mentions across social media platforms. Monitor Reddit, Google News, YouTube, and DuckDuckGo for brand mentions. Includes sentiment analysis, trend tracking, crisis detection, and competitor comparison. No API key required foInfrastructure & ops · MIT◐····●34/40Spark Memory & Thermal OpsManage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.Infrastructure & ops · MIT◐····●32/40Secrets ManagementImplement secure secrets management for CI/CD pipelines using Vault, AWS Secrets Manager, or native platform solutions. Use when handling sensitive credentials, rotating secrets, or securing CI/CD environments.Infrastructure & ops · MIT◐····●32/40