You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/articles/on-call-rotation-small-teams-sustainable-coverage/index.mdx
+59-24Lines changed: 59 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,25 +15,31 @@ A three-person team I worked with had 47 pages in a single week. Most were trans
15
15
16
16
They didn't fix this by hiring more people. They fixed it by deleting alerts.
17
17
18
-
Within two months, pages dropped to 2-3 per week. The team started sleeping again. The counterintuitive lesson: on-call sustainability isn't about headcount or rotation schedules. It's about the quality of the alerts that wake people up.
18
+
Within two months, pages dropped to 2-3 per week. The team started sleeping again. The counterintuitive lesson: <Highlighter>on-call sustainability isn't about headcount or rotation schedules. It's about the quality of the alerts that wake people up.</Highlighter>
19
19
20
20
One metric predicts whether your on-call will burn out your team: signal-to-noise ratio. Here's how to measure it, recognize when it's killing you, and fix it before someone quits.
21
21
22
22
## What Signal-to-Noise Ratio Actually Measures
23
23
24
-
Signal-to-noise ratio is the percentage of pages that required human action. Calculate it monthly: pages that required someone to __do something__ divided by total pages.
24
+
Signal-to-noise ratio is the percentage of pages that required human action. Calculate it monthly: pages that required someone to _do something_ divided by total pages.
25
25
26
26
<List
27
-
variant="check-icons-list"
27
+
variant="colored-marker-list"
28
+
classes={{
29
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
30
+
}}
28
31
items={[
29
32
{
30
-
text: 'Greater than 80%: healthy',
33
+
lead: 'Greater than 80%: <span class="font-normal">Healthy</span>',
lead: 'Below 50%: <span class="font-normal">Your alerting is broken</span>',
42
+
color: "bg-danger",
37
43
},
38
44
]}
39
45
/>
@@ -43,22 +49,37 @@ The math is simple but brutal. A three-person team can sustainably handle maybe
43
49
Here's an example from a real team audit:
44
50
45
51
<List
46
-
variant="check-icons-list"
52
+
variant="plain-icon-list"
53
+
classes={{
54
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
55
+
li: "[&>span]:!font-semibold [&>span]:ml-2 mb-4",
56
+
}}
57
+
size={7}
47
58
items={[
48
59
{
49
60
text: '60 total pages this month',
61
+
icon: "alarm-clock",
62
+
color: "page-inverse",
50
63
},
51
64
{
52
65
text: '35 required action',
66
+
icon: "gear",
67
+
color: "page-inverse",
53
68
},
54
69
{
55
70
text: '20 auto-resolved before anyone could respond',
71
+
icon: "clock",
72
+
color: "page-inverse",
56
73
},
57
74
{
58
75
text: '5 were false positives',
76
+
icon: "error",
77
+
color: "page-inverse",
59
78
},
60
79
{
61
80
text: 'SNR: 58%—concerning, needs work',
81
+
icon: "graph",
82
+
color: "page-inverse",
62
83
},
63
84
]}
64
85
/>
@@ -68,7 +89,8 @@ That 20 auto-resolved pages is the killer. If an alert fires and resolves before
68
89
Every alert outcome falls into one of four categories, each with a target:
69
90
70
91
<Table
71
-
variant="vertical-column-delineation-table"
92
+
variant="criteria-highlight-table"
93
+
fullWidth={false}
72
94
content={{
73
95
thead: {
74
96
th: ['Alert Outcome', 'Target', 'Action If Exceeded'],
@@ -100,7 +122,8 @@ Every alert outcome falls into one of four categories, each with a target:
100
122
The weekly review ritual makes this actionable. During every on-call handoff, spend 30 minutes reviewing every page from the past week:
101
123
102
124
<Table
103
-
variant="vertical-column-delineation-table"
125
+
variant="grid-and-accent-header-table"
126
+
fullWidth={false}
104
127
content={{
105
128
thead: {
106
129
th: ['Question', 'What "No" Reveals'],
@@ -135,22 +158,27 @@ The weekly review ritual makes this actionable. During every on-call handoff, sp
135
158
136
159
The answers drive improvements. Alert too sensitive? Tune the threshold. Could be automated? Build auto-remediation. Provided no value? Delete it. The goal is for every alert to earn its place in the rotation.
137
160
138
-
<Newsletter />
139
-
140
161
## Recognizing Burnout Before It's Too Late
141
162
142
163
The insidious thing about on-call burnout is that it accumulates slowly. By the time it's obvious, someone is already job hunting.
143
164
144
165
Individual burnout shows up in behavior first. Someone starts acknowledging alerts but not actually investigating them. Response times gradually increase. They snooze alerts instead of addressing them. There's resentment in handoff meetings — subtle comments about the unfairness of the rotation or the quality of alerts.
145
166
167
+
<Newsletter />
168
+
146
169
Emotional signs follow: dread when an on-call shift approaches, anxiety about phone notifications even when off rotation, the feeling that you can never truly disconnect. Eventually physical symptoms emerge — sleep disruption that persists even off rotation, exhaustion that doesn't recover between shifts.
147
170
148
171
At the team level, watch for alerts being suppressed rather than fixed, runbooks not being updated, post-incident reviews getting skipped, or transfer requests. These are all signs that people have given up on improving the system and are just trying to survive it.
@@ -192,19 +220,22 @@ Every alert must earn its place in the rotation. If it doesn't require immediate
192
220
For each problematic alert, you have three options:
193
221
194
222
<List
195
-
variant="check-icons-list"
223
+
variant="chevron-list"
224
+
classes={{
225
+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
226
+
}}
196
227
items={[
197
228
{
198
-
lead: 'Delete it',
199
-
text: 'if the alert never leads to action. This sounds scary, but an alert that nobody acts on is worse than no alert — it trains your team to ignore pages. If you\'re nervous, demote it to a Slack notification for a month and see if anyone notices.',
229
+
lead: 'Delete it:',
230
+
text: 'If the alert never leads to action. This sounds scary, but an alert that nobody acts on is worse than no alert — it trains your team to ignore pages. If you\'re nervous, demote it to a Slack notification for a month and see if anyone notices.',
200
231
},
201
232
{
202
-
lead: 'Tune it',
203
-
text: 'if the alert fires too often or at the wrong times. Add a duration requirement so transient spikes don\'t page (require the condition to persist for 5 minutes instead of firing immediately). Add hysteresis so alerts don\'t flap (fire at 90%, resolve at 80%). Adjust thresholds based on actual behavior rather than theoretical limits.',
233
+
lead: 'Tune it:',
234
+
text: 'If the alert fires too often or at the wrong times. Add a duration requirement so transient spikes don\'t page (require the condition to persist for 5 minutes instead of firing immediately). Add hysteresis so alerts don\'t flap (fire at 90%, resolve at 80%). Adjust thresholds based on actual behavior rather than theoretical limits.',
204
235
},
205
236
{
206
-
lead: 'Automate it',
207
-
text: 'if the fix is always the same. If every disk space alert ends with "clear /tmp and rotate logs," that\'s not a human problem — that\'s a script. Auto-remediate the common case, only escalate to a human if automation fails.',
237
+
lead: 'Automate it:',
238
+
text: 'If the fix is always the same. If every disk space alert ends with "clear /tmp and rotate logs," that\'s not a human problem — that\'s a script. Auto-remediate the common case, only escalate to a human if automation fails.',
208
239
},
209
240
]}
210
241
/>
@@ -213,12 +244,12 @@ The biggest win for most teams is after-hours filtering. Not everything needs to
213
244
214
245
This isn't ignoring problems — it's acknowledging that "one replica down out of three" at 2 AM doesn't justify waking someone up when the service is still functional. The on-call person can check in the morning.
215
246
216
-
If your page budget is consistently exceeded despite these efforts, there's a nuclear option: stop feature work until alerting is fixed. This sounds dramatic, but reliability debt is real debt. A team that can't sleep can't ship features either. Sometimes you need to stop digging before you can climb out.
217
-
218
247
<Callouttype="success">
219
248
The weekly review is the highest-leverage practice for on-call sustainability. Thirty minutes per week of deliberate improvement compounds into dramatically better on-call within a quarter.
220
249
</Callout>
221
250
251
+
If your page budget is consistently exceeded despite these efforts, there's a nuclear option: stop feature work until alerting is fixed. This sounds dramatic, but reliability debt is real debt. A team that can't sleep can't ship features either. Sometimes you need to stop digging before you can climb out.
252
+
222
253
## Start Here
223
254
224
255
The team I mentioned at the start didn't need a new rotation schedule or a new incident management platform. They needed fewer, better alerts. The constraint of being a small team forced discipline that larger teams often lack — when you can't spread the pain across twenty people, you have to actually fix the problems.
@@ -241,15 +272,19 @@ Start this week:
241
272
242
273
<List
243
274
variant="numbered-with-background-list"
275
+
classes={{
276
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
277
+
li: "[&>div]:!pt-2",
278
+
}}
244
279
items={[
245
280
{
246
-
text: 'Calculate your signal-to-noise ratio. If it\'s below 80%, you have work to do.',
281
+
lead: 'Calculate your signal-to-noise ratio. If it\'s below 80%, you have work to do.',
247
282
},
248
283
{
249
-
text: 'Schedule your first weekly review during the next on-call handoff.',
284
+
lead: 'Schedule your first weekly review during the next on-call handoff.',
250
285
},
251
286
{
252
-
text: 'Pick one high-volume alert and either tune it, automate it, or delete it.',
287
+
lead: 'Pick one high-volume alert and either tune it, automate it, or delete it.',
0 commit comments