Skip to content

Commit ffdd55d

Browse files
committed
List and table styles to on-call-rotation-small-teams-sustainable-coverage article
1 parent 092e88e commit ffdd55d

2 files changed

Lines changed: 193 additions & 92 deletions

File tree

  • src/content/articles/on-call-rotation-small-teams-sustainable-coverage

‎src/content/articles/on-call-rotation-small-teams-sustainable-coverage/index.mdx‎

Lines changed: 59 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -15,25 +15,31 @@ A three-person team I worked with had 47 pages in a single week. Most were trans
1515

1616
They didn't fix this by hiring more people. They fixed it by deleting alerts.
1717

18-
Within two months, pages dropped to 2-3 per week. The team started sleeping again. The counterintuitive lesson: on-call sustainability isn't about headcount or rotation schedules. It's about the quality of the alerts that wake people up.
18+
Within two months, pages dropped to 2-3 per week. The team started sleeping again. The counterintuitive lesson: <Highlighter>on-call sustainability isn't about headcount or rotation schedules. It's about the quality of the alerts that wake people up.</Highlighter>
1919

2020
One metric predicts whether your on-call will burn out your team: signal-to-noise ratio. Here's how to measure it, recognize when it's killing you, and fix it before someone quits.
2121

2222
## What Signal-to-Noise Ratio Actually Measures
2323

24-
Signal-to-noise ratio is the percentage of pages that required human action. Calculate it monthly: pages that required someone to __do something__ divided by total pages.
24+
Signal-to-noise ratio is the percentage of pages that required human action. Calculate it monthly: pages that required someone to _do something_ divided by total pages.
2525

2626
<List
27-
variant="check-icons-list"
27+
variant="colored-marker-list"
28+
classes={{
29+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
30+
}}
2831
items={[
2932
{
30-
text: 'Greater than 80%: healthy',
33+
lead: 'Greater than 80%:&nbsp;&nbsp;<span class="font-normal">Healthy</span>',
34+
color: "bg-success",
3135
},
3236
{
33-
text: '50-80%: concerning',
37+
lead: '50-80%:&nbsp;&nbsp;<span class="font-normal">Concerning</span>',
38+
color: "bg-warning-offset",
3439
},
3540
{
36-
text: 'Below 50%: your alerting is broken',
41+
lead: 'Below 50%:&nbsp;&nbsp;<span class="font-normal">Your alerting is broken</span>',
42+
color: "bg-danger",
3743
},
3844
]}
3945
/>
@@ -43,22 +49,37 @@ The math is simple but brutal. A three-person team can sustainably handle maybe
4349
Here's an example from a real team audit:
4450

4551
<List
46-
variant="check-icons-list"
52+
variant="plain-icon-list"
53+
classes={{
54+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
55+
li: "[&>span]:!font-semibold [&>span]:ml-2 mb-4",
56+
}}
57+
size={7}
4758
items={[
4859
{
4960
text: '60 total pages this month',
61+
icon: "alarm-clock",
62+
color: "page-inverse",
5063
},
5164
{
5265
text: '35 required action',
66+
icon: "gear",
67+
color: "page-inverse",
5368
},
5469
{
5570
text: '20 auto-resolved before anyone could respond',
71+
icon: "clock",
72+
color: "page-inverse",
5673
},
5774
{
5875
text: '5 were false positives',
76+
icon: "error",
77+
color: "page-inverse",
5978
},
6079
{
6180
text: 'SNR: 58%—concerning, needs work',
81+
icon: "graph",
82+
color: "page-inverse",
6283
},
6384
]}
6485
/>
@@ -68,7 +89,8 @@ That 20 auto-resolved pages is the killer. If an alert fires and resolves before
6889
Every alert outcome falls into one of four categories, each with a target:
6990

7091
<Table
71-
variant="vertical-column-delineation-table"
92+
variant="criteria-highlight-table"
93+
fullWidth={false}
7294
content={{
7395
thead: {
7496
th: ['Alert Outcome', 'Target', 'Action If Exceeded'],
@@ -100,7 +122,8 @@ Every alert outcome falls into one of four categories, each with a target:
100122
The weekly review ritual makes this actionable. During every on-call handoff, spend 30 minutes reviewing every page from the past week:
101123

102124
<Table
103-
variant="vertical-column-delineation-table"
125+
variant="grid-and-accent-header-table"
126+
fullWidth={false}
104127
content={{
105128
thead: {
106129
th: ['Question', 'What "No" Reveals'],
@@ -135,22 +158,27 @@ The weekly review ritual makes this actionable. During every on-call handoff, sp
135158

136159
The answers drive improvements. Alert too sensitive? Tune the threshold. Could be automated? Build auto-remediation. Provided no value? Delete it. The goal is for every alert to earn its place in the rotation.
137160

138-
<Newsletter />
139-
140161
## Recognizing Burnout Before It's Too Late
141162

142163
The insidious thing about on-call burnout is that it accumulates slowly. By the time it's obvious, someone is already job hunting.
143164

144165
Individual burnout shows up in behavior first. Someone starts acknowledging alerts but not actually investigating them. Response times gradually increase. They snooze alerts instead of addressing them. There's resentment in handoff meetings — subtle comments about the unfairness of the rotation or the quality of alerts.
145166

167+
<Newsletter />
168+
146169
Emotional signs follow: dread when an on-call shift approaches, anxiety about phone notifications even when off rotation, the feeling that you can never truly disconnect. Eventually physical symptoms emerge — sleep disruption that persists even off rotation, exhaustion that doesn't recover between shifts.
147170

148171
At the team level, watch for alerts being suppressed rather than fixed, runbooks not being updated, post-incident reviews getting skipped, or transfer requests. These are all signs that people have given up on improving the system and are just trying to survive it.
149172

150173
The numbers tell a story too:
151174

152175
<Table
153-
variant="vertical-column-delineation-table"
176+
variant="numbered-review-table"
177+
fullWidth={false}
178+
classes={{
179+
tbody:
180+
'[&_td:nth-child(3)]:text-success [&_td:nth-child(4)]:text-warning [&_td:nth-child(5)]:text-danger',
181+
}}
154182
content={{
155183
thead: {
156184
th: ['Metric', 'Healthy', 'Concerning', 'Unsustainable'],
@@ -192,19 +220,22 @@ Every alert must earn its place in the rotation. If it doesn't require immediate
192220
For each problematic alert, you have three options:
193221

194222
<List
195-
variant="check-icons-list"
223+
variant="chevron-list"
224+
classes={{
225+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
226+
}}
196227
items={[
197228
{
198-
lead: 'Delete it',
199-
text: 'if the alert never leads to action. This sounds scary, but an alert that nobody acts on is worse than no alert — it trains your team to ignore pages. If you\'re nervous, demote it to a Slack notification for a month and see if anyone notices.',
229+
lead: 'Delete it:',
230+
text: 'If the alert never leads to action. This sounds scary, but an alert that nobody acts on is worse than no alert — it trains your team to ignore pages. If you\'re nervous, demote it to a Slack notification for a month and see if anyone notices.',
200231
},
201232
{
202-
lead: 'Tune it',
203-
text: 'if the alert fires too often or at the wrong times. Add a duration requirement so transient spikes don\'t page (require the condition to persist for 5 minutes instead of firing immediately). Add hysteresis so alerts don\'t flap (fire at 90%, resolve at 80%). Adjust thresholds based on actual behavior rather than theoretical limits.',
233+
lead: 'Tune it:',
234+
text: 'If the alert fires too often or at the wrong times. Add a duration requirement so transient spikes don\'t page (require the condition to persist for 5 minutes instead of firing immediately). Add hysteresis so alerts don\'t flap (fire at 90%, resolve at 80%). Adjust thresholds based on actual behavior rather than theoretical limits.',
204235
},
205236
{
206-
lead: 'Automate it',
207-
text: 'if the fix is always the same. If every disk space alert ends with "clear /tmp and rotate logs," that\'s not a human problem — that\'s a script. Auto-remediate the common case, only escalate to a human if automation fails.',
237+
lead: 'Automate it:',
238+
text: 'If the fix is always the same. If every disk space alert ends with "clear /tmp and rotate logs," that\'s not a human problem — that\'s a script. Auto-remediate the common case, only escalate to a human if automation fails.',
208239
},
209240
]}
210241
/>
@@ -213,12 +244,12 @@ The biggest win for most teams is after-hours filtering. Not everything needs to
213244

214245
This isn't ignoring problems — it's acknowledging that "one replica down out of three" at 2 AM doesn't justify waking someone up when the service is still functional. The on-call person can check in the morning.
215246

216-
If your page budget is consistently exceeded despite these efforts, there's a nuclear option: stop feature work until alerting is fixed. This sounds dramatic, but reliability debt is real debt. A team that can't sleep can't ship features either. Sometimes you need to stop digging before you can climb out.
217-
218247
<Callout type="success">
219248
The weekly review is the highest-leverage practice for on-call sustainability. Thirty minutes per week of deliberate improvement compounds into dramatically better on-call within a quarter.
220249
</Callout>
221250

251+
If your page budget is consistently exceeded despite these efforts, there's a nuclear option: stop feature work until alerting is fixed. This sounds dramatic, but reliability debt is real debt. A team that can't sleep can't ship features either. Sometimes you need to stop digging before you can climb out.
252+
222253
## Start Here
223254

224255
The team I mentioned at the start didn't need a new rotation schedule or a new incident management platform. They needed fewer, better alerts. The constraint of being a small team forced discipline that larger teams often lack — when you can't spread the pain across twenty people, you have to actually fix the problems.
@@ -241,15 +272,19 @@ Start this week:
241272

242273
<List
243274
variant="numbered-with-background-list"
275+
classes={{
276+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
277+
li: "[&>div]:!pt-2",
278+
}}
244279
items={[
245280
{
246-
text: 'Calculate your signal-to-noise ratio. If it\'s below 80%, you have work to do.',
281+
lead: 'Calculate your signal-to-noise ratio. If it\'s below 80%, you have work to do.',
247282
},
248283
{
249-
text: 'Schedule your first weekly review during the next on-call handoff.',
284+
lead: 'Schedule your first weekly review during the next on-call handoff.',
250285
},
251286
{
252-
text: 'Pick one high-volume alert and either tune it, automate it, or delete it.',
287+
lead: 'Pick one high-volume alert and either tune it, automate it, or delete it.',
253288
},
254289
]}
255290
/>

0 commit comments

Comments
 (0)