Skip to content

Better handling of transient failures - #44

Merged
jasoncrawford merged 7 commits into
mainfrom
feature/transient-failures
Feb 22, 2026
Merged

jasoncrawford merged 7 commits into
mainfrom
feature/transient-failures

Conversation

@jasoncrawford

Copy link
Copy Markdown
Collaborator

Summary

Implements the spec from #39 — distinguishes between hard and soft check failures for smarter incident handling.

  • Hard failures (HTTP 500, DNS errors, SSL errors, connection refused) create incidents immediately on first occurrence
  • Soft failures (HTTP 502, 503, 504, request timeout) only create incidents after 3 occurrences within a 1-hour window
  • Site status now has three states, derived from the last hour of checks:
    • Up (green) — all checks passing
    • Transient Failures (amber) — only soft failures present
    • Failures (red) — any hard failure present

Files changed

File Change
src/lib/checker.ts Added isSoftFailure(), computeSiteStatus(), and SiteStatus type
src/app/api/checks/run/route.ts Updated incident creation logic with hard/soft distinction
src/app/page.tsx Status page fetches last-hour checks, shows 3-state dot color
src/app/sites/[id]/page.tsx Site detail shows 3-state status (Up / Failures / Transient Failures)
src/__tests__/checker.test.ts Tests for isSoftFailure and computeSiteStatus
src/__tests__/check-runner.test.ts Tests for hard/soft incident creation logic

Test plan

  • All 106 tests pass
  • Lint clean
  • Verify on preview deploy: single 503 does NOT create an incident
  • Verify on preview deploy: three 503s within an hour creates an incident
  • Verify amber dot appears for sites with only soft failures
  • Verify red dot appears for sites with hard failures

Closes #39

🤖 Generated with Claude Code

Hard failures (500, DNS, SSL, etc.) create incidents immediately.
Soft failures (502, 503, 504, timeout) only create incidents after
3 occurrences within an hour. Site status now shows three states:
Up (green), Transient Failures (amber), Failures (red).

Closes #39

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vercel

vercel Bot commented Feb 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
site-status Ready Ready Preview, Comment Feb 22, 2026 10:39pm

Request Review

@supabase

supabase Bot commented Feb 22, 2026 •

Copy link
Copy Markdown

Updates to Preview Branch (feature/transient-failures) ↗︎

Deployments Status Updated
Database ✅ Sun, 22 Feb 2026 22:39:32 UTC
Services ✅ Sun, 22 Feb 2026 22:39:32 UTC
APIs ✅ Sun, 22 Feb 2026 22:39:32 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks Status Updated
Configurations ✅ Sun, 22 Feb 2026 22:39:33 UTC
Migrations ✅ Sun, 22 Feb 2026 22:39:34 UTC
Seeding ✅ Sun, 22 Feb 2026 22:39:34 UTC
Edge Functions ✅ Sun, 22 Feb 2026 22:39:34 UTC

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

Adds 4 sites with check histories demonstrating each status:
- Healthy Site: all passing → Up (green)
- Flaky Site: one 503 among passing checks → Transient Failures (amber)
- Broken Site: one HTTP 500 → Failures (red)
- Down Site: 6 consecutive 503s with open incident

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The seed runs before migrations, so the position column doesn't
exist yet. The migration backfills positions alphabetically.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jasoncrawford

Copy link
Copy Markdown
Collaborator Author

I changed my mind. Forget the "site status" and drop that concept completely. Instead:

  • Individual checks should be shown red/amber/green in accordance with whether they are hard failure / soft failure / success, as per above definition.
  • On the main page, instead of "Sites", the header should say "Latest checks", and it should show red/amber/green as per the latest check.
  • On the site detail page, it should not show an overall site status at all. This is not needed, because the full check log is right there.

Instead of computing an aggregate site status from the last hour of
checks, show red/amber/green based on the individual latest check
(hard failure / soft failure / success). Removes computeSiteStatus and
SiteStatus type. Renames main page header to "Latest checks". Removes
the Up/Down/Transient status indicator from site detail page.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jasoncrawford

Copy link
Copy Markdown
Collaborator Author

The check log on the incident detail page is using a different display from the check log on the site detail page, and isn't following our new red/amber/green conventions. Please update the incident detail to use the same UI component as the site detail so they are always in sync.

Unifies the check log display between site detail and incident detail
pages. Both now use the same table with red/amber/green dot colors.
The incident page passes triggeringCheckId to highlight the check that
opened the incident.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keeps SiteList component with drag-reorder from main, adds isSoftFailure
3-color dots. Seed uses position column. Header says "Latest checks".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jasoncrawford
jasoncrawford merged commit 4f25e0b into main Feb 22, 2026
4 checks passed

This branch was successfully deployed

1 active deployment
Preview — dd67f9bd Deployed Feb 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Better handling of transient failures

1 participant