Escalate findings that miss their response target

[richardmhope]

Part 4 of #146, and the last.

Closes #146

An announcement says "a new secret appeared". This says the opposite thing: nobody
fixed the one we told you about. It fires when an open, unsuppressed finding passes
the response target for its severity — which the maintenance loop has to notice,
because a deadline passing is not something that happens to a finding, so there is
no transaction to hang an outbox row off and nothing else would ever see it.

Reuses the outbox rather than building one

notification_delivery becomes a two-kind table, so retry, exponential backoff, the
attempt ceiling, and the "never lost silently" guarantee all come from #60 unchanged.

kind	caused by	deduplicated on
finding_opened	a scan	(channel_id, finding_id, scan_id) — the existing constraint
finding_overdue	the clock	(channel_id, finding_id, due_at) — a new partial index

The three calls worth arguing about

Who hears about it is narrower than an announcement.

Finding	Escalates to
Owned, team has a channel	that team's channel, and nowhere else
Owned, team has no channel	nobody — silent by choice; the overdue queue is the record
Unowned, or owned by a disbanded team	every enabled channel whose filter selects it

Telling six channels about work that has an owner is how alerting becomes noise. The
unowned fallback exists because "late, and nobody has picked it up" is the state
most worth saying out loud, and it is exactly the state with no team to tell. A
disbanded team is that state with extra steps — nobody is reading that channel.

Once per deadline, structurally. The row carries the due_at it is about. The
existing unique constraint cannot do this job: it includes scan_id, which is NULL
on an escalation, and NULLs do not collide in a unique constraint — so without the
partial index the loop would insert a fresh escalation every beat and mail the owning
team once a minute until somebody stopped it. test_a_duplicate_escalation_is_refused_by_the_database
exercises the constraint rather than assuming it.

The deadline is copied onto the row rather than read back off the finding, because a
reopened finding gets a fresh clock: a team that misses the new target should hear
about it, and one that already heard about the old one should not hear twice.

One finding block, two events. The payload keeps the same finding and source
blocks as finding.opened, so a receiver parses one shape and switches on event;
what differs is an escalation block with the deadline, whole hours late, and who was
supposed to be looking at it. No scan block — nothing scanned. Factoring the finding
block into one function is deliberate: an explicit field list is what stops a new
Finding column silently leaving the deployment, and a second copy would be the one
that grew a field nobody reviewed.

Smaller things

• A beat is bounded at 200 findings, so turning escalation on after months of backlog
  does not try to mail the whole history in one round; the rest go on the next beat.
• "Actionable" is the same definition the queue and the badge use — a suppressed
  finding keeps its date, because the suppression can lapse, but mailing somebody
  about it would honour "stop telling me" in the console and ignore it in their inbox.
• Migration 0015 drops escalation rows on downgrade rather than inventing a scan id
  for a message that never had one.

#146 closed out

• Owner and assignee fields with group-based routing rules — Route findings to an accountable group, with a response target #164
• Severity-based target dates and overdue states — Route findings to an accountable group, with a response target #164
• Reassignment history, notifications, filters, and dashboard summaries — Manage ownership policy, and queue findings by it #165, Put ownership on the console #170, and this
• API and audit-event coverage — Manage ownership policy, and queue findings by it #165
• Routing rules can use source, detector, severity, and tags
• Manual overrides are preserved and audited
• Due and overdue calculations are deterministic and timezone-safe
• Unowned actionable findings are visible as a first-class queue
• Permissions prevent unauthorized assignment changes

22 tests in apps/api/tests/test_escalation.py; docs/notifications.md carries the
routing table and the payload. make check green, 1767 passed.

[icebergai-review-bot]

Verdict

CHANGES_REQUESTED

Completed bounded review across 1 immutable scope(s). 2 high-severity correctness blockers found.

Scope health

Convergence: healthy. Review mode: initial.
Recommended action: CONTINUE_INCREMENTAL.

• No escalation signals.

Prior findings

Finding	Status
—	No prior finding state

New findings

Root cause: Pagination is applied before the deduplication predicate.

• BLOCKER · high: Escalation backlog stalls permanently after the first page — apps/api/src/iceberg_api/notifications/dispatch.py
  Status: NEW. Attribution: new_in_scope.
  The query limits overdue findings before excluding findings that already have an escalation. On the second maintenance beat it selects the same oldest 200 rows, skips them in Python as already queued, and never reaches later overdue findings.
  Invariant: Every actionable overdue finding must eventually receive one escalation per eligible channel and deadline.
  Ownership: Maintenance escalation queueing. Behaviour: Overdue notification delivery.
  Evidence: escalate_overdue() applies .order_by(Finding.due_at).limit(limit) and only afterwards checks for an existing NotificationDelivery inside the loop. Reproduction: create 201 overdue findings, run one beat and commit (200 rows queued), then run subsequent beats; each selects and skips the same 200, leaving the 201st unqueued.
  Independent assessment: The fixed .limit(limit) is applied before per-row deduplication. Once the oldest page has deliveries, every later beat reselects and skips that page, so later overdue findings are never reached.

Root cause: Delivery payload construction reads mutable finding state rather than the immutable outbox event deadline.

• BLOCKER · high: Delayed escalation delivers the finding's newer deadline instead of its queued deadline — apps/api/src/iceberg_api/notifications/payload.py
  Status: NEW. Attribution: new_in_scope.
  Although the outbox row stores due_at, delivery passes only the current Finding to finding_overdue(), which reads finding.due_at. A pending escalation therefore changes meaning if the finding is resolved and reopened with a new response target before retry delivery.
  Invariant: An escalation must describe the specific deadline recorded on its outbox row.
  Ownership: Notification outbox delivery. Behaviour: Escalation payload and email accuracy.
  Evidence: NotificationDelivery.due_at is populated when queueing, but finding_overdue() assigns due_at = finding.due_at. Reproduction: queue an old-deadline escalation, make its first delivery retryable-fail, reopen the finding with a future due_at, then retry delivery; the old outbox row is sent using the future deadline.
  Independent assessment: Queueing records the event deadline in NotificationDelivery.due_at, but delivery constructs the payload from mutable Finding.due_at. The change explicitly allows reopened findings to receive a fresh deadline, so a retried older delivery can report the newer deadline.

Fix-induced regressions

• None evidenced.

Uncertainty

• No material uncertainty recorded.

Validation

• Reviewed the supplied immutable diff only.
• Exact-head CI was reported as passing.

Residual risks

• None identified.

[richardmhope]

Both blockers fixed in 3b22df1. Both were real, and both were where you said.

Pagination before deduplication. The exclusion moved into SQL, before the
.limit(), so each beat takes the next page rather than re-selecting and
skipping the same oldest one forever. The per-row dedup check inside the loop is
gone with it — nothing that reaches the loop has an escalation any more, and the
partial unique index is still the structural guard behind both.

One deliberate consequence worth naming: the exclusion is keyed on
(finding, deadline), not (channel, finding, deadline). A channel added after
a finding went overdue therefore does not receive a retrospective escalation for
it. That matches enqueue_for_scan, where a new channel hears about the next
scan rather than every finding in the table, and it is documented in
docs/notifications.md.

Mutable finding state in the payload. finding_overdue() now takes due_at
as an argument and delivery passes delivery.due_at — which was the reason the
column existed in the first place, so this was a wiring gap rather than a design
one.

Both are pinned by tests that fail without the fix (verified by stashing the two
source files and re-running):

• test_a_backlog_larger_than_one_beat_drains_across_beats — five overdue
  findings drain 2/2/1/0 across four bounded beats.
• test_a_retried_escalation_reports_the_deadline_it_was_queued_for — the
  finding is reopened with a later deadline between queueing and delivery; the
  message still reports the one it was queued for.

make check green: 1769 passed.

[icebergai-review-bot]

Verdict

APPROVE

Completed bounded review across 1 immutable scope(s). No new actionable findings. Both prior blockers are resolved by the supplied change.

Scope health

Convergence: healthy. Review mode: delta.
Recommended action: CONTINUE_INCREMENTAL.

• No escalation signals.

Prior findings

Finding	Status
F-3F5529592D91	Resolved
F-B1C81A948AB3	Resolved

New findings

No new findings.

Fix-induced regressions

• None evidenced.

Uncertainty

• No material uncertainty recorded.

Validation

• Prior blocker F-3F5529592D91 resolved: dispatch.py passes due_at=delivery.due_at to finding_overdue(), and the regression test asserts a reopened finding's retried escalation reports the original deadline.
• Prior blocker F-B1C81A948AB3 resolved: The correlated NOT EXISTS predicate precedes .limit(limit); the new multi-beat test verifies a five-item backlog drains as [2, 2, 1, 0].
• Reviewed the supplied immutable diff; exact-head CI is reported as passed.
• Escalation selection now excludes already-queued (finding, deadline) rows before applying the page limit.
• Delivery now passes the immutable outbox deadline into the overdue payload.

Residual risks

• None identified.