Skip to content

fix: keep detector findings and writer byte accounting correct under stalls - #316

Merged
vishr merged 1 commit into
mainfrom
fix/detector-and-accounting
Oct 15, 2026
Merged

vishr merged 1 commit into
mainfrom
fix/detector-and-accounting

Conversation

@vishr

@vishr vishr commented Oct 15, 2026

Copy link
Copy Markdown
Member

Summary

An audit of detector and writer behaviour under saturation reproduced five defects. An adversarial review of the fixes found a sixth. Each one now has a regression test.

  • Observed-time-only logs. Log patterns filtered and took extrema on the original OTLP timestamp, so logs with only an observed time disappeared. Patterns now use canonical native event time.
  • Frozen patterns. A stalled service watermark froze log patterns. Anomaly evaluation is memoised per completed rollup minute; patterns run on their own cadence and the snapshot is rebuilt.
  • Failed pattern queries. A failed pattern query was published as a healthy snapshot and never retried. The error now propagates and the query is retried.
  • Replaced pending annotations. A pending annotation batch replaced before persistence lost a short incident. Displaced findings now merge into the replacement with their evaluated intervals.
  • Dropped anomaly findings. Anomaly findings evaluated before a failed pattern query were dropped once the watermark advanced. They now carry over, bounded, to the next successful publication.
  • Byte budget bypass. The writer released a request's byte reservation when its caller gave up while the request was still queued or committing, which bypassed the decoded-byte budget. The reservation now moves with the request and is released exactly once.

Durable acknowledgement, configuration and formats are unchanged.

Test plan

  • just check
  • just test-race
  • Regression tests for each defect, including recovery while the watermark advances
  • Codex adversarial review; its one finding is fixed here

https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW

…stalls

An audit of detector and writer behaviour under saturation reproduced
these defects, and a review found one more:

- Log patterns filtered and took extrema on the original OTLP timestamp,
  so logs with only an observed time disappeared. They now use canonical
  native event time.
- A stalled service watermark froze log patterns. Anomaly evaluation is
  memoised per completed minute; patterns run on their own cadence and
  the snapshot is rebuilt.
- A failed pattern query was published as a healthy snapshot and never
  retried. The error now propagates and the query is retried.
- A pending annotation batch replaced before persistence lost a short
  incident. Displaced findings now merge into the replacement with their
  evaluated intervals.
- Anomaly findings evaluated before a failed pattern query were dropped
  once the watermark advanced. They now carry over, bounded, to the next
  successful publication.
- The writer released a request's byte reservation when its caller gave
  up while the request was still queued or committing, bypassing the
  decoded-byte budget. The reservation now moves with the request and is
  released exactly once when its payload is no longer held.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
@vishr
vishr force-pushed the fix/detector-and-accounting branch from 8258b87 to 11a88b4 Compare October 15, 2026 20:36
@vishr
vishr merged commit 846e101 into main Oct 15, 2026
22 of 23 checks passed
@vishr
vishr deleted the fix/detector-and-accounting branch October 15, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant