Skip to content

fix(sentry): report a removed transaction once, after two polls without it - #947

Open
samcm wants to merge 1 commit into
masterfrom
fix/mempool-removed-dedup
Open

samcm wants to merge 1 commit into
masterfrom
fix/mempool-removed-dedup

Conversation

@samcm

@samcm samcm commented Nov 11, 2026

Copy link
Copy Markdown
Member

Follow-up to #946. Reth sentries write about 80% of mainnet mempool_transaction_removed rows, almost all queued-pool dropped rows at 3–4 rows per hash, because queued transactions leave reth's txpool_content and come back. Mainnet over 3 hours of v1.32.0: about 527k rows/hour, of which 515k are drops and 441k are from reth.

Change

  • A transaction missing from one txpool_content snapshot is held. It is reported only if the next snapshot is missing it too and it was not included in between. event_date_time stays the first snapshot without it.
  • A reported hash is remembered for 2 hours, so a transaction that keeps leaving and returning is reported once per sentry. In production, reth re-admits a dropped queued transaction a median 8 minutes and a p99 46 minutes after the drop.
  • The wiped-pool and over-half-vanished guards also clear held transactions, so a restart's losses aren't reported on the next poll.

Verification

  • Unit tests:
    • Reworked for the two-poll rule.
    • New flapping test: a transaction missing from one snapshot then back is not reported; one missing then included is not reported; one reported, re-admitted and gone again within 2h is not reported again; after the TTL it can be.
    • Removing either rule (the reported-hash memory, or the "came back" check) fails it.
  • Against real geth --dev and reth --dev pools with --txpool.lifetime 20s --txpool.nolocals (temporary test, not committed), with a fresh filler transaction each poll so expiry never empties the pool:
    • A queued nonce-gap transaction was evicted and reported once as dropped. It was then re-sent, evicted again, and not reported again.
    • A fee-bump replacement was reported as replaced with the right replacement hash.
    • The included transaction was never reported.
    • No hash was reported twice, on either client.

…ut it

Reth sentries wrote about 80% of mainnet's mempool_transaction_removed
rows, almost all queued-pool drops at 3-4 rows per hash, because queued
transactions leave reth's txpool_content and come back.

A transaction missing from one snapshot is now held, and reported only
if the next snapshot is missing it too and it was not included in the
meantime. A reported hash is remembered for two hours (production reth
re-admits a dropped queued transaction a median 8 and a p99 46 minutes
later), so a transaction that keeps leaving and returning is reported
once. event_date_time stays the first snapshot without it.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c90a-57e2-779d-ad72-fa519d20e045
Co-authored-by: Sam Calder-Mason <samcaldermason@gmail.com>

@redpandabot redpandabot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This follow-up delays removal reporting until a transaction is missing from two consecutive txpool snapshots and remembers reported hashes for two hours to suppress flapping re-reports. The held/confirmed bookkeeping, inclusion check, replacement lookup, and guard resets are consistent, and the reworked tests cover the two-poll rule, flapping, inclusion, and TTL expiry. I found no blocking correctness issues.


Reviewed 2 changed file(s) @ 6c4bfba5 — no blocking issues found.
"Don't drink the bear's skin before you've shot it." — Hungarian

@redpandabot

redpandabot Bot commented Nov 11, 2026

Copy link
Copy Markdown

CI failed @ 6c4bfba5

One failed job: govulncheck / govulncheck. It fails on three vulnerability advisories (GO-2026-6888 brotli v1.2.0, GO-2026-6865 klauspost/compress v1.19.1, GO-2026-6859 net/http go1.26.10) that are not in the workflow's ALLOWED regex. The PR only changes pkg/sentry/execution/mempool_removals.go and its test, and go.mod/go.sum are identical to origin/master, so this is pre-existing; the same three advisories also failed govulncheck on unrelated branches (runs 42974332955 and 42970596458).

  • govulncheck / govulncheck — pre-existing — govulncheck ./... reported three advisories not matched by the ALLOWED regex at .github/workflows/govulncheck.yaml:65: GO-2026-6888 in github.com/andybalholm/brotli@v1.2.0 (fixed in v1.2.7), GO-2026-6865 in github.com/klauspost/compress@v1.19.1 (fixed in v1.20.2), and GO-2026-6859 in net/http@go1.26.10 (fixed in go1.26.11). The PR diff touches only pkg/sentry/execution/mempool_removals.go and pkg/sentry/execution/mempool_removals_test.go; go.mod still pins klauspost/compress v1.19.1 (line 35), brotli v1.2.0 (line 81) and go 1.26.10 (line 3), identical to origin/master. The same three advisories also failed govulncheck on unrelated branches feat/mempool-lifecycle-events (run 42974332955) and feat/libp2p-goodbye (run 42970596458). (.github/workflows/govulncheck.yaml:65) — Bump github.com/klauspost/compress to v1.20.2 and github.com/andybalholm/brotli to v1.2.7 (go get + go mod tidy), and move go.mod:3, .tool-versions:1 and Dockerfile:1 to Go 1.26.11 for GO-2026-6859; if the build image cannot move yet, add GO-2026-6859 (and the other two IDs if the module bumps are deferred) to the ALLOWED list at .github/workflows/govulncheck.yaml:65. Re-running this PR's job alone will fail again.

Diagnosed govulncheck @ 6c4bfba5
"Don't drink the bear's skin before you've shot it." — Hungarian

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants