Skip to content

Cut fire-history write load and register meters lazily - #16

Merged
mirzabob merged 14 commits into
mainfrom
develop_shantanu
Oct 15, 2026
Merged

mirzabob merged 14 commits into
mainfrom
develop_shantanu

Conversation

@tshan10

@tshan10 tshan10 commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Why: a deployment with ~800 magazines × 32 shards saw a burst of Aerospike calls every checkpoint window. Each process wrote one checkpoint per shard per window for every magazine, whether or not anything had changed.

What changed

  • Skip unchanged shards. A shard whose fire pointer equals its newest checkpoint gets no write. The pointer only increases, so the old checkpoint stays true. Idle magazines cost nothing.
  • Skip windows already written. The active-shard refresh read now also fetches each shard's newest checkpoint (index 0 of the map, not the whole map). If any process has already written this window, the shard is skipped.
  • Batch. The remaining shards are written in one operate(BatchPolicy, List) call per magazine instead of one call per shard.
  • Spread writes. Background writes wait a random offset in the first half of each window, chosen per process and per magazine. The first process to write is usually the only one. This is always on with fire history and not configurable.
  • Window-end labels. Checkpoints are keyed by the window's end. The window start could precede the observation by up to half a window once writes are delayed, which would claim a pointer position earlier than it was actually reached.
  • firePointerBefore applies the same skip rules, without the delay. It still advances history when nothing is consuming.
  • Refactor. New FireHistoryStore (windows, checkpoints, per-process state). MagazineMetadataStore is back to pointers and counters and delegates; its public API is unchanged. FireHistoryConfig moved into FireHistoryStore.
  • Lazy meters. MagazineMetrics registers each counter and timer on first use instead of 30 per magazine up front. It stays allocation-free once a meter exists.

Compatibility

  • No public API changes.
  • Requires Aerospike server 6.0 or later for batch writes, which batch deletes already required.
  • Rolling upgrades are safe: entries labelled by 2.0.1 stay readable, and racing writers are harmless.
  • write_fire_history now counts batch calls.
  • Meters that have never recorded don't exist until they do.

@sonarqubecloud

Copy link
Copy Markdown

@mirzabob
mirzabob merged commit a8f1adb into main Oct 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants