Repository navigation
Conversation
… gateway listRuns parsed every stored record to build one page, so its cost grew with the bytes stored rather than the number of runs; with megabyte-sized records a 3-run page took seconds. The daily retention sweep did the same inside the gateway process, synchronously. - The store keeps a small summary of each top-level run at stateDir/_index/<run>.json, written after every record write. An entry names the record version it was built from (mtime to the millisecond, and size) and is used only while the record still matches, so a record rewritten by any process is never listed stale. The index is derived data: a missing or stale entry means the record is parsed. Older readers skip it (the name starts with "_"). - listRuns reads entries and parses a record only when its entry is missing or stale. Measured on 150 runs / 273 MB: 243 ms per page before, 3 ms after. - sweepRuns replaces pruneRuns: it backfills missing or stale entries, applies retention using the entries, drops entries whose run is gone, and yields to the event loop as it goes. It runs daily in the serving process whether or not retention is enabled. - Version 1.7.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
listRuns parsed every stored record to build one page, so its cost grew with
the bytes stored rather than the number of runs; with megabyte-sized records a
3-run page took seconds. The daily retention sweep did the same inside the
gateway process, synchronously.
stateDir/index/.json, written after every record write. An entry names
the record version it was built from (mtime to the millisecond, and size) and
is used only while the record still matches, so a record rewritten by any
process is never listed stale. The index is derived data: a missing or stale
entry means the record is parsed. Older readers skip it (the name starts
with "").
stale. Measured on 150 runs / 273 MB: 243 ms per page before, 3 ms after.
retention using the entries, drops entries whose run is gone, and yields to
the event loop as it goes. It runs daily in the serving process whether or
not retention is enabled.