Skip to content

fix(insights): make large-history scans fast and resumable - #110

Draft
GaganSD wants to merge 14 commits into
mainfrom
fix/insights-large-history
Draft

GaganSD wants to merge 14 commits into
mainfrom
fix/insights-large-history

Conversation

@GaganSD

@GaganSD GaganSD commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

Validation

  • 217 tests pass on Node 22.15.0, 22.23.4, and 24.22.1; 29 tooling tests, package checks, and changed-file lint pass.
  • Final-head synthetic 10,008-file benchmark: 3.16 s cold, 0.62 s warm; zero warm transcript reads or store rewrites. Cached status responds in 56 ms median.
  • Verified concurrent launchers, kill/restart recovery, rebuild fencing, and a throttled worker lasting over 30 seconds.
  • Native loaded/settings/menu checks pass with 10,000 rows; an existing-app synthetic scan completes without mod errors.
  • Installed locally through the standard installer; staged native check passes, existing data and layout preserved.
  • Regression coverage includes concatenated/skippable/empty zstd frames and archive replacement on supported Node versions.

Compatibility

  • No new dependencies or raw-transcript persistence; Node 22.15+ remains required.
  • An interrupted compressed file is replayed from its beginning; committed checkpoints survive.
  • Legacy ownerless locks remain fail-closed and require verified manual recovery.

Replace whole-store clones with a precise per-file mutation journal,
add bounded foreground slices with plain-file offsets, incremental worker
checkpoints, generation fencing, and a committed presentation cache.
…omotion

- Record indexer failure status inside the store lock, re-reading the
  committed generation; never write when the lock was never held.
- Spawn the promoted indexer only after the launcher releases the lock;
  the worker waits a bounded handoff instead of exiting on EEXIST.
- Make worker claims exclusive-create and token-verified on release
  instead of last-write-wins/check-then-unlink.
- Fail closed on an unreadable generation fence during worker commits.
- Regression tests: queued generation survives a failing worker; one
  worker survives promotion with a held lock, no respawn, non-blocking status.
Add deterministic process integration coverage for four-launcher coalescing, SIGKILL checkpoint/restart exactness, generation fencing, and presenter responsiveness under the writer lock. Add a separate opt-in long-worker run that proves >30s synthetic throttled completion with one decoder pass. Clean up lifecycle temp roots and child processes.
Node 22 pipeline can wrap the destination ScanStopError as AbortError, which was misclassified as archive corruption and permanently fingerprinted a valid zstd file. Capture the exact stop at the destination and keep genuine stream failures distinct. Also validate zstd frame structure because Node 22 can silently accept truncated trailing frames, making existing corruption rollback tests runtime-dependent.
Replace the 2ms mtime tolerance with an exact dev/ino/size/mtimeNs fingerprint captured after the atomic store rename while writer authority is held. Validate the current store before and after reading the cache, so a crash before the cache update cannot serve old rows for a newer commit. Legacy un-fingerprinted caches fall back to the store.
Run the Herdr Insights suite on Node 22.15.0, latest Node 22, and Node 24 so the zstd budget compatibility bug cannot recur.
Node 22 stops a zstd decoder after the first frame of a concatenated stream. Split frames structurally and run a fresh decoder per data frame, skipping skippable frames, while feeding one shared line consumer. Truncated frames still roll back and fingerprint; budget stops stay retriable.
@GaganSD
GaganSD marked this pull request as ready for review November 2, 2026 15:24
@greptile-apps

greptile-apps Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 2/5

[Medium impact] The changes affect the local Insights scan and dashboard workflow.

Fix the generation mismatch, foreign-host signal target, and plain-file stall before merging.

Findings

  1. P1 Unchanged settings block refreshes ▶
  2. P1 Wrong local process gets stopped ▶
  3. P1 Long transcript lines stall indexing ▶
  4. P2 Worker exits during startup ▶
  5. P2 Empty frames bypass scan limits ▶

Summary

Replaces whole-history rollback copies with per-file transactions and adds resumable scans, a detached worker, committed presentation caches, and live progress.

  • File scans limit rollback work to the records they change.
  • Large history scans continue in one worker while the dashboard stays responsive.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Dashboard refresh] --> B{Writer lock available?}
  B -->|No| C[Read committed cache or store]
  B -->|Yes| D[Run bounded scan]
  D --> E[Commit accepted file changes]
  E --> F{Unfinished archive or database?}
  F -->|Yes| G[Claim and start worker]
  F -->|No| C
  G --> H[Worker holds writer lock]
  H --> I[Commit completed files]
  I --> C
  C --> J[Show committed dashboard]
  J --> K{Work remains?}
  K -->|Yes| A
Loading

Reviews (1) · Last reviewed commit: "fix(insights): reset archive parser cont..." · Reviewed by Greptile

Unchanged settings no longer leave a live fence ahead of the committed
generation. Signal only a same-host worker, publish claim pids without an
empty file, read plain lines through to a boundary, and honor budget and
cancel stops on empty and skippable zstd frames.
@GaganSD
GaganSD marked this pull request as draft November 2, 2026 16:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

herdr-insights: first scan never finishes on large session histories because Herdr stops it at 30 seconds

1 participant