fix(insights): make large-history scans fast and resumable

.github/workflows/ci.yml

@@ -144,13 +144,17 @@ jobs:
     name: Herdr Insights
     runs-on: ubuntu-latest
     timeout-minutes: 10
+    strategy:
+      fail-fast: false
+      matrix:
+        node: ['22.15.0', '22', '24']
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
       - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
         with:
-          node-version: '24'
+          node-version: ${{ matrix.node }}
           package-manager-cache: false
       # Dependency-free; native Herdr checks run locally, not on headless CI.
       - run: npm run test:insights

herdr-insights/INSTALL.md

@@ -27,3 +27,12 @@ only if requested: click **Insights**, or add `-- --open` to the install command
 Stop and report; do not force overwrites or rebuild to bypass errors. Preserve
 local edits and any failed layout clone. Remove the app home's
 `.insights-install.lock` directory only after confirming no installer/uninstaller is running.
+
+Long scans are expected. Foreground refreshes are bounded and auto-continue;
+giant zstd archives may finish in one managed local Node worker. Do not raise
+Herdr's exec timeout. Lock/claim files fail closed: never delete a live or
+foreign owner by age. Remove stale lock, `lock.acquire-*`, or quarantine
+leftovers only after proving no writer is running and rechecking first.
+
+Node.js **22.15+** is required. A partial save updates `savedAt`/`lastScanAt`,
+not `lastCompletedScanAt`.

herdr-insights/README.md

@@ -37,6 +37,28 @@ click-state rules. Set `HERDR_APP_HOME` to use a different app home.
 - Open **Settings** to choose sources, edit paths, toggle cached tokens, or
   change the palette and refresh interval.
 
+## Long scans and recovery
+
+- A foreground refresh is a bounded slice. When work remains it returns
+  committed data, the dashboard keeps rendering, and the view automatically
+  continues. Herdr's exec timeout is unchanged.
+- A giant non-seekable zstd archive that cannot finish in one slice continues
+  in one managed local detached Node worker. Polling never restarts it. There
+  is no fabricated compressed-offset resume; after a crash the unfinished
+  archive and any uncheckpointed batch are replayed, while completed files stay
+  committed.
+- `prepare` and status reads use a metadata-only cache plus sanitized live progress
+  (current file, decoded bytes, counters). Only a matching live local worker
+  claim and writer lock at the same generation may publish it; rows, config,
+  and accounting always come from committed state.
+- `savedAt`/`lastScanAt` is the last persisted write; `lastCompletedScanAt` is
+  the last fully completed scan. A saved partial scan is not a completed scan.
+- Locks fail closed. Empty, foreign, or legacy lock state is never age-evicted
+  automatically. Clean up manually only after proving no writer is running;
+  `lock.acquire-*`/quarantine leftovers are inactive, but recheck before
+  deleting anything.
+- Node.js 22.15+ is required.
+
 ## Privacy
 
 No network requests or uploads. The mod runs a local Node helper, reads session
@@ -57,3 +79,21 @@ without modifying them (see [INSTALL.md](INSTALL.md)).
 Developer tests, fixtures, and release validation live outside this distribution;
 see the repo's [development guide](../docs/herdr-insights-development.md). They are
 not installation steps.
+
+### Reproducible performance benchmark
+
+The opt-in benchmark uses only synthetic sessions and never reads a real home:
+
+```sh
+node scripts/herdr-insights/benchmark.mjs
+```
+
+It starts three fresh presenter processes per case and records medians plus peak
+RSS, source/store/payload bytes, filesRead, bytesRead, structuredClone counts,
+and every gate in `research/issue-109/round2-perf-benchmark.json` (gitignored).
+Cases cover 1,000 and 10,008 unique compressed Codex sessions cold and warm, a
+single 250,000-event session, and 50MiB/500MiB plain and zstd payloads with fixed
+usage cardinality. It also checks cached `status` latency while a stream-throttled
+worker indexes. The benchmark is a development measurement, not native Herdr
+proof and not a Node 22 compatibility certification; see the artifact's pending
+gates for that separate work.

herdr-insights/RECIPE.md

@@ -162,8 +162,15 @@ including tool-result, standalone usage, compaction, and branch-summary events.
 Task duration uses exact boundaries or conservative capped spans, never idle session
 lifetime. Use local dates, not UTC days. Append-only incremental caches may require
 explicit rebuild after same-size rewrites. Use ownership-checked store locks;
-never steal live/unknown owners' locks based on age. Preserve corrupt stores with
-a backup before rebuilding; read-only prepare must not repair or create stores.
+never steal live/unknown owners' locks based on age or evict empty/foreign locks
+automatically. Preserve corrupt stores with a backup before rebuilding; read-only
+prepare must not repair or create stores. Foreground slices are bounded and
+auto-continue; a managed local worker may finish a giant non-seekable zstd archive.
+There is no arbitrary compressed-offset resume: crash recovery replays the
+unfinished archive and uncheckpointed batch, keeping completed commits. Live
+progress is a metadata-only overlay gated by the matching live claim and writer
+lock; rows, config, and accounting stay committed. `savedAt`/`lastScanAt` is not
+`lastCompletedScanAt`. Require Node >=22.15; do not change Herdr's exec timeout.
 
 Never modify shipped layouts/mods, keys, source transcripts, or existing usage
 data. Write only the owned personal layout clone. Preserve collisions and local