feat(health): surface search-index freshness

[sbynode-ux]

Problem

When index persistence stops working, nothing says so.

• search keeps answering, because the in-memory index is fine
• /agentmemory/health keeps returning healthy
• the only trace is a warn line that FAILURE_LOG_THROTTLE_MS emits at most once a minute
• the loss materialises on the next restart, when the server loads whatever snapshot last succeeded

On an install here, state::set had been timing out for 20 days. Every restart reloaded a snapshot of 4944 docs against a real corpus of 12 801 — 61% of memory silently unsearchable — and the install reported itself healthy the whole time. I only found it by comparing a file mtime against kv.list counts.

(That specific timeout is what the sharded manifest work in 0.9.25 addresses. This PR is about the class of failure being visible, whatever the cause: a shard write that fails, a KV that goes read-only, a disk that fills.)

Change

IndexPersistence tracks its own outcome and exposes it:

getIndexPersistenceStatus(): {
  bm25Docs, vectors,
  lastSavedAt, lastError,
  dirtySince,   // when the index first went dirty with no save since
  stale,        // dirty for over an hour
}

scheduleSave() marks dirty, a successful save() clears it and records counts, a failed one records the error. The health snapshot carries the status and evaluateHealth reports it:

• note: index_docs_12400_vec_12400_saved_5m_ago
• alert (degrades): index_persist_error
• alert (degrades): index_persist_stale_<n>m

A snapshot without indexPersistence produces exactly today's output, so anything constructing HealthSnapshot itself is unaffected.

Verification

test/index-freshness.test.ts (new, 5 cases): the note with counts and save age; degrade + index_persist_stale_ past the window; degrade + index_persist_error on a failed save; silence when the snapshot carries no index info; and the dirty-tracking transition on scheduleSave().

Full suite green (1601 passed, 1 skipped), npm run build clean. The three strict-mode tsc errors touching these files are pre-existing on main.

Run the suite with an isolated HOME — a real ~/.agentmemory/.env on the machine leaks into 14 unrelated tests.

Summary by CodeRabbit

• New Features

  • Health monitoring now reports search-index persistence status, including document counts, save times, errors, and freshness.
  • Healthy indexes provide informational metrics, while stale or failed persistence is surfaced as degraded health.
  • Indexes are tracked as pending during saves and marked current after successful persistence.

• Tests

  • Added coverage for healthy, stale, failed, unavailable, and pending index states.

[vercel]

@sbynode-ux is attempting to deploy a commit to the rohitg00's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change tracks search-index persistence state, adds it to health snapshots, evaluates stale or failed persistence, and adds tests for healthy, stale, failed, unavailable, and dirty states.

Changes

Index persistence health

Layer / File(s)	Summary
Persistence status tracking src/state/index-persistence.ts	Tracks BM25/vector counts, save timestamps, errors, dirty state, and one-hour staleness. Save scheduling and completion update this state.
Health snapshot reporting and evaluation src/types.ts, src/health/monitor.ts, src/health/thresholds.ts, test/index-freshness.test.ts	Adds optional persistence data to health snapshots, reports it from the monitor, evaluates alerts and notes, and tests the resulting behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant IndexPersistence
  participant HealthMonitor
  participant evaluateHealth
  IndexPersistence->>HealthMonitor: provide persistence status
  HealthMonitor->>evaluateHealth: evaluate snapshot
  evaluateHealth-->>HealthMonitor: persistence alerts or notes

Loading

Possibly related PRs

• rohitg00/agentmemory#1136: Both changes modify search-index persistence tracking and related tests.

Suggested reviewers: rohitg00

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: exposing search-index freshness through health reporting.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

src/state/index-persistence.ts (1)

28-32: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove explanatory comments from source files.

• src/state/index-persistence.ts#L28-L32: remove the persistence-layer explanation.
• src/state/index-persistence.ts#L38-L38: remove the field explanation.
• src/state/index-persistence.ts#L52-L52: remove the health-snapshot explanation.
• src/state/index-persistence.ts#L60-L60: remove the test-seam explanation.
• src/types.ts#L214-L214: remove the persistence-status explanation.

As per coding guidelines, src/**/*.ts must not add comments that explain code. Use clear naming instead.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/state/index-persistence.ts` around lines 28 - 32, Remove the explanatory
comments from src/state/index-persistence.ts at lines 28-32, 38, 52, and 60, and
from src/types.ts at line 214. Leave the associated persistence implementation,
fields, health snapshot, test seam, and status type unchanged.

Source: Coding guidelines

test/index-freshness.test.ts (1)

85-105: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Exercise the stale transition. Use fake timers, call persistence.stop() to cancel the 5-second debounce, advance time by 60 * 60 * 1000 + 1, and assert getIndexPersistenceStatus().stale is true. Restore real timers in afterEach.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/index-freshness.test.ts` around lines 85 - 105, Extend the test around
IndexPersistence.scheduleSave to use fake timers, call persistence.stop() to
cancel the debounce, advance time by 60 * 60 * 1000 + 1, and assert
getIndexPersistenceStatus().stale is true. Add an afterEach hook that restores
real timers, while preserving the existing clean and immediately-dirty
assertions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/state/index-persistence.ts`:
- Line 125: Update scheduleSave() and save() to track a monotonically increasing
revision for each scheduled update, capturing the revision when save begins and
clearing dirtySince only when that revision is still current after KV writes
complete. Preserve dirtySince when a newer update was scheduled during the
active save so health continues to report the index as dirty until the latest
save finishes.

---

Nitpick comments:
In `@src/state/index-persistence.ts`:
- Around line 28-32: Remove the explanatory comments from
src/state/index-persistence.ts at lines 28-32, 38, 52, and 60, and from
src/types.ts at line 214. Leave the associated persistence implementation,
fields, health snapshot, test seam, and status type unchanged.

In `@test/index-freshness.test.ts`:
- Around line 85-105: Extend the test around IndexPersistence.scheduleSave to
use fake timers, call persistence.stop() to cancel the debounce, advance time by
60 * 60 * 1000 + 1, and assert getIndexPersistenceStatus().stale is true. Add an
afterEach hook that restores real timers, while preserving the existing clean
and immediately-dirty assertions.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7f7cc4e0-e4b4-450d-93de-1d421890ff3a

📥 Commits

Reviewing files that changed from the base of the PR and between 2973e4e and 49cf34f.

📒 Files selected for processing (5)

• src/health/monitor.ts
• src/health/thresholds.ts
• src/state/index-persistence.ts
• src/types.ts
• test/index-freshness.test.ts

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Keep dirty state for updates scheduled during an active save.

If scheduleSave() runs while an earlier save() awaits KV writes, the earlier save clears dirtySince at Line 150. The newer update can still be unsaved. Health then reports a clean index until the newer save completes.

Track a revision for each scheduled update. Clear dirtySince only if no later revision was scheduled.

Proposed fix

 let status: IndexPersistenceStatus = {
   // ...
 };
+let dirtyRevision = 0;

 export function resetIndexPersistenceStatus(): void {
   status = {
     // ...
   };
+  dirtyRevision = 0;
 }

 scheduleSave(): void {
   if (status.dirtySince === null) status.dirtySince = Date.now();
+  dirtyRevision += 1;
   // ...
 }

 async save(): Promise<void> {
+  const saveRevision = dirtyRevision;
   // ...
   try {
     // ...
     status.lastSavedAt = new Date().toISOString();
     status.lastError = null;
-    status.dirtySince = null;
+    if (dirtyRevision === saveRevision) status.dirtySince = null;

Also applies to: 146-150

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/state/index-persistence.ts` at line 125, Update scheduleSave() and save()
to track a monotonically increasing revision for each scheduled update,
capturing the revision when save begins and clearing dirtySince only when that
revision is still current after KV writes complete. Preserve dirtySince when a
newer update was scheduled during the active save so health continues to report
the index as dirty until the latest save finishes.