Skip to content

feat(learn): add continuous knowledge consolidation layer#84

Open
ashu17706 wants to merge 1 commit into
mainfrom
claude/refine-local-plan-ff68rg
Open

feat(learn): add continuous knowledge consolidation layer#84
ashu17706 wants to merge 1 commit into
mainfrom
claude/refine-local-plan-ff68rg

Conversation

@ashu17706

Copy link
Copy Markdown
Contributor

Summary

Smriti currently only captures sessions and can manually distill one batch of them via smriti share --segmented. Nothing persists locally unless a user runs share, and nothing tracks whether a piece of knowledge is a one-off or something that keeps coming back.

This PR adds a Continuous Knowledge Consolidation Layer, applying two ideas from Building a Second Brain (Tiago Forte):

  • Progressive Summarization — cheap Stage-1 extraction (segmentSession) runs broadly over dense sessions; expensive Stage-2 polish (generateDocument) only runs once a unit proves it's reused (recalled repeatedly) or scored high relevance at extraction time.

  • Intermediate Packets — the existing KnowledgeUnit type now has somewhere to live before export, in a new smriti_knowledge_units table, so reuse can accumulate.

  • src/db.ts — new smriti_knowledge_units table + CRUD helpers (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits, incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits)

  • src/search/recall.ts — tracks retrieval_count on every recall() exit path (best-effort, never breaks recall)

  • src/learn/consolidate.ts — new consolidateKnowledge(): segment phase (dense, unsegmented sessions → Stage 1) + promote phase (units that cleared the reuse/relevance bar → Stage 2 → written to .smriti/knowledge/ + recorded in smriti_shares)

  • src/index.ts, src/format.ts — CLI wiring for smriti consolidate and smriti learnings

  • src/team/share.ts — exported getSessionMessages for reuse by the new module

CLI-only, not wired into the daemon — consolidation runs two LLM-driving stages, more than enrich, which is already excluded from the daemon's flush path for the LlamaCpp-crash reason documented in src/daemon/index.ts.

Test plan

  • New test/learn-consolidate.test.ts covers: segment-phase content-hash dedup, promotion threshold (retrieval/relevance bar), graceful degradation (a per-unit write failure doesn't abort the run), and retrieval tracking via recall()
  • bun test --cwd ./test — 378 pass, 0 fail across all 31 files, no regressions
  • Manual smoke test: smriti consolidate --min-density 0 and smriti learnings run cleanly against a fresh DB

Generated by Claude Code

Adds `smriti consolidate` and `smriti learnings`, applying Progressive
Summarization: cheap Stage-1 segmentation runs broadly over dense sessions
into a new smriti_knowledge_units table, and expensive Stage-2 polish
(existing segmentSession/generateDocument pipeline) only runs once a unit
proves reuse via recall or scored high relevance at extraction time.

- src/db.ts: smriti_knowledge_units table + CRUD helpers
  (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits,
  incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits)
- src/search/recall.ts: track retrieval_count on every recall() path
- src/learn/consolidate.ts: segment + promote phases, reusing the existing
  3-stage segmentation pipeline from src/team/segment.ts and document.ts
- src/index.ts, src/format.ts: CLI wiring for `consolidate` and `learnings`

CLI-only, not wired into the daemon — consolidation runs two LLM stages,
which the daemon's flush path deliberately avoids (see the enrichOnIngest
comment in src/daemon/index.ts).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants