feat(learn): add continuous knowledge consolidation layer#84
Open
ashu17706 wants to merge 1 commit into
Open
Conversation
Adds `smriti consolidate` and `smriti learnings`, applying Progressive Summarization: cheap Stage-1 segmentation runs broadly over dense sessions into a new smriti_knowledge_units table, and expensive Stage-2 polish (existing segmentSession/generateDocument pipeline) only runs once a unit proves reuse via recall or scored high relevance at extraction time. - src/db.ts: smriti_knowledge_units table + CRUD helpers (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits, incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits) - src/search/recall.ts: track retrieval_count on every recall() path - src/learn/consolidate.ts: segment + promote phases, reusing the existing 3-stage segmentation pipeline from src/team/segment.ts and document.ts - src/index.ts, src/format.ts: CLI wiring for `consolidate` and `learnings` CLI-only, not wired into the daemon — consolidation runs two LLM stages, which the daemon's flush path deliberately avoids (see the enrichOnIngest comment in src/daemon/index.ts). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Smriti currently only captures sessions and can manually distill one batch of them via
smriti share --segmented. Nothing persists locally unless a user runsshare, and nothing tracks whether a piece of knowledge is a one-off or something that keeps coming back.This PR adds a Continuous Knowledge Consolidation Layer, applying two ideas from Building a Second Brain (Tiago Forte):
Progressive Summarization — cheap Stage-1 extraction (
segmentSession) runs broadly over dense sessions; expensive Stage-2 polish (generateDocument) only runs once a unit proves it's reused (recalled repeatedly) or scored high relevance at extraction time.Intermediate Packets — the existing
KnowledgeUnittype now has somewhere to live before export, in a newsmriti_knowledge_unitstable, so reuse can accumulate.src/db.ts— newsmriti_knowledge_unitstable + CRUD helpers (insertKnowledgeUnit,findUnsegmentedDenseSessions,findPromotableUnits,incrementRetrievalCount,promoteKnowledgeUnit,listKnowledgeUnits)src/search/recall.ts— tracksretrieval_counton everyrecall()exit path (best-effort, never breaks recall)src/learn/consolidate.ts— newconsolidateKnowledge(): segment phase (dense, unsegmented sessions → Stage 1) + promote phase (units that cleared the reuse/relevance bar → Stage 2 → written to.smriti/knowledge/+ recorded insmriti_shares)src/index.ts,src/format.ts— CLI wiring forsmriti consolidateandsmriti learningssrc/team/share.ts— exportedgetSessionMessagesfor reuse by the new moduleCLI-only, not wired into the daemon — consolidation runs two LLM-driving stages, more than
enrich, which is already excluded from the daemon's flush path for the LlamaCpp-crash reason documented insrc/daemon/index.ts.Test plan
test/learn-consolidate.test.tscovers: segment-phase content-hash dedup, promotion threshold (retrieval/relevance bar), graceful degradation (a per-unit write failure doesn't abort the run), and retrieval tracking viarecall()bun test --cwd ./test— 378 pass, 0 fail across all 31 files, no regressionssmriti consolidate --min-density 0andsmriti learningsrun cleanly against a fresh DBGenerated by Claude Code