LLM-642: conversation retention in the app, with quota accounting and tombstones - #334
Merged
Merged
Conversation
… tombstones Conversation notes were retired by raw SQL in scripts/db-cleanup.sh (root cron, 03:15): soft-delete at retention+1 days, hard-delete a window later. That bypassed namespace_usage, so the quota never drained (home counted 51 MB against 18 MB live), and the hard-delete made the memory-sync diff report every retired session as missing, so the client re-uploaded it as a new row with a new created_at — 359 home / 363 work sessions on 2026-08-21 alone, and the same three on this box today, 16 hours after retirement. - services/cleanup.js: retireConversations() runs as task 3 of the nightly cleanup cron. Soft-delete credits the quota like deleteNote; the second stage empties content and chunks but keeps the row as a tombstone with a purged_at stamp. runDecayCleanup now credits the quota too. - routes/agent.js: the sync diff counts deleted and tombstoned rows as present (classifySessions); stale only when the file outgrew every size the server holds. - services/documents.js: restoreNote refuses a tombstone (410); updateUsage exported. - scripts/db-cleanup.sh: conversation steps removed. - migrations/MEM-145: re-date re-uploaded conversations to their session date, rebuild namespace_usage from live rows, reword the config key. Rehearsed on a clone of production: counters land exactly on the live totals; the first nightly run then retires 424 home / 420 work sessions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ing, decay race, honest counts - retireConversations ages a conversation by metadata.session_date (the session's own date), falling back to created_at; pg_input_is_valid guards the cast, so a malformed client-supplied date skips instead of throwing. Same guard in MEM-145's re-dating step. - runDecayCleanup's UPDATE now requires deleted_at IS NULL, so a note deleted between the candidate SELECT and the UPDATE is not credited twice. - softDeleted reports the UPDATE's row count; chunk failures are counted and logged separately instead of shrinking the retired count. - classifySessions lowercases the client id before lookup; test added. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Conversation notes were retired by raw SQL in
scripts/db-cleanup.sh(root cron, 03:15): soft-delete at retention+1 days, hard-delete a window later. That bypassednamespace_usage(home counted 51 MB against 18 MB live), and the hard-delete made the memory-sync diff report every retired session as missing, so the client re-uploaded it as a new row with a newcreated_at: 359 home / 363 work sessions on 2026-08-21 alone, and three more on the home box on 09-02, sixteen hours after retirement.Change
services/cleanup.js:retireConversations()runs as task 3 of the nightly cleanup cron. Ages a conversation by itssession_date(created_at fallback), soft-deletes withdeleteNote-style quota accounting, then a window later empties content and chunks but keeps the row as a tombstone (metadata.purged_at).runDecayCleanupnow credits the quota too, with adeleted_at IS NULLguard.routes/agent.js: the sync diff counts deleted and tombstoned rows as present (classifySessions); stale only when the file outgrew every size the server holds.services/documents.js:restoreNoterefuses a tombstone (410);updateUsageexported.scripts/db-cleanup.sh: conversation steps removed.migrations/MEM-145: re-date re-uploaded conversations to their session date (888 rows), rebuildnamespace_usagefrom live rows, reword the config key. Rehearsed twice on a clone of production: counters land exactly on the live totals; the first nightly run then retires 424 home / 420 work sessions.Verification:
node --test220/220 (9 new). code_review round 1 findings all applied (safe date parsing, session-date predicate, decay race, honest counts, id case).Ticket: https://jeffdafoe.atlassian.net/browse/LLM-642
Agent: Home
🤖 Generated with Claude Code