Conversation
WIP note plus local enqueue/score helpers. No product behavior change. Co-authored-by: aria <aria@engramhq.com>
ProcessAvailable used to drop idle goroutines after five empty claims, so newly enqueued subjects waited on a shrunken pool during slow extract. Idle slots now wait while another slot holds a job; per-subject FIFO and lease fencing stay in ClaimNext. Co-authored-by: aria <aria@engramhq.com>
Product scoring sends mode=answer, keeps transport failures out of not-in-memory, and treats malformed judge JSON as UNRESOLVED so final JSON cannot be written with a substring CORRECT fallback. The runner persists identity and per-question traces, rejects resume when the API or store changed, and refuses an empty queue as success. Co-authored-by: aria <aria@engramhq.com>
The previous full-product run wrote JSON only at the end, so a VM reset lost every judgment. This wrapper pins loopback, refuses /tmp artifacts, and scores the restored hash-embedding tenant with eval-protocol-v2. Co-authored-by: aria <aria@engramhq.com>
Hash+trigram search misses the single token "conversation" on some restored LoCoMo tenants, which aborted skip-ingest verify_store before any /recall scoring. Co-authored-by: aria <aria@engramhq.com>
run_staged imports ingest_conversation from run_smoke, so skip-ingest n=1540 scoring crashed on IndentationError before preflight. Co-authored-by: aria <aria@engramhq.com>
5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the first implementation slice of the qualified-memory goal: trustworthy measurement plus the worker-pool bug that made LoCoMo extract look single-digit parallel.
bbe55f7, plus the n=1540 handoff helpers from PR docs: n=1540 product-recall handoff for next-cycle planning #185 (not leftover-covering)./recallscoring for qualification is versioned aseval-protocol-v2: alwaysmode: "answer", transport failures stay transport failures, malformed judge JSON isUNRESOLVEDand blocks final JSON.run_smoke.pystill defaults to v1 so a restarted old scorer stays v1 if--eval-protocolis unset.The previous live n=1540 skip-ingest run died on VM reset with no result JSON. The extract dump survived at
/opt/cursor/artifacts/n1540-live/brainy_n1540_20260911T080043Z.dump(1571/1571 jobs, 22,757 128-d hash embeddings). This PR restores a durable remasure path: local API on:18200against that dump, incremental JSONL under/opt/cursor/artifacts/eval-runs/locomo-full-n1540-bbe55f7-protocol-v2.This PR does not claim a LoCoMo 90% pin, replace README 11.4%, or merge leftover-covering branches.
Changes
internal/jobs/processor.go—ProcessAvailablewaits whileinFlight > 0instead of exiting after five empty claims.internal/store/postgres/runtime.go— includecurrent_database()in/runtimefor store identity.evals/public/protocol.py,client.py,staged_runner.py,locomo/run_staged.py— resumable stagespreflight → enqueue → drain → verify_store → answer → judge → diagnose → finalize.evals/public/judge.py— v2 product recall and judge behavior; v1 substring CORRECT fallback retained behind the protocol flag.evals/public/locomo/run_smoke.py— restore thepublic.schemaimport that protocol-v2 accidentally overwrote (IndentationError blocked skip-ingest).evals/tools/n1540_run_staged_recall.sh— local-only skip-ingest scorer; refuses non-loopbackBRAINY_BASE_URLand/tmpartifacts.verify_storeprobesthe/I/conversationso hash+trigram miss on a single token does not abort skip-ingest.Test plan
gofmtclean on touched Go filesgo vet ./...go test ./...cd evals && python3 -m unittest public.test_staged_runner public.test_public_framework(49 tests)brainy_n1540(1571 jobs, 22757 128-d embeddings) → local API:18200hash embedder,BRAINY_RECALL_LLM=1→GET /healthzok,POST /recall200 on loopbackevals/tools/n1540_run_staged_recall.sh, run idlocomo-full-n1540-bbe55f7-protocol-v2, 1540 questions, incrementalanswers.jsonl(hash embeddings; not qualification-profile)Do not score leftover-covering
pr/locomo-180-p29-1e9e.Linear / tracking
Qualified-memory goal (no Linear id in the request). Follow-on work: S1 live-provider audit, S5 RRF/pgvector, S2/S3 planner budgets, S4 rebuild, staging + reference clients.
Risk / follow-ups
text-embedding-3-small768 and no hash fallback.run_smokedefault remains v1; callers must pass--eval-protocol eval-protocol-v2or userun_staged./recallcode is stillbbe55f7. Do not commit further harness changes untilresult.jsonlands, or resume will reject the identity.