Repository navigation
Give vinga a lookup over its pages: search_docs, gate, docs - #638
Merged
Merged
Conversation
The #612 gate's inputs were kept outside the repository; plan review round 2, finding 6, asked for them here, so a later maintainer can rerun and judge the gate without the session's working files. - frozen.json: the 32 questions the gate froze before its first model call, byte for byte (sha256 1442b569...). - amendments.json: the key facts of B4 and S5 follow the pages M1b changed (vinga info no longer prints a try link; vinga device invite prints an invite link). The questions' text does not change. - rephrased.json: the same questions in other words, written and hashed before retrieval was touched, for M5's second model run. - queries.json: the 39 search() calls the models made on lookup questions in the gate's shape-A runs. - scoring.py: the gate's automatic rules, with an undecided category where the hand-check used to decide, reported and never credited. A unit test holds the frozen set to its hash, both sets to the same 32 ids, and every key fact to the page it quotes, found by text in the packaged copy; it failed when one phrase was edited out of a copy and passed once restored. Attribution: anthropic/claude-opus-5-5, thinking medium
knowledge.search(query, guide) is the lookup the #612 gate chose (shape A): the packaged pages cut at ## and ### headings into passages of at most 1,000 characters, scored by BM25 over word stems with the title counted twice, the session's own board guide weighed up by 1.5, and the best three answered within 3,300 characters, a passage past the budget left out whole with a line saying so. A query that matches nothing answers a sentence telling the model to say it does not know. boards.board_guide maps a reported board type to the guide's packaged path, so a caller may keep the path where it may not keep the string. Measured with no model against the 39 queries the models sent in the gate: 29 of 39 answers carry the fact the question needed with the LCD-1.54 guide weighed up, 25 of 39 with no board, against 22 of 39 in the gate. The plan's target was 32, and it is not met; the unit test holds the measured floor rather than the target, and the implementation doc records every variant measured. Every test was watched failing: the board weight dropped, any path weighed as a guide, stems not cut, the fence rule off, "On this page" kept, the budget not enforced, the left-out line dropped, long paragraphs left whole, the query echoed into an answer, and a level three heading titled without its section each failed a test here (.logs/m5-mutations.log). The budget mutation survived the first test set, since three passages always fit; the budget test was added for it. Attribution: anthropic/claude-opus-5-5, thinking medium
search_docs is a builtin (its name joins names.BUILTIN_TOOL_NAMES, so a call is recorded with source builtin and TOOL_SOURCES does not move), offered by BuiltinTools to the built-in agent alone, by the predicate its device-scoped memory and threads already use, and answered from knowledge.search with the session's board guide, a packaged path the runtime maps once at construction and never the reported string. An operator's agent that asks for it is answered as a name nobody publishes, by both askers of the shared withheld rule: the builtin source at the call, and the runtime before it answers anything about a call at all, so a mangled call is not told its arguments were wrong, which only a tool that exists would say. The rule gains the predicate beside the memory policy; a ToolExecution built without it withholds the lookup from everyone. The name is search_docs rather than the gate's bare search: a builtin's name is an entry name no mcp_servers entry may take, and search is one an operator's own search server could plausibly have. The persona says where the lookup fits: buttons and controls from the board's facts, anything else about the device or vinga searched first, and only what neither covers declined. A first gate run with M4's "from those facts alone" wording had the model decline three of the first six questions that needed the lookup without making it. The session tests and their mutations are in the commit that tests the lookup in a session. Attribution: anthropic/claude-opus-5-5, thinking medium
server.llm_lookup_first_token_timeout_s (default 30) bounds the first token of the one round that follows a search_docs call, in place of llm_first_token_timeout_s and never shorter than it: that round sends the passages found, and a small local model reads them before it says a word. The original gate measured that wait at a median 44 s for Gemma 4 e4b on a Raspberry Pi 5, against the 10 s watchdog. ProviderWatch.reply_stream takes the round's own bound as an optional argument and keeps its retry and give-up unchanged, so a round after a lookup that stalls twice is given up as FirstTokenTimeout like any other. The tool loop sets it for the round after a lookup and clears it for every other round. The LLM clients' own read timeout (providers/kit.py, 30 s without a byte, not configuration) still applies, so against a runner that sends nothing before its first token the allowance buys nothing past 30 s; the key's description, config.example.yaml and docs/run/llm.md say so. Attribution: anthropic/claude-opus-5-5, thinking medium
The round after vinga's lookup is the slow one on a small local model, so the tool loop asks the filler runner to hold: the agent's next filler phrase, played at once when a round's calls include search_docs, before the tools run. It is the one exception to one filler per turn: the turn's timer covers the wait for the first round, and a lookup starts a second wait after it, so the hold plays whether or not the timer already has. An unfired timer is stood down in its favour, so a turn still holds one clip task and tail and settle see it as they see the timer's; a clip still sounding is already holding, and the mask still yields to a user speaking or a barge-in being confirmed. The holding phrase is the agent's own filler, so a vinga whose filler section is off waits in silence through the lookup round, as every agent waits at the start of a turn. Attribution: anthropic/claude-opus-5-5, thinking medium
Driven through real sessions with scripted models: vinga is offered search_docs and an operator's agent is not; an operator's agent asking for it anyway, or mangling the call, is told there is no such tool; vinga's lookup answers from its board's guide, asks for a missing query, and says when nothing matched. The round after a lookup waits out its allowance where any other round is retried; past it, the round is retried once and given up as FirstTokenTimeout; a shorter allowance bounds nothing tighter; and it is the round after the lookup alone. The lookup plays the agent's filler as a holding phrase, no other tool does, and a turn whose timer already played still holds. A credential-shaped query reaches no log line in either format and no span with LLM input export off; with it on, it is in the tool span's arguments and the LLM spans' messages, the export's own content fields, and nowhere else. tests/support/llm_input.traced takes the board and the agent, so a traced session can be vinga's. Every behavior was broken once and its test watched failing, then restored by copy and touch (.logs/m5-mutations.log): the lookup offered to every agent (the plan's M5 target), the withheld rule ignoring it, the runtime never told who is built in, the guide not handed over, the allowance never passed, applied after any tool, used when shorter or kept past its round, the hold never asked for, asked for after any tool, skipped once the timer had played or held to the timer's speaking-started rule, and the query logged by the tool. No survivor. Attribution: anthropic/claude-opus-5-5, thinking medium
tests/local/lookup_gate/harness.py asks one question set of a real model with what vinga is sent on the LCD-1.54 board: its persona and summary, the board's facts, the tools BuiltinTools offers the built-in agent on a device bound to it alone with search_docs among them, and the board's volume tool, through the server's own OpenAI-compatible provider at temperature 0 and seed 42 with thinking off, round after round as the tool loop runs them. The lookup is the real one; the other tools answer a fixed line. It reads the shared prefix once before the first question, unmeasured, since a cold runner on a Pi spends minutes on it, and resumes a stopped run where it stopped. Each answer is one JSON line with every round's first byte and first spoken text. tests/local/test_lookup_gate.py, opt-in with the rest of the lane, runs both sets and holds each to the gate's bar: at least 70% correct by the gate's own category, at most 10% hallucinated, and both device commands made. The hand-check is replaced by the scorer's undecided category, listed in the lane's summary and never credited. Attribution: anthropic/claude-opus-5-5, thinking medium
docs/reference/server-config.md gains the row for llm_lookup_first_token_timeout_s, written by its generator (vinga-server config reference server), as the drift check requires. Attribution: anthropic/claude-opus-5-5, thinking medium
The persona M5 first wrote still had Gemma 4 e4b decline every lookup question probed without calling search_docs: with vinga's real prompt and tools, D5, D9, B1 and S9 of the gate's frozen set got "I do not have information ... check the device guide" and no call. Probes (.logs/m5-probe-*.log) placed the cause in the hand-written persona: the gate's own prompt with vinga's tools looked up three of the four, and kept doing so with vinga's board facts or vinga's summary swapped in, and stopped the moment vinga's persona replaced its opening. Two smaller changes (only two tools offered, the gate's rule appended) did not move it. The persona now says where each answer comes from: buttons and controls from the board's facts, what vinga is from the summary, and anything else from search_docs first, searched before it ever says it does not know and never by sending the person to a guide instead. With it the same probe looked up two of the four. The pin follows the wording, and dropping the search-first sentence fails it (.logs/m5-mutations.log). Attribution: anthropic/claude-opus-5-5, thinking medium
The rephrased set asks "What is the vinga info command for?", which the command-spellings census read as the invocation `vinga info command` and failed as a command the tree does not have. The two question sets are a person's words frozen and hashed before any model answered them, so they join _HISTORICAL_PATHS beside the other committed captures, and the manifest is regenerated by its generator. Attribution: anthropic/claude-opus-5-5, thinking medium
server.llm_lookup_first_token_timeout_s goes, by Rafael's decision on 2026-10-09: it could not do what its name says. The LLM clients give up after 30 s without a byte (providers/kit.py, fixed rather than configuration), and the gate measured Ollama sending nothing before the first token: the round after a lookup reached its first byte in 32.9 to 51.3 s on a Raspberry Pi 5 with Gemma 4 e4b, every one of eleven past 30 s, so no allowance above 30 could have bought those rounds anything. The key, its config.example.yaml entry, the round's own bound in ProviderWatch.reply_stream, the tool loop's bookkeeping and the generated server reference row go. The holding phrase stays. A correction to "Give the round after a lookup its own allowance": its body said docs/run/llm.md explained that the read timeout caps the allowance. It did not; llm.md says it now about the round after a lookup, with the measurements, in this milestone's documentation. The tests for the allowance are replaced by tests that the round after a lookup is watched like any other (retried once, given up on a second stall) and that a retried round still hears one holding phrase. Each was watched failing: the watchdog's bound multiplied tenfold, the hold never asked for, and the hold asked for at every round (.logs/m5-mutations.log). Attribution: anthropic/claude-opus-5-5, thinking medium
Plan D3 wanted the vague text, what vinga is told when the server does not know the board, to send a device question to the common device guide "a lookup away". M4 could not, with no lookup yet, and had vinga decline and point to the guide. With search_docs offered to the built-in agent on every stack (Rafael's decision, 2026-10-09), the text now tells vinga to search for what the common guide says, never to guess which board this is, and to say so when the common guide does not cover the question. It also stops contradicting the persona, which tells vinga never to send someone to a guide instead of searching. The pin follows, and putting M4's decline-and-point wording back fails it (.logs/m5-mutations.log). Attribution: anthropic/claude-opus-5-5, thinking medium
The harness drives Anthropic's API through vinga's own AnthropicLlm
adapter when VINGA_LOCAL_LLM_PROVIDER=anthropic, its key read from
VINGA_DEV_ANTHROPIC_API_KEY and never printed, and records each
round's reported tokens. It sends exactly what a deployment's entry
sends: no temperature, since the adapter sets none and claude-sonnet-5
refuses one ("temperature is deprecated for this model", HTTP 400 on
the first attempt). A vendor's API has no cold prefix, so it is not
warmed.
The first Anthropic request also failed with "Tool names must be
unique": the harness appended search_docs to a list BuiltinTools
already put it in. Ollama accepted the duplicate silently, so every
Gemma run until then offered the lookup twice; the list is now each
tool once, asserted, and the Gemma runs were repeated.
Attribution: anthropic/claude-opus-5-5, thinking medium
M6 reworded the concepts page's wake rule: a fresh wake opens on the
first agent the device reaches. The frozen S2 fact quoted the old
sentence ("A fresh wake always gets the default agent"), and the
fixture test that holds every key fact to its page failed on the
rebased tree, which is what it is for. S2's fact now quotes the new
sentence and accepts "first agent" beside "default"; the question
does not change. The file is rewritten by json.dump, hence the
reindented lines.
Attribution: anthropic/claude-opus-5-5, thinking medium
docs/concepts.md (Agent), the glossary's vinga entry and the device guides' common page say what M5 built: vinga alone is offered a search over the concepts page, the glossary and the device guides packaged with the build, is told to search before it says it does not know, and never guesses which board an unknown device is. They say honestly how often it searches, from the 2026-10-09 measurements: claude-sonnet-5 on 13 of the 15 questions that needed it, 78% correct; Gemma 4 e4b, the local preset's model, on 5 of 15, 50% correct, so a small local model answers the controls and the summary reliably and deeper questions less often. The device page no longer says vinga points to a guide for a board it does not know. The packaged copy is regenerated with `uv run python -m tests.census.test_packaged_pages`. Attribution: anthropic/claude-opus-5-5, thinking medium
docs/run/llm.md gains "The built-in agent's lookup", beside M7's paragraphs on the local model and the cold first turn rather than over them: the same 32 questions, in two wordings, on claude-sonnet-5 and on Gemma 4 e4b on a Raspberry Pi 5, with what each answered, invented, searched, declined and commanded, how long a question took, and the tokens the vendor model spent. It says plainly that on the Pi the round after a search reached its first byte after 34 to 53 s, past the 30 s at which a request ends without a byte, so a turn that searches is given up there today, and that raising the watchdog cannot help. This is the explanation of the 30 s cap that "Give the round after a lookup its own allowance" promised in its body and did not deliver. docs/run/configuration.md's "Overriding vinga" names the lookup and links the measurements. Attribution: anthropic/claude-opus-5-5, thinking medium
The enumerated local baseline gains the built-in agent, as the plan's "Local baseline: joins" line said it would in the change that ships the lookup: vinga answers the device from its board's guide and vinga from the concept summary on local providers, and its lookup is offered there, used on a minority of the questions that need it by Gemma 4 e4b (5 of 15, 50% correct against claude-sonnet-5's 13 of 15 and 78%), and too slow on a Raspberry Pi 5 for the server to wait for. The promise is written from what the local stack measurably does: the first half works, the lookup is there, and how well it is used is the model's. It cites the plan that joins it, under the enumerated-baseline record. Attribution: anthropic/claude-opus-5-5, thinking medium
The implementation doc gains M5's section: the gate as measured (retrieval 29 of 39 against 32; Gemma 4 e4b 50% and 41% correct against 70%, with 0 and 1 hallucinated; claude-sonnet-5 78% and 81%), recorded as missed, with Rafael's decision of 2026-10-09 to ship the lookup on every stack and drop the allowance; why no synonym list was fitted to reach 32; every voided and superseded run; the deviations, resolutions and discoveries; and the mutations. The plan's Gate section records the decision and both measurements, and M5 is ticked with a link to its section. The changelog fragment carries the lookup, the unknown board's new behavior, and an Upgrade: line for an mcp_servers entry named search_docs. Attribution: anthropic/claude-opus-5-5, thinking medium
The milestone was ticked "PR TBD" before the pull request existed; #638 is its number, so the checklist links it. Attribution: anthropic/claude-opus-5-5, thinking high
Owner
Author
External review roundAutomated external review of this PR's diff (origin/main...8026ed2). Reviewed 2026-10-09 by openai/gpt-6.1-sol, thinking high via codex CLI 0.162.0, read-only sandbox, runtime 3m51s, at commit 8026ed2. Posted verbatim by the review run itself; resolutions follow as replies.
Verdict: mergeable after the listed fixes. |
PR #638's review, finding 2: the harness turned any query into text with str() and searched it, where the shipped search_docs refuses a missing, blank or non-string query with its own sentence, so the gate measured a friendlier tool than the one that ships. A lookup call in the harness is now answered by the shipped builtin, and a call whose arguments never parsed by the runtime's own sentence, which tool_execution names UNPARSEABLE_ARGUMENTS so both read one constant. A unit test drives harness.ask with a scripted model over a missing, an empty, a blank, a numeric and a list query and a mangled call, and each fails with the old str() line back (.logs/m5-mutations.log). The audit of the four recorded runs (frozen and rephrased, Gemma 4 e4b and claude-sonnet-5) read every search_docs call from their raw files for a missing, blank or non-string query or unparseable arguments: 55 calls, 0 that production would have refused (.logs/m5-audit-queries.log). The published numbers stand. Attribution: anthropic/claude-opus-5-5, thinking medium
PR #638's review, finding 1: run as documented with VINGA_LOCAL_OLLAMA naming a malformed address, the harness printed the address back, httpx's own words about it and the traceback. main() now contains every failure, building the provider included: the report is built in the except arm from failure_name(exc) alone (the fixed phrase for a class name it will not print) and said after the block, so neither the exception, its message nor the chain behind it outlives the arm. The harness exits 1 with one line, "the lookup gate stopped: InvalidURL" for that address. Two sentinel tests: the documented command run in a subprocess against http://localhost:<sentinel>/v1, holding stdout and stderr to that one line, sentinel-free and traceback-free; and an exception class whose name carries the sentinel and a newline, raised from a cause carrying it too, holding the output to the fixed phrase and leaving no exception live after main returns. Each was watched failing with the except arm removed, the message reported, the raw class name reported, and a re-raise from the arm (.logs/m5-mutations.log). Attribution: anthropic/claude-opus-5-5, thinking medium
PR #638's review, finding 4: docs/run/llm.md described the lookup measurements as taken "on an ESP32-S3-Touch-LCD-1.54" and called 1.8 s the time to the first spoken word. The harness sends vinga's prompt with that board's facts in it and reads the model's text back, with no board, no speech recognition or synthesis and no device output, and it times the first non-blank text. Both pages now say so: llm.md describes the harness and calls 1.8 s the time to first text, and the implementation doc's figure is labelled the same way. An untruncated `git grep` of docs/, changelog.d/ and README.md for "spoken word" and "on an ESP32-S3-Touch-LCD-1.54" found no other M5 claim of the kind; the plan's Risks line saying the gate "records time to first spoken word" is the plan's intention from before M5 and is left as written. Attribution: anthropic/claude-opus-5-5, thinking medium
PR #638's review, finding 3: docs/run/llm.md said a turn that searches on the Pi "is given up as a ProviderCallTimeout". With the defaults the 10 s first-token watchdog fires long before the 30 s transport read timeout: it cancels the round, sends it once more, and when the second attempt stalls too gives the turn up with a FirstTokenTimeout and the agent's fallback phrase, as M7's cold-turn paragraph above it says. A ProviderCallTimeout is reached only with the watchdog raised past 30 s. The harness waits for as long as the model takes, so its 34 to 53 s first bytes do not observe the retry; the page now says the running server's outcome is inferred from them, not observed. The same correction is made in the implementation doc, and where M5's other pages stated the outcome (the baseline item in product-promises.md, the changelog fragment, the plan's Gate paragraph) they now say it was measured in a model harness. The implementation doc's line about "the one hallucination on each side" is also reworded: there was one, in Gemma's rephrased run. Attribution: anthropic/claude-opus-5-5, thinking medium
The implementation doc's M5 section gains the round: the four findings, each adopted with its fix commit, and the audit of the four recorded runs for lookup calls production would refuse (55 calls, none), so the published numbers stand. Attribution: anthropic/claude-opus-5-5, thinking medium
Owner
Author
Resolutions for the external review roundAll four findings adopted; fixes by the milestone's implementer (anthropic/claude-opus-5-5, thinking medium), recorded in the implementation doc's M5 "PR review round" subsection (7d52961).
Checks after the fixes: ruff clean; the knowledge, builtin-tool, harness and session-lookup unit tests |
rafacm
added a commit
that referenced
this pull request
Oct 9, 2026
The milestone was ticked "PR TBD" before the pull request existed; #638 is its number, so the checklist links it. Attribution: anthropic/claude-opus-5-5, thinking high
rafacm
added a commit
that referenced
this pull request
Oct 9, 2026
PR #638's review, finding 2: the harness turned any query into text with str() and searched it, where the shipped search_docs refuses a missing, blank or non-string query with its own sentence, so the gate measured a friendlier tool than the one that ships. A lookup call in the harness is now answered by the shipped builtin, and a call whose arguments never parsed by the runtime's own sentence, which tool_execution names UNPARSEABLE_ARGUMENTS so both read one constant. A unit test drives harness.ask with a scripted model over a missing, an empty, a blank, a numeric and a list query and a mangled call, and each fails with the old str() line back (.logs/m5-mutations.log). The audit of the four recorded runs (frozen and rephrased, Gemma 4 e4b and claude-sonnet-5) read every search_docs call from their raw files for a missing, blank or non-string query or unparseable arguments: 55 calls, 0 that production would have refused (.logs/m5-audit-queries.log). The published numbers stand. Attribution: anthropic/claude-opus-5-5, thinking medium
rafacm
added a commit
that referenced
this pull request
Oct 9, 2026
PR #638's review, finding 1: run as documented with VINGA_LOCAL_OLLAMA naming a malformed address, the harness printed the address back, httpx's own words about it and the traceback. main() now contains every failure, building the provider included: the report is built in the except arm from failure_name(exc) alone (the fixed phrase for a class name it will not print) and said after the block, so neither the exception, its message nor the chain behind it outlives the arm. The harness exits 1 with one line, "the lookup gate stopped: InvalidURL" for that address. Two sentinel tests: the documented command run in a subprocess against http://localhost:<sentinel>/v1, holding stdout and stderr to that one line, sentinel-free and traceback-free; and an exception class whose name carries the sentinel and a newline, raised from a cause carrying it too, holding the output to the fixed phrase and leaving no exception live after main returns. Each was watched failing with the except arm removed, the message reported, the raw class name reported, and a re-raise from the arm (.logs/m5-mutations.log). Attribution: anthropic/claude-opus-5-5, thinking medium
rafacm
added a commit
that referenced
this pull request
Oct 9, 2026
PR #638's review, finding 4: docs/run/llm.md described the lookup measurements as taken "on an ESP32-S3-Touch-LCD-1.54" and called 1.8 s the time to the first spoken word. The harness sends vinga's prompt with that board's facts in it and reads the model's text back, with no board, no speech recognition or synthesis and no device output, and it times the first non-blank text. Both pages now say so: llm.md describes the harness and calls 1.8 s the time to first text, and the implementation doc's figure is labelled the same way. An untruncated `git grep` of docs/, changelog.d/ and README.md for "spoken word" and "on an ESP32-S3-Touch-LCD-1.54" found no other M5 claim of the kind; the plan's Risks line saying the gate "records time to first spoken word" is the plan's intention from before M5 and is left as written. Attribution: anthropic/claude-opus-5-5, thinking medium
rafacm
added a commit
that referenced
this pull request
Oct 9, 2026
PR #638's review, finding 3: docs/run/llm.md said a turn that searches on the Pi "is given up as a ProviderCallTimeout". With the defaults the 10 s first-token watchdog fires long before the 30 s transport read timeout: it cancels the round, sends it once more, and when the second attempt stalls too gives the turn up with a FirstTokenTimeout and the agent's fallback phrase, as M7's cold-turn paragraph above it says. A ProviderCallTimeout is reached only with the watchdog raised past 30 s. The harness waits for as long as the model takes, so its 34 to 53 s first bytes do not observe the retry; the page now says the running server's outcome is inferred from them, not observed. The same correction is made in the implementation doc, and where M5's other pages stated the outcome (the baseline item in product-promises.md, the changelog fragment, the plan's Gate paragraph) they now say it was measured in a model harness. The implementation doc's line about "the one hallucination on each side" is also reworded: there was one, in Gemma's rephrased run. Attribution: anthropic/claude-opus-5-5, thinking medium
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M5 of #612 (plan, implementation doc section), the last milestone. vinga, the built-in agent, can look things up in the Use pages it ships with: a
search_docs(query)builtin, offered to the built-in agent alone, on every stack, returning the top three passages. M5's own gate missed its bar on the local default model, and the lookup ships everywhere by Rafael's decision of 2026-10-09 (#612 comment), recorded in the plan's Gate section as a decision, not a pass.The gate
Same 32 questions (frozen set, sha
1442b569...) and a rephrased set written and hashed before any retrieval work, same harness, prompt and tools (lookups answered through the shippedsearch_docsbuiltin since the review round), every answer hand-checked (overrides noted in the hand-check files):search_docstwice (the harness re-added a toolBuiltinToolsalready offers; Ollama accepted it, Anthropic refused it). The harness now offers each tool once and checks it; both Gemma sets were re-run, and the table holds the clean runs (the duplicate runs scored 47% and 41%).AnthropicLlm, unchanged, which sends none, andclaude-sonnet-5refuses the parameter. Tokens: about 268,000 input and 4,000 to 5,000 output per set, none cached (about 8,400 input tokens per question).What changes
knowledge/lookup.py:search(query, guide)cuts the packaged Use pages at##/###into passages of at most 1,000 characters, ranks them with BM25 over word stems with the session's board guide weighed up 1.5x, and returns the top three within a 3,300-character budget.knowledge.board_guide()maps a reported board type to a packaged path, never the reported string.search_docsbuiltin (tools/builtin.py,tools/source.py, the names registry): offered to the built-in agent only; an operator agent's call, mangled spellings included, is answered "there is no tool called". Namedsearch_docs, not the gate's baresearch, because builtin names are reserved as MCP entry names.VAGUE_BOARD_FACTSnow sends device questions to the lookup over the common device guide and still never guesses the board.FillerRunner.hold()plays vinga's filler while a lookup runs (silent when vinga's filler is off). The round after a lookup is watched like any other round (the first-token watchdog, one retry); the lookup-round allowance M5 first added is removed by Rafael's decision, because the fixed 30 s transport read timeout caps it.vinga-server/tests/local/lookup_gate/(frozen and rephrased sets, the 39 recorded queries, amendments with two re-anchored key facts, the scorer with anundecidedcategory never credited, the harness) and an opt-in local test.docs/concepts.mdAgent, the glossary's vinga entry, the device guide's "What vinga knows" section (packaged copy regenerated),docs/run/configuration.md, a new "The built-in agent's lookup" section indocs/run/llm.mdafter M7's paragraphs, the local-baseline item indocs/architecture/product-promises.mdwritten from what Gemma 4 e4b measurably does, and the plan's Gate section.changelog.d/612-lookup-tool.md, with anUpgrade:line: anmcp_serversentry namedsearch_docsis now refused.Discoveries
Verification
uv run ruff check .: all checks passed.-n auto --dist loadfile:8675 passed, 19 skipped in 1067.74s.-n auto --dist loadfile:365 passed in 225.48s.checked 337 files, 0 failures; Run/Use pages0 findings; fragments0 failures.tests/censusrun last:69 passed.search_docscalls found none production would refuse, so the gate's numbers stand. Resolutions in the PR comments.tests/local/test_lookup_gate.pyas a pytest case: the harness it calls produced every number, run directly.Attribution: plan and PR text by anthropic/claude-opus-5-5, thinking high; commits by anthropic/claude-opus-5-5, thinking medium; review round 1 by openai/gpt-6.1-sol, thinking high.
🤖 Generated with Claude Code