Select Deferred Tools and Gate Memory With a Classifier

[aligulzar729]

Summary

What this changes: two things, both off by default. Before a turn, the deferred tools most likely to be needed have their schemas shipped with the first model call instead of costing a tool_search round trip. After a turn, one yes/no question asks whether the user's newest message asks to remember, update or forget something, and the memory model only runs when it does.

An agent holding a large MCP catalog pays for every tool's argument schema on every turn. Deferring those schemas fixes the cost, but the model then spends a tool_search round trip discovering the tool it needs, on the turn it needs it.

This ranks the deferred tools against the request before the model runs and surfaces the few it is likely to need, so their schemas arrive with the first call. It also puts one yes/no question in front of the memory model, which otherwise runs after every turn even though the default memory instructions only store what the user asks to keep.

Measured on my own instance, with a stand-in model endpoint that reports the tool block it was handed rather than answering:

Prompt	Everything loaded	Selected	Cut
competitor ranked keywords for a domain	202 tools, 237,464 tokens	4 tools, 3,185 tokens	98.7%
"thanks, that was really helpful"	202 tools, 237,464 tokens	1 tool, 286 tokens	99.9%

And for memory: across 120 conversations, 713 user turns produced 2 stored memories holding 101 tokens between them. With memory on, the memory model runs once per completed turn, so that is 713 model calls to produce two durable facts.

Selection only ever adds. A tool it passes over stays listed by name in the tool_search description and remains one search away, so a wrong prediction costs a round trip rather than an answer.

_{Depends on #16180. The diff below includes its commits: GitHub cannot show a delta between two cross-fork branches. The first commit and fix: bound each judgment by one deadline... belong to #16180; the rest is this pull request. The memory gate also needs #16276, which keeps the newest message in the memory window after a turn with tool calls.}

How it works

resolve tools               existing deferred-tool machinery
attach tool_search          unchanged
rank deferred tools    <--  one request: a choice over candidates,
                            plus a yes/no asking whether any tool is needed
createRun                   predictedToolNames joins the surfaced set
attach schemas              existing machinery
...turn runs...
memory gate            <--  one request: asks for a memory change? which key? an update?

Predictions take their own parameter rather than reusing discoveredToolNames, which is persisted into a paused job and replayed on resume: a guess must not be recorded as a real discovery. The step resolves ahead of the checkpoint setup, because an await between the checkpoint prune and createRun would serialize two things that are deliberately overlapped.

The memory gate asks its extra questions in the request it already makes, so categorize and detectUpdates cost no additional round trip. The result reaches the memory model as a suggestion it may ignore, not as an override:

<memory_hint>
This request most likely belongs under `work_context`. It looks
like a change to what is already stored there, not a new fact.
Ignore this if it does not fit what the user actually said.
</memory_hint>

Each capability can set its own timeoutMs, because one ceiling does not fit both: a ranking request over 202 tools is about 9,600 input tokens and takes roughly 4s, while the gate's yes/no answers in under a second.

Both questions ship with default wording an operator can replace from librechat.yaml.

The default question matches the default memory instructions, which only store what the user asks to keep ("Remember...", "Forget..."). Asking instead whether a turn holds something durable let through preferences the memory model then refused, and it scored "Forget where I live" at 0.21, below the threshold, so an explicit forget request could be skipped. Measured against the classifier, requests score 0.88 to 0.98 ("keep in mind that I am vegetarian", "call me Ali from now on", "yes, remember that", "forget where I live", one in Japanese) and everything else 0.39 or lower, so the default threshold is 0.5. A deployment whose memory.instructions keep other facts replaces the question in librechat.yaml.

The gate judges only the user's newest message, sent as latest, with the few messages before it as conversation for context. "yes, remember that" is read against the offer it answers, and a request is not judged again on the next turn.

The classification block is parsed when the app config loads (loadClassificationConfig). The loaded yaml is otherwise the raw object, so none of the schema defaults existed at runtime: tool selection ranked an empty batch because maxCatalogTools was undefined, and categoryThreshold was never met.

Every judgment is billed. Tool selection and the gate report their usage through an onUsage callback, and the client records it with context: 'classification', priced from the shared table the way activity labels are, without touching the response's stream usage. A jev row prices TypeSafe's model at $0.042 per million input tokens with free output; a host whose model has no row falls back to the default rate like any other unknown model.

The prediction starts where it did but is awaited after the memory run starts, so the two overlap. createRun still starts in the same tick as the checkpoint prune.

Name matching checks every occurrence of a tool name, not only the first, and stops at shortlist so a common word cannot surface a tool from every server. Tools a tool_search already loaded earlier in the conversation are left out of the ranking.

Type of change

• Feature

Testing

How the numbers were produced: a stand-in model endpoint that replies with the tool block it was handed rather than answering, so the count is what the provider would have received. The agent held 202 tools from 8 servers, all deferred, shortlist: 5. The baseline row is the same agent with every tool pinned open. Tool definitions were measured as JSON.stringify(tools).length / 3.6. The memory numbers come from a read-only script over GET /api/convos and GET /api/messages/{id}.

predict.spec.ts and gate.spec.ts: batching at the option ceiling, the needs-a-tool gate both ways, widening at the confidence boundary and when confidence is unmeasured, name matching that will not match inside a longer word, a named tool surviving both a conversational gate and an outright failure, categorization skipped without validKeys, a low-confidence key dropped rather than suggested, the update phrasing, and three gate failure modes that all resolve to processing the turn.

Also: the gate inside createMemoryProcessor with the real one-message buffer (the newest message is judged, and a skip never starts the memory model), the newest message sent as latest with earlier turns only as context, a window with no user message skipped without a request, schema defaults filled by loadClassificationConfig (including the 0.5 threshold), and name matching past a partial match and at the cap.

Also: the gate and tool selection report usage (and report nothing when a judgment fails), the client bills it as a classification transaction without replacing the stream usage, the jev row prices every name the model is served under, and the memory run starts before the prediction resolves (this test fails with the prediction awaited first).

cd packages/api && npx jest src/memory/gate src/tools/predict src/agents/memory src/classification passes 154, cd packages/data-schemas && npx jest src/app/service.spec.ts src/methods/tx.spec.ts passes, and cd api && npx jest server/controllers/agents/client.test.js passes 258.

Live, a six-turn conversation wrote 12 classification transactions as jev-latest at a rate of 0.042 with free completion: 1,140 input tokens cost 47.88 credits. npx tsc --noEmit clean for all four workspace projects, plus openapi:check, openapi:test and circular-deps.

The gate question against the live classifier:

Newest message	Score
"Please remember that I prefer answers in Japanese"	0.93
"Forget where I live"	0.93
"Actually I moved to Osaka, update my location"	0.98
"yes, remember that", after the assistant offered to remember the user's editor	0.97
"I prefer answers in Japanese from now on"	0.39
"my name is Ali"	0.05
"I work as a product designer at YankoDesign, can you help me draft an email?"	0.04
"thanks, that was helpful"	0.01

Ranking a 61-option subset of the same catalog returned dataforseo_labs_google_ranked_keywords at 0.39 with dataforseo_labs_google_competitors_domain at 0.38, and a confidence of 0.37. That is what lowConfidenceBelow is for: two genuinely close candidates produce a low confidence, and widening the shortlist by lowConfidenceExtra surfaces both rather than betting on a 0.01 margin. Surfacing the wrong one alone costs a tool_search round trip; surfacing both costs one extra schema.

A conversational turn on the 202-tool catalog logged needs_tool 0.09 below 0.15: surfacing 0 named tool(s) only.

An A/B run on a 25-tool Playwright agent, averaged over 3 runs of the same six prompts (open a page, find links, "thanks", a stated preference, "please remember" that preference, a screenshot), with everything off against deferSchemaChars: 300, tool selection and the memory gate on, and endpoints.all.maxToolResultChars: 8000:

	Off	On	Change
Prompt tokens per model call	17,183	5,243	-69%
Prompt tokens per conversation	442,108	123,553	-72%
Memory model runs	6	1	-83%
Memories saved	3 of 3 runs	3 of 3 runs
tool_search round trips	0	0
Classifier input tokens	0	4,542

The ranges do not overlap: the most expensive run with everything on used 234,049 prompt tokens, the cheapest with everything off 324,746. Tool selection surfaced browser_take_screenshot for the screenshot turn, so the model called it with no search. The gate ran the memory model only on "Please remember..." (0.94 each run, with preferences suggested) and the memory was saved every time.

Pointed at an unreachable endpoint, the turn completed normally with a warning logged and no user-visible error. Under a real timeout the gate logged judgment failed, processing memory as usual and the memory was still written, which is the intended direction.

Risk / compatibility

The gate threshold has a thin margin on one side: in the A/B runs the stated preference without a request scored 0.45 to 0.47 against the 0.5 threshold, while requests scored 0.94. A missed request costs a memory, and a false pass costs one memory model call that the default instructions then decline, so an operator who prefers to err toward running can lower threshold.

Both capabilities default to false; off, the turn is byte-identical to dev. Categorization is skipped when memory.validKeys is not configured, since there would be nothing to choose between. No database change.

Latency: createRun needs the names to build the model binding, so the step is serial and lands at the start of the turn, around 740ms on a 202-tool catalog.

Prompt caching: tool definitions sit at the front of a request, so a changing tool set invalidates a cached prefix. The saving is unambiguous on uncached turns; operators running a warm cache should measure first.

Checklist

• I reviewed my own changes
• Relevant tests have been added or updated
• Existing relevant tests pass
• The change does not introduce new warnings or errors
• User-facing or complex behavior is documented where necessary
• Required dependency changes have been merged/published
• Required documentation PR: docs: Document Classification, deferSchemaChars and the Tool Result Cap docs#778