perf(tools): slim tool definitions by a third - #83
Merged
Conversation
This was referenced Aug 1, 2026
lcottercertinia
force-pushed
the
feat/lean-tool-responses
branch
2 times, most recently
from
August 3, 2026 09:20
e63d5f3 to
22cbd6f
Compare
lukecotter
force-pushed
the
perf/slim-tool-definitions
branch
from
August 3, 2026 13:39
7c774b2 to
37006f6
Compare
Tool definitions load into context on every turn, whether a tool is called or not. Cut tools/list from ~1,612 to ~1,087 tokens with no tool renamed, no parameter removed and no response changed. - debugLevels.ts owns LOG_LEVELS and TRACE_CATEGORIES as const tuples, so the schema builds from them instead of restating them - executeAnonymous uses z.partialRecord over those tuples, replacing ten per-category describes with one (~844 -> ~428 tokens) - drop the annotations.title duplicate from all four tools; the top-level title is the spec field - drop describe text the agent cannot act on: response-shaping policy and durations-in-ms, both stated elsewhere - server instructions trimmed to the two server-wide facts
The tool description hand-wrote the per-category defaults, so a change to DEFAULT_TRACE_CONFIG would leave the wire text stale with nothing to catch it. Build the clause from the config instead, grouped by level so the description stays the same length. Also drop the throwaway local in isLogLevel.
lcottercertinia
force-pushed
the
perf/slim-tool-definitions
branch
from
August 3, 2026 13:42
37006f6 to
c40b792
Compare
lcottercertinia
previously approved these changes
Aug 3, 2026
destructiveHint and idempotentHint are defined as meaningful only when readOnlyHint is false, so on the three read-only tools they cost tokens on every request and tell a client nothing. Pin what is left with one assertion per tool, because the definition budget has headroom and would not notice a hint quietly coming back.
Fixed fields are always reported, so a zero is a measured zero. That holds for every tool, and instructions are sent once per session where a tool description is sent on every request.
The definitions are sent on every request whether a tool is called or not, and nothing measured them, so a careless sentence could give the saving back. Budget the whole wire object as the client receives it: a budget over a subset of the fields leaves title, annotations and execution unwatched. Assert a total under the 1.x baseline, measured at b79328f over this same stdio path, which also catches a fifth tool that no per-tool budget can. Assert the keywords clients select on, so a trim cannot cost discovery. Publish the figures from the run, beside the response table.
lcottercertinia
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #87. #81 is merged, so this now sits directly on
main: six commits.Tool definitions load into context on every request, whether a tool is called or not. That is the standing cost of having the server connected, and nothing in the repository measured it. This cuts it from ~1,529 tokens in 1.x to ~1,053 (-31%), with no tool renamed, no parameter removed and no response changed — and then budgets it, so the saving cannot be given back unnoticed.
Every figure is the whole definition as the client receives it:
name,title,description,inputSchema,annotationsand the SDK's own fields, serialised from a livetools/listover stdio.maintodayanalyze_apex_log_performanceget_apex_log_summaryfind_performance_bottlenecksexecute_anonymousThe middle column is why the table is worth showing: #81 spent ~84 tokens on a per-tool sentence saying what each response omits. This PR removes those and reaches 31% under 1.x.
What changed
debugLevels.tsownsLOG_LEVELSandTRACE_CATEGORIESasas consttuples, so the schema builds from them instead of restating them.toSObjectFieldscapitalises the first letter rather than listing tenifbranches.execute_anonymoususesz.partialRecord(z.enum(TRACE_CATEGORIES), z.enum(LOG_LEVELS))— one describe instead of ten near-identical per-category ones. The wire form ofdebugLevelis unchanged, and an unknown category key is now rejected. This one change is most of the saving: ~844 to ~428.annotations.titleremoved from all four tools; the top-leveltitleis the spec's display field and the annotation copy was sent twice.destructiveHintandidempotentHintremoved from the three read-only tools. The specification defines both as meaningful only whenreadOnlyHintis false, so they cost tokens on every request and tell a client nothing.execute_anonymouskeeps all four; it is the one tool where a client that misreads a default runs Apex against an org.CLAUDE.mdandDEVELOPING.md) and "durations in ms" (said once in the serverinstructions). The guarantee that a reported zero is a measured zero moves the same way — out of four descriptions sent on every request, intoinstructions, sent once per session.The gate
pnpm run evalnow makes three assertions about the definitions, once per run:b79328fthrough this same stdio path and estimator and held as a static constant, because a released figure cannot change. The total also catches a fifth tool, which no per-tool budget can.The budget covers the whole wire object on purpose. A budget over
{name, description, inputSchema}would leavetitle,annotationsandexecution— about 215 tokens — unwatched. Three unit tests pin the annotations, which a budget with headroom would not notice coming back.The definitions table in the README is generated from the run, beside the response table #81 added, so a change that moves a published figure fails until
pnpm run eval:updateregenerates it.Not done, deliberately
analyze_apex_log(type). It saves ~390 tokens once but costs a breaking change, ~98 tests, every eval golden, and the VS Code extension that registers this server by tool name — and a flattened schema stops saying which parameter belongs to which mode. The 1.x baseline is beaten without it.$schema(~52 across the four tools) comes from zod'stoJSONSchema, andexecution(~40) is added byMcpServeritself. Neither has a public hook, and reaching into SDK internals is not worth 90 tokens.Verification
pnpm run build,pnpm run lintclean; 232 tests, 12 suites passpnpm run eval6/6 with "checked tool definitions — ~1053 tokens across 4 tools", and all six goldens unchanged, proving no response movedidempotentHintfails its unit test; one extra sentence in a description fails with "definition is ~183 tokens, over its budget of 161"; a hand-edited README figure fails the README checkSystem.debug('');only, once withdebugLevel: "FINEST"and once with{ apexCode: "FINEST", database: "NONE" }— both set the trace flag and returned a logFollow-up
Two defects found while measuring are fixed in #85, stacked on this PR:
totalMethodsdisagreed between tools (11 vs 13), and the log was parsed three times by the identical copy-pasted preamble in the three analysis tools.