Skip to content

perf(tools): slim tool definitions by a third - #83

Merged
lcottercertinia merged 6 commits into
mainfrom
perf/slim-tool-definitions
Aug 3, 2026
Merged

perf(tools): slim tool definitions by a third#83
lcottercertinia merged 6 commits into
mainfrom
perf/slim-tool-definitions

Conversation

@lukecotter

@lukecotter lukecotter commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Closes #87. #81 is merged, so this now sits directly on main: six commits.

Tool definitions load into context on every request, whether a tool is called or not. That is the standing cost of having the server connected, and nothing in the repository measured it. This cuts it from ~1,529 tokens in 1.x to ~1,053 (-31%), with no tool renamed, no parameter removed and no response changed — and then budgets it, so the saving cannot be given back unnoticed.

Every figure is the whole definition as the client receives it: name, title, description, inputSchema, annotations and the SDK's own fields, serialised from a live tools/list over stdio.

Tool 1.x main today this PR
analyze_apex_log_performance ~247 ~290 ~238
get_apex_log_summary ~171 ~212 ~153
find_performance_bottlenecks ~267 ~267 ~234
execute_anonymous ~844 ~844 ~428
Total ~1,529 ~1,613 ~1,053

The middle column is why the table is worth showing: #81 spent ~84 tokens on a per-tool sentence saying what each response omits. This PR removes those and reaches 31% under 1.x.

What changed

  • debugLevels.ts owns LOG_LEVELS and TRACE_CATEGORIES as as const tuples, so the schema builds from them instead of restating them. toSObjectFields capitalises the first letter rather than listing ten if branches.
  • execute_anonymous uses z.partialRecord(z.enum(TRACE_CATEGORIES), z.enum(LOG_LEVELS)) — one describe instead of ten near-identical per-category ones. The wire form of debugLevel is unchanged, and an unknown category key is now rejected. This one change is most of the saving: ~844 to ~428.
  • annotations.title removed from all four tools; the top-level title is the spec's display field and the annotation copy was sent twice.
  • destructiveHint and idempotentHint removed from the three read-only tools. The specification defines both as meaningful only when readOnlyHint is false, so they cost tokens on every request and tell a client nothing. execute_anonymous keeps all four; it is the one tool where a client that misreads a default runs Apex against an org.
  • Describe text the agent cannot act on is gone: response-shaping policy (a contributor rule, now in CLAUDE.md and DEVELOPING.md) and "durations in ms" (said once in the server instructions). The guarantee that a reported zero is a measured zero moves the same way — out of four descriptions sent on every request, into instructions, sent once per session.

The gate

pnpm run eval now makes three assertions about the definitions, once per run:

  • A per-tool budget, with about 5% headroom, so one careless sentence fails the build.
  • A total under the 1.x baseline, measured at b79328f through this same stdio path and estimator and held as a static constant, because a released figure cannot change. The total also catches a fifth tool, which no per-tool budget can.
  • Selection keywords, one or two per tool. A description is a selection prompt, and clients match the words in it, so a trim that saves tokens can cost discovery. This makes that trade fail loudly instead of passing quietly.

The budget covers the whole wire object on purpose. A budget over {name, description, inputSchema} would leave title, annotations and execution — about 215 tokens — unwatched. Three unit tests pin the annotations, which a budget with headroom would not notice coming back.

The definitions table in the README is generated from the run, beside the response table #81 added, so a change that moves a published figure fails until pnpm run eval:update regenerates it.

Not done, deliberately

  • The three read-only tools are not merged into one analyze_apex_log(type). It saves ~390 tokens once but costs a breaking change, ~98 tests, every eval golden, and the VS Code extension that registers this server by tool name — and a flattened schema stops saying which parameter belongs to which mode. The 1.x baseline is beaten without it.
  • About 90 tokens are an SDK floor, recorded rather than fixed: $schema (~52 across the four tools) comes from zod's toJSONSchema, and execution (~40) is added by McpServer itself. Neither has a public hook, and reaching into SDK internals is not worth 90 tokens.

Verification

  • pnpm run build, pnpm run lint clean; 232 tests, 12 suites pass
  • pnpm run eval 6/6 with "checked tool definitions — ~1053 tokens across 4 tools", and all six goldens unchanged, proving no response moved
  • An independent stdio probe, written before the harness and run against the same build, gives the same 238 / 153 / 234 / 428 the README publishes
  • Injected regressions, each reverted: a restored idempotentHint fails its unit test; one extra sentence in a description fails with "definition is ~183 tokens, over its budget of 161"; a hand-edited README figure fails the README check
  • Live smoke test against a sandbox with System.debug(''); only, once with debugLevel: "FINEST" and once with { apexCode: "FINEST", database: "NONE" } — both set the trace flag and returned a log

Follow-up

Two defects found while measuring are fixed in #85, stacked on this PR: totalMethods disagreed between tools (11 vs 13), and the log was parsed three times by the identical copy-pasted preamble in the three analysis tools.

@lcottercertinia
lcottercertinia force-pushed the feat/lean-tool-responses branch 2 times, most recently from e63d5f3 to 22cbd6f Compare August 3, 2026 09:20
@lukecotter
lukecotter force-pushed the perf/slim-tool-definitions branch from 7c774b2 to 37006f6 Compare August 3, 2026 13:39
Base automatically changed from feat/lean-tool-responses to main August 3, 2026 13:42
Tool definitions load into context on every turn, whether a tool is
called or not. Cut tools/list from ~1,612 to ~1,087 tokens with no
tool renamed, no parameter removed and no response changed.

- debugLevels.ts owns LOG_LEVELS and TRACE_CATEGORIES as const tuples,
  so the schema builds from them instead of restating them
- executeAnonymous uses z.partialRecord over those tuples, replacing
  ten per-category describes with one (~844 -> ~428 tokens)
- drop the annotations.title duplicate from all four tools; the
  top-level title is the spec field
- drop describe text the agent cannot act on: response-shaping policy
  and durations-in-ms, both stated elsewhere
- server instructions trimmed to the two server-wide facts
The tool description hand-wrote the per-category defaults, so a change
to DEFAULT_TRACE_CONFIG would leave the wire text stale with nothing to
catch it. Build the clause from the config instead, grouped by level so
the description stays the same length.

Also drop the throwaway local in isLogLevel.
@lcottercertinia
lcottercertinia force-pushed the perf/slim-tool-definitions branch from 37006f6 to c40b792 Compare August 3, 2026 13:42
lcottercertinia
lcottercertinia previously approved these changes Aug 3, 2026
destructiveHint and idempotentHint are defined as meaningful only when
readOnlyHint is false, so on the three read-only tools they cost tokens on
every request and tell a client nothing. Pin what is left with one assertion
per tool, because the definition budget has headroom and would not notice a
hint quietly coming back.
Fixed fields are always reported, so a zero is a measured zero. That holds
for every tool, and instructions are sent once per session where a tool
description is sent on every request.
The definitions are sent on every request whether a tool is called or not,
and nothing measured them, so a careless sentence could give the saving back.
Budget the whole wire object as the client receives it: a budget over a subset
of the fields leaves title, annotations and execution unwatched. Assert a total
under the 1.x baseline, measured at b79328f over this same stdio path, which
also catches a fifth tool that no per-tool budget can. Assert the keywords
clients select on, so a trim cannot cost discovery.

Publish the figures from the run, beside the response table.
@lcottercertinia
lcottercertinia merged commit 8d0f7fa into main Aug 3, 2026
13 checks passed
@lcottercertinia
lcottercertinia deleted the perf/slim-tool-definitions branch August 3, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: tool definitions cost more tokens than the facts require

2 participants