bench: measure the path a request actually takes - #195
Merged
Merged
Conversation
`compile.bench.ts` times the whole pipeline, which is what `.toSQL()` runs on
every call. That is the right measurement for a query whose shape genuinely
varies per request, and the wrong one for the shape a request usually has,
where the query was written once and only the values change. Nothing measured
that at all — the benchmark had no opinion about the architecture the last
three PRs put in.
scenario toSQL() toCompiled()
select-all 1438ns 51ns 28x
select-where-eq 3291ns 58ns 57x
select-where-deep-and 10614ns 148ns 72x
insert-values 3356ns 63ns 53x
update-where 3481ns 59ns 59x
Two things the table does not show, and the README says both: a compiled
query's SQL text is fixed, so the server keeps its plan — 243µs against 303µs
with PREPARE / EXECUTE — and end to end a single query costs ~330µs either
way, because the compile this removes is ~1% of it.
While measuring: `PERF_GUARD=1` failed on 9, then 13, then 12 of 48 scenarios,
with a different set each run. The machine was at load average 14 from
unrelated work, and every failure was 0–5% under its floor. At load 2 the
guard passes 48/48, and `select-where-deep-and` measures 1.25–1.33x behind
kysely across three rounds, against the 1.46x in the README. No regression —
but a throughput guard on a shared machine reports one, so it is worth
knowing that is what the failures look like.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAwiBVhLmNhWjC9Ro6UMpb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The benchmark had no opinion about the architecture the last three PRs put in.
compile.bench.tstimes the whole pipeline — what.toSQL()runs on every call — which is the right measurement for a query whose shape genuinely varies per request, and the wrong one for the shape a request usually has.toSQL()toCompiled()select-allselect-where-eqselect-where-deep-andinsert-valuesupdate-whereThe README states the two things the table does not show: a compiled query's SQL text is fixed, so the server keeps its plan (243µs against 303µs with
PREPARE/EXECUTE), and end to end a single query costs ~330µs either way because the compile this removes is ~1% of it.On the perf guard
Worth recording, because it looked like a regression and was not.
PERF_GUARD=1failed on 9, then 13, then 12 of 48 scenarios, with a different set each time. The machine was at load average 14 from unrelated work (chrome at 177% CPU), and every failure was 0–5% under its floor —join-2-tablesat 99,834 hz against a 100,000 floor.At load 2 the guard passes 48/48. Measured directly across three rounds,
select-where-deep-andis 1.25–1.33× behind kysely, against the 1.46× recorded in the README — slightly better, not worse.So: no regression. But a throughput guard on a shared machine reports one, and it is worth knowing that is what the failures look like — marginal, and a different set each run.
pnpm testgreen: 3315 tests.