Skip to content

test(python): speed-of-light e2e tests for the article's use cases - #1200

Merged
thekevinscott merged 2 commits into
mainfrom
claude/1198-e2e-use-cases
Oct 2, 2026
Merged

thekevinscott merged 2 commits into
mainfrom
claude/1198-e2e-use-cases

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1198

Five speed-of-light e2e tests, one per article use case, plus the embeddings one in the plugin suite. Each test builds its corpus in a temp dir, grows it by doubling until the fastest native pipeline takes at least a second, runs native and dirsql three times each, and asserts two things: the rows are identical, and dirsql wall time minus measured launcher startup (ten --help runs averaged) is within 1.1x native.

All five rows checks pass. All five timing checks fail today, each for the reason the test asserts, and are marked expected failures. Measured on this branch (32 cores, best of three; dirsql minus startup, native best of three):

test corpus at the 1 s bar native dirsql ratio why
dossier (use_case_dossier_test.py) 400k markdown files 0.66 s (3 x find) 11.2 s 17x three CTEs over ./ walk the tree three times, and one walk is ~4x find at this size (#1201)
notebook (use_case_notebook_test.py) 160k sections, one file 1.18 s (parser + sort/head) 1.85 s 1.6x ingesting the parser's JSON rows is slower than json.load + sort (#1202)
join (use_case_join_test.py) 20k paper folders ~1 s (find x2, awk, sort, join) >60 s (timed out) - unindexed nested-loop join, #1196
transcripts (use_case_transcripts_test.py) 1,024 JSONL files, 100 MB 1.41 s (one parser process) 52.0 s 37x one parser process per file (#1203)
embeddings (plugin speed_of_light_test.py) 32k abstracts 1.31 s (model2vec batch encode) 4.46 s 3.4x one worker round trip per value (#1204)

The grow until native >= 1 s rule sizes every corpus on the machine running it; there is no fixed millisecond number anywhere. A dirsql run more than ten times over budget ends its best-of-three early, since noise cannot account for an order of magnitude, which keeps the suite's red runtime bounded (about six minutes for the five tests here).

Each test carries pytest.mark.xfail(strict=True) with its gap as the reason, so the suite is green and attestation writes a receipt on any branch. Strict mode fails the suite the moment a test clears the bar with the marker still present, so each fix PR removes its marker in the same change. Epic #1205 tracks the five fixes, one PR each.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (38 passed, 4 xfailed in 3 m 56 s); plugin e2e: pass (17 passed, 1 xfailed in 24 s)
  • TypeScript SDK e2e: N/A
  • Rust core e2e (if applicable): N/A
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed: not required, test-only diff
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed: N/A
  • Command(s) run: cd packages/python && uv run python -m pytest tests/e2e/ -q; cd plugins/dirsql-plugin-embeddings && uv run --with-editable ../../packages/python python -m pytest tests/e2e/ -q
  • Result summary: five new tests xfail at the speed assertion with rows identical; existing suites green

Changelog / Migrations

  • Changelog fragment: not required, test-only diff (the gate exempts tests/)
  • Migration fragment: not required

Five e2e tests, one per use case plus the embeddings one in the plugin
suite. Each builds its corpus in a temp dir, grows it by doubling until the
fastest native pipeline takes a second, runs native and dirsql three times
each, and asserts identical rows and dirsql wall time minus measured
launcher startup within 1.1x native.

All five are red today at the timing assertion, rows identical:
dossier 17x (three CTEs walk the tree three times, one walk ~4x find at
400k files), notebook 1.6x (on-file JSON ingestion), join timed out at 60 s
(#1196), transcripts 37x (one parser process per file), embeddings 3.4x
(one worker round trip per value).
@thekevinbot
thekevinbot marked this pull request as ready for review October 2, 2026 00:15
@thekevinscott
thekevinscott merged commit d486ec8 into main Oct 2, 2026
52 checks passed
@thekevinscott
thekevinscott deleted the claude/1198-e2e-use-cases branch October 2, 2026 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

e2e: four use-case tests asserting correctness and wall time within 1.1x the fastest native pipeline

2 participants