Repository navigation
test(python): speed-of-light e2e tests for the article's use cases - #1200
Merged
Merged
Conversation
Five e2e tests, one per use case plus the embeddings one in the plugin suite. Each builds its corpus in a temp dir, grows it by doubling until the fastest native pipeline takes a second, runs native and dirsql three times each, and asserts identical rows and dirsql wall time minus measured launcher startup within 1.1x native. All five are red today at the timing assertion, rows identical: dossier 17x (three CTEs walk the tree three times, one walk ~4x find at 400k files), notebook 1.6x (on-file JSON ingestion), join timed out at 60 s (#1196), transcripts 37x (one parser process per file), embeddings 3.4x (one worker round trip per value).
This was referenced Oct 1, 2026
Closed
…ach gap is closed
5 tasks done
thekevinbot
marked this pull request as ready for review
October 2, 2026 00:15
thekevinscott
enabled auto-merge
October 2, 2026 00:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1198
Five speed-of-light e2e tests, one per article use case, plus the embeddings one in the plugin suite. Each test builds its corpus in a temp dir, grows it by doubling until the fastest native pipeline takes at least a second, runs native and
dirsqlthree times each, and asserts two things: the rows are identical, anddirsqlwall time minus measured launcher startup (ten--helpruns averaged) is within 1.1x native.All five rows checks pass. All five timing checks fail today, each for the reason the test asserts, and are marked expected failures. Measured on this branch (32 cores, best of three; dirsql minus startup, native best of three):
use_case_dossier_test.py)find)./walk the tree three times, and one walk is ~4xfindat this size (#1201)use_case_notebook_test.py)sort/head)json.load+ sort (#1202)use_case_join_test.py)findx2,awk,sort,join)use_case_transcripts_test.py)speed_of_light_test.py)The
grow until native >= 1 srule sizes every corpus on the machine running it; there is no fixed millisecond number anywhere. A dirsql run more than ten times over budget ends its best-of-three early, since noise cannot account for an order of magnitude, which keeps the suite's red runtime bounded (about six minutes for the five tests here).Each test carries
pytest.mark.xfail(strict=True)with its gap as the reason, so the suite is green and attestation writes a receipt on any branch. Strict mode fails the suite the moment a test clears the bar with the marker still present, so each fix PR removes its marker in the same change. Epic #1205 tracks the five fixes, one PR each.E2E Verification
packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged: not required, test-only diffpackages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged: N/Acd packages/python && uv run python -m pytest tests/e2e/ -q;cd plugins/dirsql-plugin-embeddings && uv run --with-editable ../../packages/python python -m pytest tests/e2e/ -qChangelog / Migrations
tests/)