Repository navigation
perf: on-file row ingestion is slower than a second interpreter's json.load + sort #1202
Description
use_case_notebook_test.py (#1200) is red at 1.6x native on one markdown file of ~160k dated sections parsed by a single --on-file script:
| pipeline | wall |
|---|---|
python3 extract.py notebook.md | python3 -c '<json.load, sort, print 5>' |
1.18 s |
dirsql query "SELECT date, words FROM './notebook.md' ORDER BY date DESC LIMIT 5" --on-file 'python3 extract.py {path}' |
1.85 s (startup 0.04 s) |
The parser runs once in both. The extra 0.7 s is dirsql's side: reading the parser's JSON array, building rows, inserting them, sorting. A second Python interpreter does the same load and sort in less time than that. Likely candidates: per-row allocation in the JSON-to-row path, or inserting through SQLite one row per statement instead of batched.
Bar from #1198: dirsql wall minus launcher startup <= 1.1x native.
Done when
packages/python/tests/e2e/use_case_notebook_test.py passes with its pytest.mark.xfail(strict=True) marker removed in the same PR: identical rows, and dirsql wall time minus launcher startup within 1.1x the native pipeline. Part of epic #1205.
Activity
thekevinbot commentedon Oct 2, 2026
Paused 2026-10-02: the on-file contract itself is being redesigned from first principles (see the dirsql-design discussion). No branch or PR was opened. Resume once the new shape is decided; the fix may change or become moot.
thekevinbot commentedon Oct 2, 2026
Unblocked 2026-10-02: the on-file redesign landed as epic #1214 (batched invocation: the command runs once per table with every matched path as a trailing argument and prints one JSON array). This issue is unaffected and stays open; the notebook fixture will drop
{path}and iteratesys.argv[1:]under #1215, so the fix must not depend on the per-file form. Resume after #1215 merges.thekevinbot commentedon Oct 2, 2026
Closed by #1219: rows now reach SQLite through one prepared statement per table and one column lookup per batch. The notebook speed-of-light test passes at 1.02x native and its
xfailis removed.