Repository navigation
REPL slice 2: line editing, history, and multi-line entry via reedline - #998
Merged
thekevinscott merged 9 commits intoAug 21, 2026
Merged
thekevinscott merged 9 commits into
thekevinscott merged 9 commits into
Conversation
The interactive path only exists when stdin is a terminal, so these drive the real binary under a real PTY (util-linux `script`) and assert on the terminal's final contents rather than on keystroke timing. Four fail today, for the reason #988 describes: a statement currently ends at the first newline, so `SELECT` / `1 AS n;` typed over two lines is two statements and two errors. Two pass as regression guards -- `exit` must still leave without a terminator, and continue-on-error must survive multi-line entry. Pushed on its own so CI is observed red before any implementation lands. Refs #988 skip-changelog: test-only commit; the fragment lands with the implementation Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
The interactive prompt was a plain line reader: no recall, no editing, one line per statement, and Ctrl+C killed the process. It now runs through reedline, which brings all four at once -- history with the up arrow and Ctrl+R, emacs editing, Ctrl+C that abandons a line instead of the session, and statements that span lines. Three decisions #988 left open, settled: - **reedline over rustyline.** rustyline covers the same four requirements natively and pulls 16 transitive crates to reedline's 52, so the issue's "needs more hand-rolling" premise did not hold; the maintainer's call went to reedline anyway, which is what #988 assumed. - **A statement ends at its semicolon**, as in `sqlite3`, with SQLite's own tokenizer (`sqlite3_complete`) deciding where that is. Hand-rolling the scan is how `SELECT ';'` gets read as two statements. `exit`, `quit` and blank input are exempt: none is SQL, and waiting for a terminator on the one input meant to end the session would trap the user. - **History is global**, at `$XDG_DATA_HOME/dirsql/history` (falling back to `~/.local/share`, then `%APPDATA%`), mirroring `sqlite3`'s single `~/.sqlite_history`: a query worked out in one project is worth recalling in the next. Resolution is a pure function over the environment, so it is unit-tested without touching the filesystem, and a session with nowhere to write still edits and recalls within itself. The pipe path is untouched, as #988 requires: no editor, no history, and still one statement per line with no terminator. A redirected script is not being typed, so there is no continuation prompt for a multi-line rule to hang off. The split is now a `Lines` trait with two implementations rather than a boolean, which also gives the loop a seam to be unit-tested through without a terminal. Testing splits along what each harness can actually see. The byte stream a pipe gives back is not a screen -- the editor repaints by moving the cursor -- so `repl_tty_e2e.rs` (a real pty, answering the editor's cursor-position query itself) asserts only what was ever drawn: multi-line entry, the continuation prompt, the literal-semicolon case. Anything turning on a keystroke landing exactly when the editor is listening was measured flaky at roughly one miss in four there, and lives in the new curtaincall suite, which drives a real VT100 and waits on the screen: history recall, history across sessions, the XDG location, Ctrl+C. curtaincall needs one workaround worth knowing about: pyte formats the cursor-position reply in `report_device_status` but its `write_process_input` hook is an inert stub, so the reply never reaches the process and every session dies with "the cursor position could not be read". Pointing that hook at the pty is the whole fix. Fixes #988 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
The editor reports Ctrl-D distinctly from every other signal, and nothing in the Rust suite covered that arm: deleting it lets Ctrl-D fall into the catch-all that abandons a line, turning the standard way out of a shell into a keystroke that clears the line and does nothing else. Found by running the mutation gate against this slice's diff; verified by hand that the case fails with the arm removed and passes with it. Refs #988 skip-changelog: test-only; the behavior's fragment is in this branch already Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
…y `...>`
The hand-rolled `Prompt` impl duplicated reedline's rendering, including a
history-search line that was byte-for-byte what `DefaultPrompt` already
produces. Wrapping it instead leaves one method ours -- the continuation
marker, where `sqlite3`'s `...>` beats reedline's `::: ` for people who
came from a SQL shell.
The duplication had a concrete cost the mutation gate found: an empty right
prompt written as a constant has no killable mutant, because
`Cow::Borrowed("")` *is* what the function returned. Delegating makes the
emptiness a property of the segment the prompt was built with rather than
of the code, so a test that builds one with non-empty segments pins the
forwarding and kills it. Verified by hand: hardcoding the return fails
`the_prompt_forwards_the_segments_it_was_built_from`, and restoring the
delegation passes it.
Refs #988
skip-changelog: no observable change -- same prompt, same indicators
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
The harness started typing once the banner appeared, but the banner is written before the editor starts reading -- so the first keystroke could land in the gap and be read by nothing. Waiting for the output to stop instead is the real signal: the last thing drawn is the prompt. `ctrl_d_leaves_the_session` had the same problem one keystroke later. It sent Ctrl-D at the prompt that follows an executed statement, which is the gap that made history recall flaky enough to move to curtaincall. Sending it at a fresh prompt tests the same thing -- with the `Signal::CtrlD` arm removed the session still fails to exit -- without racing anything. Caught by running the whole integration suite in parallel, where the contention made the gap wide enough to lose the key; three consecutive full runs are clean now. Refs #988 skip-changelog: test-only Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
8 tasks done
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #988
Slice 2 of 3 under #953. Stacked on #991 — this PR's base is slice 1's branch, so the diff shown is slice 2's own. Stack: #991 ← this PR ← #989.
The interactive prompt was a plain line reader: no recall, no editing, one line per statement, and Ctrl+C killed the process. It now runs through reedline, which brings all four at once — ↑/↓ and Ctrl+R history, emacs editing, Ctrl+C that abandons a line rather than the session, and statements that span lines.
The pipe path is untouched, as #988 requires: no editor, no history, and still one statement per line with no terminator. A redirected script is not being typed, so there is no continuation prompt for a multi-line rule to hang off.
repl_pipe_e2e.rspins that and is unchanged.The three open decisions, settled
Editor: reedline. Measured against rustyline first, since #988 asked to confirm: rustyline covers all four requirements natively —
ReadlineError::{Interrupted, Eof},FileHistoryand Ctrl+R in default features, aValidatortrait withValidationResult::Incomplete— and pulls 16 transitive crates to reedline's 52 (crossterm, chrono, serde + syn proc-macros, strum, parking_lot, signal-hook). So the issue's "rustyline needs more hand-rolling" premise did not hold. Maintainer's call went to reedline, the original assumption. It is optional and behind theclifeature, so a library consumer still pulls none of it.A typed statement ends at its semicolon, exactly as in
sqlite3— and SQLite's own tokenizer decides where that semicolon is, viasqlite3_complete()throughrusqlite::ffi. Hand-rolling the scan is howSELECT ';'gets read as two statements; a comment or aBEGIN … ENDbody is the same trap.exit,quit, and blank input are exempt from the rule: none is SQL, and waiting for a terminator on the one input meant to end the session would trap the user.This is the "explicit terminator" option #988 floated, and it does change slice 1's interactive behavior:
SELECT 1⏎ now needsSELECT 1;. That is the cost of multi-line working for every statement rather than only for unclosed literals, and it is what asqlite3user already expects. A migration fragment records it.History is global:
$XDG_DATA_HOME/dirsql/history, falling back to~/.local/share/dirsql/history(%APPDATA%\dirsql\historyon Windows), last 1000 statements, in-memory if none resolve. One file for every directory mirrorssqlite3's single~/.sqlite_history— a query worked out in one project is worth recalling in the next. Resolution is a pure function over an injected environment lookup, so it unit-tests without touching the filesystem.Shape
Slice 1's
interactive: boolreader became aLinestrait with two implementations —PipedLines(the plain reader) andEditorLines(reedline) — yieldingEntry::{Statement, Cancelled, Ended}.Cancelledis Ctrl+C, which is why the interrupt needs no signal handler. The trait is also the unit-test seam:FakeLinesdrives the whole loop, Ctrl+C included, with no terminal in sight.sqlite3_completeis FFI, and the crate setsunsafe_code = "deny", so the one call carries an#[expect(unsafe_code, reason = …)]— the explicit opt-in that lint exists to require.Testing: what a pty can and cannot tell you
This took the most work and is the part worth reading.
A pty is not a terminal. reedline asks the terminal where the cursor is (an
ESC [ 6 nDevice Status Report) and dies with "the cursor position could not be read within a normal duration" when nothing answers.scripthands you a pty but no emulator behind it, so the harness has to answer that query itself — oneESC [ 1;1 Rper query is the whole of the emulation needed. curtaincall needs the same fix for a different reason: pyte formats the reply inreport_device_status, but itswrite_process_inputhook is an inert stub, so it never reaches the process. Pointing that hook at the pty fixes it.You cannot reconstruct a screen by stripping escapes — the editor repaints by moving the cursor, so a byte stream is not a screen and cannot be turned into one. That decides the split:
packages/rust/tests/repl_tty_e2e.rs(7 tests, runs in CI) drives a real pty and asserts only what was ever drawn: multi-line entry, the continuation prompt, the semicolon-in-a-literal case, the lone terminator, exit words, Ctrl-D, failure-continues. Verified stable over 16 consecutive single-threaded runs, and over three full parallel runs of the whole integration suite.packages/python/tests/e2e/repl_interactive_test.py(5 tests, curtaincall, e2e tier) covers everything that turns on a keystroke landing exactly when the editor is listening — up-arrow recall, history surviving the process, the XDG location on disk, Ctrl+C. Those cases were measured flaky at roughly one miss in four in the pty harness, which is precisely the gap curtaincall's screen-aware waiting closes. This is the dependency slice 1 deferred to here.Everything decidable without a terminal is unit-tested and runs in CI:
sql_is_complete,is_submittable,SqlValidator,history_path, the prompt rendering, and the loop itself throughFakeLines.One practical note for the python suite: curtaincall declares
requires-python >=3.12against this package's>=3.10floor, so it carries a marker plus animportorskipguard — and the local venv must be recreated on 3.12 (uv venv --python 3.12), or the suite silently skips instead of running. Upstream: curtaincall#26.PARITY
No drift. The CLI is one shared implementation in the Rust core that every SDK's launcher reaches through
run_cli; no SDK public API changed.PARITY.mdneeds no edit — stated here rather than skipped silently, per AGENTS.md.Manual run (AGENTS.md)
Driven through a real pty with a cursor-query responder, against a directory holding
a.md(6 B) andb.md(10 B), withXDG_DATA_HOME=/tmp/manualhist:And the history file it left behind —
<\n>is reedline's own encoding for a statement that spanned lines — showing it stored as one entry, with the abandoned line absent:E2E Verification
cargo test --features cli --tests, including the newrepl_tty_e2e(7) and the unchangedrepl_pipe_e2e(9)packages/pythone2e attestation refreshed (apackages/rust/srcchange stales both bindings)packages/tse2e attestation refreshedtests/vtab.rs::unreadable_file_yields_null_content_without_erroring_the_row— this sandbox runs as uid 0, so achmod 000file is still readable and the NULL-content assertion cannot hold. Green in CI, unrelated to this change.Changelog / Migrations
packages/rust/changelog.d/2026-08-20-repl-line-editor.md, pluspackages/python/changelog.d/2026-08-20-curtaincall-dev-dependency.md—pyproject.tomlis not exempt fromchangelog-gate, so the dev-group addition needs its own fragment even though nothing about the published package changespackages/rust/migrations.d/2026-08-20-repl-line-editor.md— required: a typed statement now needs its terminator, and Ctrl+C no longer ends the processRed/green
aab9c74pushed the failing pty suite alone, and CI was observed red on Integration tests alone, for the right reason: a statement typed over two lines was executed as two, givingincomplete inputand then a syntax error. The implementation follows incb75f4e.Worth recording honestly: that RED commit also contained three cases the implementation could not turn green, because reedline will not run at all under a pty that does not answer its cursor query — the harness had to grow that responder before any of it could pass. Two more were then found flaky rather than failing and moved to curtaincall. The seven that remain are the ones a byte stream can actually answer.
Mutation, and what it caught
Worth calling out, because it changed the code twice. Run against this slice's diff with the full suite (CI's scope), it surfaced three survivors:
Signal::CtrlDarmctrl_d_leaves_the_sessionin the pty suite (75a8009). Verified by hand: fails with the arm removed, passes with it.render_prompt_right→Cow::Borrowed("")"", so no test could ever kill it.f51e578) makes emptiness a property of the segment rather than of the code, so a test that builds one with non-empty segments pins the forwarding. Also deletes a history-search line that duplicated upstream byte for byte.render_prompt_right→Cow::Owned("".to_owned())One
TIMEOUTremains, on deleting the EOF arm of the piped reader: any mutant that stops EOF being EOF makes the loop read an endless empty line, so it hangs rather than failing. The identical construct is already on the base branch from slice 1 and passed this gate there, so timeouts are treated as caught.A last flake worth recording: running the whole integration suite in parallel starved the pty tests enough that a keystroke was lost. The harness had been typing as soon as the banner appeared, but the banner is written before the editor starts reading. Waiting for the output to stop instead (
b04f77d) is the real signal — the last thing drawn is the prompt.