Skip to content

REPL slice 2: line editing, history, and multi-line entry via reedline - #998

Merged
thekevinscott merged 9 commits into
claude/epic-953-stacked-prs-x59wm3from
claude/epic-953-slice2-reedline
Aug 21, 2026
Merged

thekevinscott merged 9 commits into
claude/epic-953-stacked-prs-x59wm3from
claude/epic-953-slice2-reedline

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Aug 20, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #988

Slice 2 of 3 under #953. Stacked on #991 — this PR's base is slice 1's branch, so the diff shown is slice 2's own. Stack: #991 ← this PR ← #989.

The interactive prompt was a plain line reader: no recall, no editing, one line per statement, and Ctrl+C killed the process. It now runs through reedline, which brings all four at once — ↑/↓ and Ctrl+R history, emacs editing, Ctrl+C that abandons a line rather than the session, and statements that span lines.

The pipe path is untouched, as #988 requires: no editor, no history, and still one statement per line with no terminator. A redirected script is not being typed, so there is no continuation prompt for a multi-line rule to hang off. repl_pipe_e2e.rs pins that and is unchanged.

The three open decisions, settled

Editor: reedline. Measured against rustyline first, since #988 asked to confirm: rustyline covers all four requirements natively — ReadlineError::{Interrupted, Eof}, FileHistory and Ctrl+R in default features, a Validator trait with ValidationResult::Incomplete — and pulls 16 transitive crates to reedline's 52 (crossterm, chrono, serde + syn proc-macros, strum, parking_lot, signal-hook). So the issue's "rustyline needs more hand-rolling" premise did not hold. Maintainer's call went to reedline, the original assumption. It is optional and behind the cli feature, so a library consumer still pulls none of it.

A typed statement ends at its semicolon, exactly as in sqlite3 — and SQLite's own tokenizer decides where that semicolon is, via sqlite3_complete() through rusqlite::ffi. Hand-rolling the scan is how SELECT ';' gets read as two statements; a comment or a BEGIN … END body is the same trap. exit, quit, and blank input are exempt from the rule: none is SQL, and waiting for a terminator on the one input meant to end the session would trap the user.

This is the "explicit terminator" option #988 floated, and it does change slice 1's interactive behavior: SELECT 1 ⏎ now needs SELECT 1;. That is the cost of multi-line working for every statement rather than only for unclosed literals, and it is what a sqlite3 user already expects. A migration fragment records it.

History is global: $XDG_DATA_HOME/dirsql/history, falling back to ~/.local/share/dirsql/history (%APPDATA%\dirsql\history on Windows), last 1000 statements, in-memory if none resolve. One file for every directory mirrors sqlite3's single ~/.sqlite_history — a query worked out in one project is worth recalling in the next. Resolution is a pure function over an injected environment lookup, so it unit-tests without touching the filesystem.

Shape

Slice 1's interactive: bool reader became a Lines trait with two implementations — PipedLines (the plain reader) and EditorLines (reedline) — yielding Entry::{Statement, Cancelled, Ended}. Cancelled is Ctrl+C, which is why the interrupt needs no signal handler. The trait is also the unit-test seam: FakeLines drives the whole loop, Ctrl+C included, with no terminal in sight.

sqlite3_complete is FFI, and the crate sets unsafe_code = "deny", so the one call carries an #[expect(unsafe_code, reason = …)] — the explicit opt-in that lint exists to require.

Testing: what a pty can and cannot tell you

This took the most work and is the part worth reading.

A pty is not a terminal. reedline asks the terminal where the cursor is (an ESC [ 6 n Device Status Report) and dies with "the cursor position could not be read within a normal duration" when nothing answers. script hands you a pty but no emulator behind it, so the harness has to answer that query itself — one ESC [ 1;1 R per query is the whole of the emulation needed. curtaincall needs the same fix for a different reason: pyte formats the reply in report_device_status, but its write_process_input hook is an inert stub, so it never reaches the process. Pointing that hook at the pty fixes it.

You cannot reconstruct a screen by stripping escapes — the editor repaints by moving the cursor, so a byte stream is not a screen and cannot be turned into one. That decides the split:

  • packages/rust/tests/repl_tty_e2e.rs (7 tests, runs in CI) drives a real pty and asserts only what was ever drawn: multi-line entry, the continuation prompt, the semicolon-in-a-literal case, the lone terminator, exit words, Ctrl-D, failure-continues. Verified stable over 16 consecutive single-threaded runs, and over three full parallel runs of the whole integration suite.
  • packages/python/tests/e2e/repl_interactive_test.py (5 tests, curtaincall, e2e tier) covers everything that turns on a keystroke landing exactly when the editor is listening — up-arrow recall, history surviving the process, the XDG location on disk, Ctrl+C. Those cases were measured flaky at roughly one miss in four in the pty harness, which is precisely the gap curtaincall's screen-aware waiting closes. This is the dependency slice 1 deferred to here.

Everything decidable without a terminal is unit-tested and runs in CI: sql_is_complete, is_submittable, SqlValidator, history_path, the prompt rendering, and the loop itself through FakeLines.

One practical note for the python suite: curtaincall declares requires-python >=3.12 against this package's >=3.10 floor, so it carries a marker plus an importorskip guard — and the local venv must be recreated on 3.12 (uv venv --python 3.12), or the suite silently skips instead of running. Upstream: curtaincall#26.

PARITY

No drift. The CLI is one shared implementation in the Rust core that every SDK's launcher reaches through run_cli; no SDK public API changed. PARITY.md needs no edit — stated here rather than skipped silently, per AGENTS.md.

Manual run (AGENTS.md)

Driven through a real pty with a cursor-query responder, against a directory holding a.md (6 B) and b.md (10 B), with XDG_DATA_HOME=/tmp/manualhist:

dirsql> SELECT basename, size
   ...> FROM './'
   ...> ORDER BY basename;
[{"basename":"a.md","size":6},{"basename":"b.md","size":10}]

dirsql> SELECT ';' AS semi;
[{"semi":";"}]

dirsql> SELECT 9 AS abandoned          # then Ctrl+C
dirsql>                                # line gone, session alive, nothing ran

dirsql> [up arrow]                     # recalls SELECT ';' AS semi;
[{"semi":";"}]

dirsql> [Ctrl-D]                       # exit 0

And the history file it left behind — <\n> is reedline's own encoding for a statement that spanned lines — showing it stored as one entry, with the abandoned line absent:

$ cat /tmp/manualhist/dirsql/history
SELECT basename, size<\n>FROM './'<\n>ORDER BY basename;
SELECT ';' AS semi;

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (37 passed — 32 existing plus the 5 new curtaincall cases)
  • TypeScript SDK e2e: pass (16 passed)
  • Rust core e2e: pass — cargo test --features cli --tests, including the new repl_tty_e2e (7) and the unchanged repl_pipe_e2e (9)
  • packages/python e2e attestation refreshed (a packages/rust/src change stales both bindings)
  • packages/ts e2e attestation refreshed
  • Command(s) run:
    cargo test --manifest-path packages/rust/Cargo.toml --features cli --lib      # 722 passed
    cargo test --manifest-path packages/rust/Cargo.toml --features cli --tests    # 44 binaries green
    cd packages/python && uv run python -m pytest tests/e2e/ -q                   # 37 passed
    cd packages/python && uv run python -m pytest dirsql -q                       # 108 passed
    cd packages/python && uv run python -m pytest tests/integration -q            # 131 passed
    cd packages/ts && pnpm test:e2e                                               # 16 passed
    cd packages/python && uvx testing-conventions e2e attest 'uv run python -m pytest tests/e2e/ -x -q'
    cd packages/ts     && uvx testing-conventions e2e attest 'pnpm test:e2e'
    
  • Result summary: all green. The one pre-existing, environment-only failure is tests/vtab.rs::unreadable_file_yields_null_content_without_erroring_the_row — this sandbox runs as uid 0, so a chmod 000 file is still readable and the NULL-content assertion cannot hold. Green in CI, unrelated to this change.

Changelog / Migrations

  • Changelog fragment: packages/rust/changelog.d/2026-08-20-repl-line-editor.md, plus packages/python/changelog.d/2026-08-20-curtaincall-dev-dependency.md — pyproject.toml is not exempt from changelog-gate, so the dev-group addition needs its own fragment even though nothing about the published package changes
  • Migration fragment: packages/rust/migrations.d/2026-08-20-repl-line-editor.md — required: a typed statement now needs its terminator, and Ctrl+C no longer ends the process

Red/green

aab9c74 pushed the failing pty suite alone, and CI was observed red on Integration tests alone, for the right reason: a statement typed over two lines was executed as two, giving incomplete input and then a syntax error. The implementation follows in cb75f4e.

Worth recording honestly: that RED commit also contained three cases the implementation could not turn green, because reedline will not run at all under a pty that does not answer its cursor query — the harness had to grow that responder before any of it could pass. Two more were then found flaky rather than failing and moved to curtaincall. The seven that remain are the ones a byte stream can actually answer.

Mutation, and what it caught

Worth calling out, because it changed the code twice. Run against this slice's diff with the full suite (CI's scope), it surfaced three survivors:

Survivor Why it mattered Fix
delete the Signal::CtrlD arm Ctrl-D falls into the catch-all that abandons a line, so the standard way out of a shell silently stops working. Nothing on the Rust side covered it — the coverage was in curtaincall, which cargo-mutants does not run. ctrl_d_leaves_the_session in the pty suite (75a8009). Verified by hand: fails with the arm removed, passes with it.
render_prompt_right → Cow::Borrowed("") Equivalent by construction — the function returned "", so no test could ever kill it. Delegating the prompt to reedline's own (f51e578) makes emptiness a property of the segment rather than of the code, so a test that builds one with non-empty segments pins the forwarding. Also deletes a history-search line that duplicated upstream byte for byte.
render_prompt_right → Cow::Owned("".to_owned()) Same. Same.

One TIMEOUT remains, on deleting the EOF arm of the piped reader: any mutant that stops EOF being EOF makes the loop read an endless empty line, so it hangs rather than failing. The identical construct is already on the base branch from slice 1 and passed this gate there, so timeouts are treated as caught.

A last flake worth recording: running the whole integration suite in parallel starved the pty tests enough that a keystroke was lost. The harness had been typing as soon as the banner appeared, but the banner is written before the editor starts reading. Waiting for the output to stop instead (b04f77d) is the real signal — the last thing drawn is the prompt.

claude added 9 commits August 20, 2026 20:56
The interactive path only exists when stdin is a terminal, so these drive
the real binary under a real PTY (util-linux `script`) and assert on the
terminal's final contents rather than on keystroke timing.

Four fail today, for the reason #988 describes: a statement currently ends
at the first newline, so `SELECT` / `1 AS n;` typed over two lines is two
statements and two errors. Two pass as regression guards -- `exit` must
still leave without a terminator, and continue-on-error must survive
multi-line entry.

Pushed on its own so CI is observed red before any implementation lands.

Refs #988

skip-changelog: test-only commit; the fragment lands with the implementation

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
The interactive prompt was a plain line reader: no recall, no editing, one
line per statement, and Ctrl+C killed the process. It now runs through
reedline, which brings all four at once -- history with the up arrow and
Ctrl+R, emacs editing, Ctrl+C that abandons a line instead of the session,
and statements that span lines.

Three decisions #988 left open, settled:

- **reedline over rustyline.** rustyline covers the same four requirements
  natively and pulls 16 transitive crates to reedline's 52, so the issue's
  "needs more hand-rolling" premise did not hold; the maintainer's call went
  to reedline anyway, which is what #988 assumed.
- **A statement ends at its semicolon**, as in `sqlite3`, with SQLite's own
  tokenizer (`sqlite3_complete`) deciding where that is. Hand-rolling the
  scan is how `SELECT ';'` gets read as two statements. `exit`, `quit` and
  blank input are exempt: none is SQL, and waiting for a terminator on the
  one input meant to end the session would trap the user.
- **History is global**, at `$XDG_DATA_HOME/dirsql/history` (falling back to
  `~/.local/share`, then `%APPDATA%`), mirroring `sqlite3`'s single
  `~/.sqlite_history`: a query worked out in one project is worth recalling
  in the next. Resolution is a pure function over the environment, so it is
  unit-tested without touching the filesystem, and a session with nowhere to
  write still edits and recalls within itself.

The pipe path is untouched, as #988 requires: no editor, no history, and
still one statement per line with no terminator. A redirected script is not
being typed, so there is no continuation prompt for a multi-line rule to
hang off. The split is now a `Lines` trait with two implementations rather
than a boolean, which also gives the loop a seam to be unit-tested through
without a terminal.

Testing splits along what each harness can actually see. The byte stream a
pipe gives back is not a screen -- the editor repaints by moving the cursor
-- so `repl_tty_e2e.rs` (a real pty, answering the editor's cursor-position
query itself) asserts only what was ever drawn: multi-line entry, the
continuation prompt, the literal-semicolon case. Anything turning on a
keystroke landing exactly when the editor is listening was measured flaky at
roughly one miss in four there, and lives in the new curtaincall suite,
which drives a real VT100 and waits on the screen: history recall, history
across sessions, the XDG location, Ctrl+C.

curtaincall needs one workaround worth knowing about: pyte formats the
cursor-position reply in `report_device_status` but its
`write_process_input` hook is an inert stub, so the reply never reaches the
process and every session dies with "the cursor position could not be read".
Pointing that hook at the pty is the whole fix.

Fixes #988

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
The editor reports Ctrl-D distinctly from every other signal, and nothing
in the Rust suite covered that arm: deleting it lets Ctrl-D fall into the
catch-all that abandons a line, turning the standard way out of a shell
into a keystroke that clears the line and does nothing else. Found by
running the mutation gate against this slice's diff; verified by hand that
the case fails with the arm removed and passes with it.

Refs #988

skip-changelog: test-only; the behavior's fragment is in this branch already

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
…y `...>`

The hand-rolled `Prompt` impl duplicated reedline's rendering, including a
history-search line that was byte-for-byte what `DefaultPrompt` already
produces. Wrapping it instead leaves one method ours -- the continuation
marker, where `sqlite3`'s `...>` beats reedline's `::: ` for people who
came from a SQL shell.

The duplication had a concrete cost the mutation gate found: an empty right
prompt written as a constant has no killable mutant, because
`Cow::Borrowed("")` *is* what the function returned. Delegating makes the
emptiness a property of the segment the prompt was built with rather than
of the code, so a test that builds one with non-empty segments pins the
forwarding and kills it. Verified by hand: hardcoding the return fails
`the_prompt_forwards_the_segments_it_was_built_from`, and restoring the
delegation passes it.

Refs #988

skip-changelog: no observable change -- same prompt, same indicators

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
The harness started typing once the banner appeared, but the banner is
written before the editor starts reading -- so the first keystroke could
land in the gap and be read by nothing. Waiting for the output to stop
instead is the real signal: the last thing drawn is the prompt.

`ctrl_d_leaves_the_session` had the same problem one keystroke later. It
sent Ctrl-D at the prompt that follows an executed statement, which is the
gap that made history recall flaky enough to move to curtaincall. Sending
it at a fresh prompt tests the same thing -- with the `Signal::CtrlD` arm
removed the session still fails to exit -- without racing anything.

Caught by running the whole integration suite in parallel, where the
contention made the gap wide enough to lose the key; three consecutive
full runs are clean now.

Refs #988

skip-changelog: test-only

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GU6WPGs4bbkad8QhQfyXDD
@thekevinscott
thekevinscott merged commit 6f9e34a into main Aug 21, 2026
61 checks passed
@thekevinscott
thekevinscott deleted the claude/epic-953-slice2-reedline branch August 21, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REPL slice 2: line editing, history, multi-line via reedline

3 participants