Repository navigation
perf: reuse executors and resolve tables by id to close the TPC-C gap with SQLite - #396
Merged
Merged
Conversation
- RocksDB transactions (pessimistic and optimistic) now set a transaction snapshot, so a write to a key committed by another transaction after this one began conflicts instead of silently overwriting it. - Remove the ReadCommitted isolation level: RocksDB could not reset the conflict snapshot per statement, so read committed allowed lost updates. TransactionIsolationLevel keeps RepeatableRead as the only level. - Remove docs/transaction-isolation.md and its RC-specific tests.
- Each terminal runs the existing measuring loop on its own connection (shared Database for KiteSQL, a new connection for SQLite); counters and latency histograms are merged at the end. Only the first terminal drives the progress bar. - Retry commit failures, which is where optimistic conflicts surface. - Let other_ware pick the last warehouse; with two warehouses it spun forever.
- ExecNode was 712 bytes (RecursiveCte, IndexScan 472, HashJoin 456), so a 3-node statement allocated a ~2KB node Vec per execution and hit glibc's large-bin path (malloc_consolidate). Boxing every variant makes the enum pointer-sized; dispatch is unchanged via deref coercion. - Add exec_node_stays_pointer_sized to keep it that way. TPCC LMDB 60s on cpu8, alternating runs: 183052/188373 -> 188003/194744 TpmC; Transaction::execute drops from ~8.4% to ~5.3% of CPU samples.
…etime - ExecArena gains a transaction lifetime 't. Scan iterators (tuple, index, table, view) live in arena-owned StorageIter slots and nodes keep an IterSlot, so ExecNode only borrows the plan and no longer carries T. - ExecArena::new takes the execution context and transaction; init_context and the Option-wrapped context are gone. - RecursiveCte's recursive arena is hosted by the outer arena, keeping the node free of transaction-bound state while reusing it across steps. - ExecNodes::push only appends; the overwrite path and seek are removed.
- Executor nodes hold child ExecNodes directly and call child.next_tuple / child.reset. The arena-owned node array, ExecId, RefCell slots, the raw slot pointer and the executing counter are gone; ExecNode::next_tuple now clears the status, dispatches, and reports Continue/End as a bool. - MarkApply owns its right root; RightInput keeps only the scan state. NestedLoopJoin no longer copies the right input id into its states. - RecursiveCte builds each step into the outer arena and truncates the step's iterator slots afterwards, replacing the hosted sub-arena. The step's scans now carry the statement stamp like the rest of the tree. - Build no longer threads an execution context or transaction: ExecArena stores the table cache, the only catalog state executors read, and ExecutionContext plus LogicalPlan::exec_capacity_hint are removed.
TableScanOperator replaces `columns` with `TupleDeserializers`: one entry per table column in table order, either decoded or skipped, cut after the last read column. It is built from the catalog at bind time and column pruning narrows it in place, so executors pass it straight to the storage reader instead of rebuilding deserializers on every execution. - storage: `read`/`read_by_index` take `&TupleDeserializers`; the column-based wrappers and `create_deserializers` are removed. - scans returning primary keys keep their primary key columns read. - `index_covering`, `is_index_covers` and `fmt_explain_list` accept iterators; `index_covering` resolves coverage in a single pass. - the persisted view plan format changes with TableScanOperator.
An executor no longer carries its transaction: it holds the built tree,
its iterator slot count and output schema, and each execution supplies
its own ExecArena. A finished iterator hands it back with
`into_executor`, reset and ready to run again, possibly in another
transaction:
let exec = tx1.execute(&prepared, params)?.into_executor()?;
tx1.commit()?;
let iter = tx2.execute(exec, other_params)?;
- `Database::execute` and `Transaction::execute` take `impl IntoExecutor`,
implemented for prepared plans and executors.
- `TransactionIter` gains a plan lifetime `'p`, separate from the
transaction borrow, so a returned executor outlives the transaction.
- Building a tree no longer needs a transaction: build functions take an
`ExecBuilder`, which counts slots ahead of execution or allocates them
in the arena while a recursive CTE runs.
- `ExecutorNode::reset` is required; write nodes keep their input and
track whether they ran instead of taking it.
- `BindSource::execute` builds return an executor; `Executor::from_plan`
covers the SQL paths.
A Program is a bound and optimized plan together with its executor tree. Each execution binds new parameters and reuses the tree instead of rebuilding it, in any transaction of the database that created it. - `Database::program_sql` / `Transaction::program_sql` and the ORM `program` replace `prepare_sql` / `prepare`. - `Program::execute(&mut tx, params)` returns a `ProgramIter`, which puts the program back when the execution ends, even if it fails. - Remove `PreparedPlan`, `Database::execute` and `Transaction::execute`; prepared statements now run inside a transaction. - `ParamArena` owns its plan arena, and `PlanInput` is gone since plans are no longer borrowed. - TPCC keeps one program per statement; add the `program` example.
An execution now borrows its program instead of moving it out: the executor and the metadata arena stay inside the `Program`, and `ProgramIter` holds the transaction guard plus `&mut Program`. - Remove `ExecuteError`, `Reclaim` and `try_with!`; `BindSource` returns `DatabaseError` again, with a `PlanArena` instead of a boxed arena. - Remove the unsafe move out and write back (`ptr::read`/`ptr::write`, `AbortOnUnwind`, `ManuallyDrop`) and `TransactionIter::into_program`. - Split `begin_execution` out of `start_execution` to start an executor in place; `TransactionGuard::next_tuple` serves both iterators. - `ParamArena::bind` overwrites the bound values in place instead of rebuilding every parameter expression.
…E semantics Insert and Update no longer look up the table or rebuild a snapshot of it on every execution. - `TableCatalog::dml_table` builds a `DmlTable` once at bind time: the columns, serializers, primary key positions and the key expressions of the secondary indexes the write maintains, chosen by a `keep_index` closure. `InsertOperator` and `UpdateOperator` carry it. - The binder hands Insert full rows in table order, filling in column defaults, so the executor no longer maps input columns or evaluates defaults; `is_mapping_by_name` is gone. - `UpdateOperator.value_exprs` holds table positions. Every assignment reads the old row, and assigning a column twice is rejected. - `get_column_by_id` returns the column position too, which removes the linear scans that looked it up again. - Add `TupleImpl::clear`; Insert reuses one tuple for its rows.
Every execution used to build a fresh `ExecArena`, so its result row and the `TableCodec` key and value buffers started empty and grew again on each run. - Split the arena: `ExecBuffer` holds what outlives an execution (the result, the table codec and the runtime probe stack) and is owned by the `Executor`; `ExecArena` keeps the transaction, the storage iterators and the other per-execution state. Nodes get both through `ExecContext`. - `ExecBuffer::reset` drops what an unfinished execution left behind. - Size the key buffers once with `DataValue::memcomparable_len_hint`, and the tuple and index values with their length hints plus the stamp.
Replace the 8-byte table-name hash prefix with a monotonically allocated
table id, so keys no longer depend on names and hash collisions are gone.
- Root is keyed by the raw name and maps to TableMeta { name, id, is_view },
shared by tables and views as the name directory.
- NextTableId holds the next id; tables and views draw from it.
- Column, index meta, statistics, index and tuple keys use the big-endian id;
views are stored under View{id}.
- Drop the Hash marker keys and DuplicateSourceHash.
- Codec and storage write paths take TableId; DML gets it from DmlTable or
the catalog.
The on-disk format is incompatible with earlier versions.
Table names are resolved once at bind time; planning, execution and
serialization work with table ids, and names only come back for display.
- The table arena owns the catalog: `NameMapping`, `TableCache` keyed by
`TableId` and `ViewCache`. `MetaArena::table(id)` looks a table up, and
`table_name` derives from it, replacing the separate name registry.
- `ColumnRelation` splits into `Table { table_id, column_id }` and
`Source { source_id, column_id, is_temp }`. Other FROM items only carry an
id; the source name list is gone and explain shows them as `#id.column`.
- The binder matches columns to sources by relation instead of names, and
`BoundSource` borrows its name from the catalog.
- Drop `table_aliases`: a FROM item names a real table, not an earlier
alias. `with_pk` holds a table id.
- Scans, DML and DDL operators carry the table id; DDL executors take the
name from the cached catalog, and binder and storage entry points take
`&str` instead of owned names.
- Column ref decoding finds catalog columns through the arena, so the
decode context no longer holds a transaction.
Decoding a string or binary column allocated a fresh buffer for every value, and projection appended its output to the input row, so the row grew and was cloned value by value on each row. - `BytesPool` in `TableCodec` keeps the buffers of dropped string and binary values. Decoding takes from it, and the row buffer gives its old values back before the next decode. It holds at most 64 buffers of up to 4096 bytes. - Char, Varchar and Varbinary decode from the input slice instead of a zeroed vector; CHAR trims the padding before copying. - `TupleValueSerializable::filling_value` takes the pool and is the only decode method; `from_raw` is gone. - `with_tuple_bound` and `with_tuple_unchecked` hand the pool to their closure, since the codec stays borrowed while it runs. - `rewrite` evaluates into a spare vector and swaps it with the row, instead of appending and rotating; borrowed values are copied through the pool. - Primary key lookups clone the tuple id only when they hit and keep it.
- tpcc: `explain_tpcc` names columns through `SchemaView::full_name`. - macros-test: drop `UserOrderSummary`, unused since the ORM refactor. - db: `test_prepare_statment` expects derived table columns as `#id.column`. - slt: explain plans match source ids with `<slt:ignore>`, since the ids depend on the views loaded before; `show views` lists views by id.
Rerun of the 720-second single-thread matrix on cpu8 (2026-10-31). KiteSQL LMDB reaches 187161 TpmC against 179039 for SQLite in the same run.
The SQLite corpus expects `UPDATE t1 SET x=3, x=4, x=5` to keep the rightmost assignment; standard SQL rejects a column assigned more than once, as KiteSQL now does. Skip that statement and the queries that check its effect, listed in `NON_STANDARD` with the reason.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #396 +/- ##
==========================================
- Coverage 93.39% 93.08% -0.31%
==========================================
Files 272 272
Lines 39089 38711 -378
==========================================
- Hits 36508 36036 -472
- Misses 2581 2675 +94
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
- AliasType::Expr(ExprRef) becomes AliasType::Column(ColumnRef): the target only carries the output column identity, so the binder no longer allocates a column expression to wrap it. - BinderContext::table_id resolves the name mapping without loading the table catalog; truncate, delete and the ORM use it. - Source names borrow from the catalog, and INSERT without a column list borrows the table schema instead of copying it. - INSERT maps targets to table columns in one pass instead of scanning the targets for every column. - Drop begin_statement_scope/end_statement_scope: no backend implements them, and several error paths never ended the scope. TransactionGuard no longer needs a raw transaction pointer. - ALTER TABLE builds its old and new serializers once per statement instead of once per row. - memcomparable_len_hint lists every DataValue variant with its encoded length instead of a 16-byte fallback. - Rename begin_execution to start_execution_inner and the table arena locals from catalog to arena. - Add an UPDATE case where every SET expression reads the old row.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts the compute-layer overhead on the TPC-C path. In the refreshed 720-second single-thread run (cpu8, no fsync), KiteSQL LMDB reaches 187161 TpmC against 179039 for SQLite in the same run; the previous README had 165956 against 191484.
Changes
ProgramreplacesPreparedPlan, runs in place, andExecuteErroris gone. Execution buffers survive across runs.DmlTable), and UPDATE follows standard semantics: every assignment reads the old row, and assigning a column twice is rejected.#id.columnin EXPLAIN.BytesPoolinTableCodec, and projection swaps its output with a spare vector instead of appending to the input row.--threadsruns multiple terminals.Compatibility
PreparedPlanandExecuteErrorare removed in favor ofProgram.Results (2026-10-31)
Latencies are p90 in µs. These are single runs; SQLite itself came in about 7% below the previous run, so the machine noise is of that order. RocksDB dropped from 52931 to 46291 TpmC and needs a rerun to tell a regression from noise.
Testing
make test,make test-wasm,make test-slt,make test-pythonandmake clippypass. EXPLAIN expectations in slt match source ids with<slt:ignore>, since the ids depend on the views loaded earlier in the run.Versions:
kite_sql0.6.1,kite_sql_serde_macros0.2.6.