Repository navigation
perf: cover scans with primary keys and estimate key prefixes for TPC-C - #397
Merged
Merged
Conversation
Hash operators now keep their keys and rows in flat buffers, with the hash table holding offsets or indexes, so new keys no longer allocate. - HashAgg evaluates each key straight into one buffer and stores accumulators back to back; the table holds the group index. - StreamAgg keeps the open group's keys and the incoming row's keys in one buffer, and swaps two accumulator sets instead of building new ones. - HashJoin keeps distinct keys in one buffer and build rows in one Vec, chaining the rows of a key through `next`. This drops the bump arena and its lifetime transmutes, and the duplicated no-match branches of the join kinds. - EXCEPT/INTERSECT store right rows flat instead of taking each tuple, and compare values only, no longer the primary key. - Accumulators gain `reset` and `take_result`, so SimpleAgg and StreamAgg reuse them across groups and executions. - Execution hash tables and DISTINCT accumulator sets use an FxHash-style hasher instead of SipHash.
`Range` is no longer generic: its bounds are `RangeValue`, either a value or a `ComputedValue` (`left op right` over parameters and constants for + - * / %). Predicates such as `k >= $3 - 20` now become index ranges with no residual filter; the bound is computed in place when IndexScan binds the range. - A NULL parameter or computed bound empties the range, and `<=>` no longer takes a parameter as a range bound. - Selectivity estimation walks the bounds' fields instead of building stand-in values, so different computed bounds are no longer estimated as a point. - `ParamArena::bind` clones each parameter once.
A secondary index entry holds the primary key, so a scan reading only index columns and primary key fields no longer looks the row up by primary key. `index_covering` gives primary key fields the slots after the index columns, and `decode_index_into` fills them from the entry's value. TPC-C Payment and Order-Status read customers by `c_last` this way. Also add `DataValue::is_tuple`, `as_slice` and `as_mut_slice` for the places that matched tuples by hand.
Index statistics kept only the whole key's NDV, so parameterized ranges on a key prefix spread it evenly over the fields. On TPC-C order_line that overestimated the primary key prefix (w, d, o) at ~23 rows instead of 10, and Delivery read order lines through fkey_order_line_1 plus a row lookup each. ANALYZE now counts the NDV of every key prefix with one HyperLogLog each; `HistogramMeta::number_of_distinct_values` replaces the single NDV, its last entry being the whole key's. A prefix range is estimated as one over its prefix NDV, falling back to the even spread when the statistics have none. This changes the stored statistics format: existing databases must ANALYZE again.
The wasm-only memory storage test still built Range bounds from DataValue, which no longer compiles since Range holds RangeValue.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #397 +/- ##
==========================================
- Coverage 93.08% 93.01% -0.08%
==========================================
Files 272 273 +1
Lines 38711 39012 +301
==========================================
+ Hits 36036 36286 +250
- Misses 2675 2726 +51
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
The closed group's output overwrites the row that closed it, so the executor kept a second set of accumulators to fold that row before writing the output. It now writes the closed group in order and keeps only the row's aggregate arguments, taken from the bytes pool, to open the next group on the next call. `aggregate_arg` is factored out of `update_accumulators` for this.
It returns an iterator, not a slice: a computed value follows its prefix.
A covered scan decodes the memcomparable index key from the index's value type, never with row deserializers, so `IndexInfo::covered_deserializers` was only ever checked for presence. It is now `covered: bool`, and `index_covering` returns just the scan-to-index mapping.
3 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #396, closing more of the TPC-C gap with SQLite on LMDB.
Changes
Rangeis no longer generic: its bounds areRangeValue, a value or aComputedValue(left op rightover parameters and constants for+ - * / %).ol_o_id >= ($4 - 20)in Stock-Level becomes an index range with no residual Filter; IndexScan computes the bound when it binds the range. A NULL bound empties the range, and<=>no longer takes a parameter as a bound. Selectivity estimation walks the bounds' fields, so different computed bounds are no longer estimated as a point.c_lastthis way.order_linenow pick the primary key prefix overfkey_order_line_1plus a row lookup each. This changes the stored statistics format: existing databases must ANALYZE again.IndexInfo::covered_deserializerswas only checked for presence; it is nowcovered: bool.RangeValue, and refactor(range) renamesRangeValue::fieldstoiter.TPC-C (720 s, 1 warehouse, pinned to one P-core, p90 µs)
Measured on the PR head; README and
tpcc/README.mdcarry the same numbers.LMDB is now at or below SQLite on every transaction except Payment (50 vs 40 µs); Delivery matches it (345 vs 344 µs) since prepared statements read
order_lineby its primary key prefix.Testing
make clippy,make test,make test-sltandmake test-slt-sqlitepass on the first three commits. The later commits were checked withmake clippy, the lib tests andmake test-slt;make test-wasm,make miriandmake native-examplespass after the wasm fix.