Repository navigation
feat: async path and SlateDB storage on S3, with TPC-C on it - #400
Merged
Merged
Conversation
`#[dual]` (kite_sql_serde_macros) compiles one async source as the sync path by default, identical to the hand-written sync code, and as the async path with the `async` feature. The storage traits, the executor and the database/ORM API run on both; binder, optimizer and codec encoding stay sync. - `src/dual.rs`: `MaybeSend`/`MaybeSync`/`MaybeStatic`, which are `Send`/ `Sync`/`'static` on the async path only, and `block_on` for async tests. - `MemoryStorage` is `Send + Sync` with snapshot transactions; on the async path every read suspends once. It is the only async storage. - `ExecNode` boxes node futures on the async path, as the tree is recursive; `OwnedPtr`/`TxnPtr`/`ArenaBound` keep futures `Send`. - `run_mut` takes a `ConsumeMut` implementation instead of a closure, and `BindSource` drops its lifetime parameter, so the futures of the database and ORM API are `Send` and can be spawned on tokio. - Tests run on both paths over `MemoryStorage`; `make test-async` and `make test-slt-async` cover the async path, and the `async_hello_world` example runs on tokio.
`SlateDbStorage` (feature `slatedb`, async path only) runs KiteSQL on SlateDB, an LSM tree kept in object storage. Each transaction is a SlateDB snapshot-isolation transaction; a commit that conflicts with a write committed since fails. Commits wait until the write is durable in object storage unless `await_durable(false)` is set. - `slatedb-s3` adds `S3Config` and `SlateDbStorage::open_s3` for Amazon S3 or an S3-compatible store; unset settings fall back to the `AWS_*` environment variables and the default credential chain, and `Debug` hides the secrets. - SlateDB is built without its default features (no `aws` unless `slatedb-s3`), keeping the `foyer` block cache. - The async sqllogictest suite now runs over SlateDB on an in-memory object store; `make test-slatedb` runs the storage unit tests. - The `async_s3` example replaces `async_hello_world`: the ORM workload on a spawned tokio task, over S3, reopened and read back.
…stricts TPC-C keeps a terminal's home warehouse constant over the measurement, but every transaction drew a random warehouse, so terminals collided on warehouse and district rows regardless of `--num-ware`. Thread `i` now uses warehouse `i % num_ware + 1`; remote warehouses are still drawn for New-Order supply lines and remote Payment customers. New-Order, Payment and Order-Status drew districts from `1..10`, never picking district 10; they now draw from `1..=10` like Stock-Level. Results are not comparable with runs before this change.
… and cache options tpcc: - The workload is async; the sync backends (LMDB, RocksDB, SQLite, dual) implement the async traits by calling their sync APIs. LMDB tpmC is unchanged within noise. - `kitesql-slatedb-s3` backend, built with `--no-default-features --features slatedb-s3`, with `--s3-bucket`, `--s3-endpoint`, `--s3-allow-http` and memory/disk cache flags. - `--terminals` runs several terminals per thread on async storages. - Conflicts are counted apart from failures and retried after a random, growing backoff; the response time covers the retries. kite_sql: - `DatabaseError::is_conflict` for RocksDB Busy/TryAgain and SlateDB transaction conflicts. - `SlateDbScanOptions`: scans read ahead 256KiB with 4 fetches in flight and cache the blocks, instead of one uncached 4KiB block per request. - `SlateDbStorage::open`/`open_s3` take `SlateDbCacheOptions`: the memory of the block cache, a local disk tier, and caching compaction output. `SlateDbStorage::close` closes the database and the cache it built.
Replace the TPC-C tables in README.md and tpcc/README.md with 720-second runs of LMDB, RocksDB, SlateDB on a local S3 server and SQLite, at 1 thread / 1 warehouse and 4 threads / 4 warehouses, and document the async tpcc options and the SlateDB build.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #400 +/- ##
==========================================
- Coverage 93.03% 92.86% -0.18%
==========================================
Files 274 274
Lines 39998 40269 +271
==========================================
+ Hits 37214 37394 +180
- Misses 2784 2875 +91
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
….7.0 kite_sql: - Rename the `#[dual]` attribute to `#[maybe_async]` (src/maybe_async.rs, kite_sql_serde_macros/src/maybe_async.rs), and bring it into scope crate-wide with `#[macro_use]` instead of per-file imports and `#[crate::dual::dual]` paths. - `AsyncIter`: the typed ORM iterators (`OrmIter`, `TxnOrmIter`, `ProjectValueIter`, `ProjectTupleIter`) read with `next().await` on the async path through one trait, `Iterator` on the sync path, in place of four inherent `async fn next`. - `MemoryStorage` is back to its single-threaded sync implementation and only compiles on the sync path. The async unit tests run on SlateDB over an in-memory object store (`TestStorage`, `new_test_storage`), on a tokio runtime; `make test-async` enables `slatedb`. The storage conformance checks also run on SlateDB. - Version 0.7.0. tpcc: - One generic `KiteSqlBackend<S: Storage>` for LMDB, RocksDB and SlateDB replaces three near-identical backends; a `kite!` macro awaits KiteSQL calls in the `slatedb-s3` build. Tests and docs: - sqllogictest picks its runner per path without moving the file loop. - examples/async_s3.rs inlines its read and write and closes the storage. - README advertises zero-copy reads and the async API on S3; details move to docs/features.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
KiteSQL only ran on local, synchronous storages. This PR adds an async path and a storage on SlateDB, so KiteSQL can keep its data in object storage such as S3, and extends TPC-C to measure it.
Issue link:
What is changed and how it works?
asyncfeature): the storage traits, executor and database API compile either sync or async from one source. The sync build is unchanged;asyncexcludesrocksdb,lmdb,python,shellandwasm.slatedb,slatedb-s3features):SlateDbStoragemaps each KiteSQL transaction to a SlateDB snapshot-isolation transaction.open/open_s3takeSlateDbCacheOptions(block cache memory, a local disk tier, caching compaction output);closecloses the database and its cache.SlateDbScanOptionsreads ahead 256KiB with 4 fetches in flight instead of one uncached 4KiB block per request.S3Configfalls back toAWS_*and redacts secrets inDebug.DatabaseError::is_conflictfor RocksDBBusy/TryAgainand SlateDB transaction conflicts.kitesql-slatedb-s3backend in a separate build (--no-default-features --features slatedb-s3) with S3 and cache flags.--terminalsruns several terminals per thread on async storages.tpcc/README.md: new 720-second results, 1 thread / 1 warehouse and 4 threads / 4 warehouses:Code changes
Check List
Tests
Manual: the sqllogictest suite runs on SlateDB with an in-memory object store under the
asyncfeature;make test-slatedb; TPC-C on all backends as intpcc/README.md, SlateDB against a locals3s-fs.Side effects
Note for reviewer
SlateDB is slow in TPC-C even with every block cached. Delivery's
MIN(no_o_id)onnew_ordersskips the tombstones of delivered orders, which accumulate until a compaction reaches the last sorted run, and SlateDB's iterator costs about 1.25µs per skipped entry against under 0.27µs for RocksDB, which skips more of them. Tuning compaction to drop tombstones sooner gained 6% at best on one core.