Skip to content

feat: wire the durable (L2) store seam — chdb-serverless[durable]#3

Open
ShawnChen-Sirius wants to merge 1 commit into
mainfrom
feat/durable-store-seam
Open

feat: wire the durable (L2) store seam — chdb-serverless[durable]#3
ShawnChen-Sirius wants to merge 1 commit into
mainfrom
feat/durable-store-seam

Conversation

@ShawnChen-Sirius

@ShawnChen-Sirius ShawnChen-Sirius commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What

Adds chdb_serverless/durable.py — the DurableStore that the store seam already dispatches to for CHDB_STORE=durable:<url>?id= (see store.py) — and points the durable extra at chdb-durable[s3]. This lands the L2 (durable object) tier of the store seam.

How it works

Same open_store() API as L1. CHDB_STORE=durable:s3://bucket/prefix?id=user-123 runs the analyst against a chdb.durable object on object storage you own (S3 / GCS / Azure Blob / any S3-compatible / local). Reads go to the object; writes are auto-detected, logged to its WAL, and flushed immediately (a committed write survives the instance dying). chDB is single-writer per process, so this store is the process's one engine.

⚠️ Dependency status — read before releasing

chdb-durable (the DAO primitive) is not yet on PyPI — it is slated to land in the main chdb package as chdb.durable, at which point the extra becomes chdb[durable]. Until then, pip install chdb-serverless[durable] will not resolve. The base package and the L1 (local:) / L3 (memory:) paths are unaffected. Recommend not cutting a chdb-serverless release advertising [durable] until chdb-durable ships. Landing this now keeps the adapter next to the seam that already references it.

Testing

Verified locally and against MinIO: durable state persists across restart (base + WAL replay). In-region cold-open measured on AWS/Azure/GCP (~0.5–0.85s for 1M rows; control-plane-round-trip-bound, not restore-bound).

🤖 Generated with Claude Code

Note

Wire the L2 durable store seam backed by chdb.durable objects

  • Adds durable.py, a new module implementing a thread-safe DurableStore that routes SQL to a chdb.durable object.
  • DurableStore.__init__ parses a durable target URL, reads config overrides from environment variables (CHDB_DURABLE_ID, CHDB_DURABLE_OWNER, CHDB_DURABLE_DB), and opens the named object.
  • Writes (classified via _is_write, which strips leading SQL comments before checking the first keyword) call execute then flush and return an empty string; reads call query with a configurable format (default JSONCompact).
  • Exposes checkpoint and close methods, both guarded by the same mutex.
  • Updates pyproject.toml with clarifying comments on the durable optional dependency group.

Macroscope summarized ddd9d53.

Comment thread src/chdb_serverless/durable.py
@ShawnChen-Sirius
ShawnChen-Sirius force-pushed the feat/durable-store-seam branch from 5a128b8 to f678e52 Compare July 21, 2026 07:17
Add chdb_serverless/durable.py — the DurableStore the store seam already
dispatches to for CHDB_STORE=durable:<url>?id= — and point the durable extra
at chdb[durable]. Same open_store() API as L1; reads go to the object, writes
are logged to its WAL and flushed immediately. chDB is single-writer per
process, so this store is the process's one engine.

The DAO primitive ships as chdb.durable (subpackage of the main chdb package).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ShawnChen-Sirius
ShawnChen-Sirius force-pushed the feat/durable-store-seam branch from f678e52 to ddd9d53 Compare July 21, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant