Replay WAL in the object's database context - #618
Merged
Conversation
Contributor
Author
|
@chibugai please review it. |
There was a problem hiding this comment.
Pull request overview
Fixes chdb.durable crash/reopen correctness by ensuring WAL replay runs with the durable object’s manifest database set as the session’s current database, so unqualified table names during replay target the intended database (instead of default).
Changes:
- Set the session database via
USE {db}before replaying WAL segments during_restore. - Document WAL replay semantics for unqualified table names vs. cross-database statements.
- Add a regression test that writes unqualified statements after
USE, flushes (WAL-only), reopens, and asserts data is restored into the manifest DB.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
chdb/durable/object.py |
Sets current DB before WAL replay to ensure unqualified statements replay into the object’s manifest database. |
chdb/durable/wal.py |
Documents that WAL replay uses the manifest DB as the current database; cross-DB statements must be qualified. |
tests/test_durable.py |
Adds regression coverage for WAL replay with unqualified table names after USE and reopen. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
WAL replay ran with `default` as the current database, so any statement
written with an unqualified table name (after the caller set its database
with `USE`) silently replayed into the wrong database — invisible to the
caller and dropped by the next `BACKUP DATABASE {db}` checkpoint, i.e.
committed writes lost after a crash/reopen.
Restore the object's database (the manifest's `db`) as the current
database before replay, and document the guarantee in the WAL usage
notes. Add a regression test: every other durable test qualifies table
names (mem.t), so none of them exercised the replay database context.
Also backtick-quote the database identifier in every DDL/USE statement
(CREATE/RESTORE/BACKUP/USE DATABASE), so a name needing quotes (e.g. one
with '-') is valid and can't break out of the statement; the durable
package quotes locally rather than importing clickhouse_connect. Covered
by a round-trip test on a hyphenated database name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ShawnChen-Sirius
force-pushed
the
fix/durable-wal-replay-db-context
branch
from
July 28, 2026 00:37
95a62e8 to
9cd8e66
Compare
Contributor
Author
|
@wudidapaopao please review it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Restore the object's database as the current database before replaying the write-ahead log in
chdb.durable.The bug
WAL replay ran with
defaultas the current database. A durable object sets its own database (the manifest'sdb) as current, and callers write with unqualified table names against it. On reopen, those logged statements replayed intodefaultinstead — so the tables were invisible to the caller and dropped by the nextBACKUP DATABASE {db}checkpoint. Net effect: committed writes silently lost after a crash/reopen, whenever the caller used unqualified names (the natural style once you've set a database).The fix
USE {db}before replay (one line in_restore), plus a note in the WAL usage rules.Test
Adds a regression test. Every existing durable test qualifies its table names (
mem.t), so none exercised the replay database context; the new test uses an unqualified name and reopens. Without the fix it fails withUNKNOWN_TABLE; with it, the full suite passes (verified locally and against MinIO).Context
Found while building a memory backend on
chdb.durable(auxten/clickmem#7), whose tables are unqualified — this fix is a correctness prerequisite for that use.Note
Replay WAL in the object's database context and quote database identifiers
_quote_identhelper in object.py to backtick-quote ClickHouse identifiers, allowing database names containing hyphens or other special characters in DDL statements.USE <db>statement before WAL replay inDurableObject._restoreso unqualified table references in WAL entries replay against the object's own database instead ofdefault._quote_identconsistently inopen,_restore, andcheckpointforCREATE DATABASEandBACKUP DATABASEstatements.default.Macroscope summarized 9cd8e66.