[mcollina]

Step 3 of PRD-2215. Builds on #36 (merged).

Replica.restorePaged()

Restores the replica's own database page by page, with the VFS of @platformatic/sqlite-js-stream-paged-vfs, now an optional peer dependency (imported only here, so the core stays pure TypeScript). It resolves once the database can be opened, long before it is downloaded:

1. The restore plan as for a full restore. The page indexes of its files (read concurrently) give, for each page, the file with its newest version and the offset of its frame, kept in typed arrays.
2. A marker <db>-litestream/paged is written, the database is created sparse through the VFS with every page missing, and page 1 and the schema (the sqlite_master b-tree with its overflow pages) are filled.
3. The first WAL and the restore record are written as in feat: keep the first sync after a restore incremental #36, so the first sync is incremental here too.

Then:

• Queries from other threads open the database with databaseUri() of the VFS package. A read of a missing page waits while it is read with the next pages of the same file (readAhead, 32 by default) in one ranged read. A page the background fill is reading is read again in this smaller read, so a query never waits for a large one.
• Background fill: ranged reads of up to 4 MiB, four at a time, retried with backoff. result.filled resolves once every page is local.
• Replication goes on: sync() and snapshot() read missing pages first, the replica's own reader opens the database through the VFS (a miss there fails instead of reading zeros), and retention keeps the files the restore reads.
• Stopping: close(), another restore, or a crash leaves the database incomplete. Without a fill in progress, localPos() is 0 and sync()/snapshot() throw ErrIncompleteDatabase, so holes are never uploaded as zeros. ensureExists() and Lavoratore's activation restore it again; a full restore removes the marker.
• It restores fully below minSize, or when the VFS cannot load (result.unavailable).

New exports: PagedRestoreOptions, PagedRestoreResult, ErrIncompleteDatabase, ErrPagedRestoreClosed, ErrPagedVfsUnavailable. decodePageData() takes an optional output buffer (the fill reuses one per read) and rejects pages that do not fit.

S3

openLtxFile() reads a range with a single GetObject instead of a HeadObject followed by a GetObject: open ranges read to the end, ranges ending past it are cut, and a range starting past it (416) reads nothing. That halves the requests of a miss. The in-memory S3 of the tests now answers 416 like S3.

Tests

test/integration/paged-restore.test.ts, with a client that holds the background fill:

• queries answered while the fill is held; the final file equals a full restore, byte for byte; the schema spans several pages, with overflow pages;
• sync (incremental), snapshot and retention during the fill;
• close(), a crash (the restore runs in a child process killed with SIGKILL), and a full restore over an incomplete database;
• minSize.

They need the native library: they run in the paged-vfs CI job (pnpm test:paged-vfs now builds the package and runs them) and are skipped elsewhere.

Also checked by hand against rustfs (the S3 store of CI): the range cases above, and a paged restore of a 9.4 MB database whose first query was answered 88 ms after the restore started.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sfj4nDS4Mc5fAjmzSLYG7k