Skip to content

test: acceptance of the paged restore, and its benchmark - #39

Open
mcollina wants to merge 1 commit into
mainfrom
test/paged-restore-acceptance
Open

mcollina wants to merge 1 commit into
mainfrom
test/paged-restore-acceptance

Conversation

@mcollina

@mcollina mcollina commented Nov 6, 2026

Copy link
Copy Markdown
Member

The two items left on PRD-2215 after #37.

Once filled, nothing is read from the replica

New test: after filled resolves, a worker thread runs PRAGMA integrity_check and a full scan through the VFS, writes a row, and the replica syncs and snapshots. The client records every openLtxFile(): none.

Acceptance: time to first response as the database grows

Test (deterministic, runs in CI): the bytes read before restorePaged() resolves, for a database of about 8.6 MB and one four times larger (33 MB). Before the database can be opened, the restore reads two LTX headers, the page indexes of the plan, which take a few bytes per page, and the schema with its read-ahead:

Database Read before opening
8.6 MB 112 KB (1.3%)
33 MB 176 KB (0.5%)

It asserts that 4× the data reads less than 2× the bytes, and under 2% of the database.

Benchmark: bench/bench-paged-restore.ts (pnpm bench:paged), which needs the paged VFS built. The replica is read through a simulated network: 20 ms per request, and one 100 MB/s link shared by concurrent reads. A worker thread reads a random row as soon as the paged restore resolves. Apple M4 Max, Node 24.18:

Database Full restore Paged: open Paged: first response Paged: every page
16 MiB 1.31 s 298 ms 335 ms 510 ms
64 MiB 4.30 s 318 ms 373 ms 951 ms
257 MiB 16.03 s 343 ms 370 ms 2.78 s
1027 MiB 64.02 s 447 ms 528 ms 10.09 s

The results are in the README, under Benchmarks.

Two findings, not addressed here

  • About 200 ms of every open is the restore plan. calcRestorePlan() lists the ten levels of the replica one after another, one request each. Listing them concurrently would cut the time to first response by about a third with S3-like latency, and would speed up full restores too.
  • The full restore is CPU-bound at about 16 MB/s on this machine. With the network simulation off, 16 MiB takes 1.06 s and 64 MiB 4.0 s, while the paged fill writes the same 64 MiB in 214 ms. The likely causes are the streaming LTX reader and one awaited write per page. Even databases below the paged threshold would gain from fixing it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sfj4nDS4Mc5fAjmzSLYG7k

Two tests of the paged restore:
- once every page is restored, queries, writes, syncs and snapshots read
  nothing from the replica;
- the bytes read before restorePaged() resolves barely grow with the
  database: 112 KB for 8.6 MB, 176 KB for 33 MB (the page indexes grow, the
  rest does not), under 2% of the database.

bench/bench-paged-restore.ts (pnpm bench:paged) measures the time to the
first response of a paged restore against a full restore as the database
grows, through a simulated network: 335 ms to 528 ms from 16 MiB to 1 GiB,
where a full restore takes 1.3 s to 64 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sfj4nDS4Mc5fAjmzSLYG7k

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant