Skip to content

feat(export): stream database dumps with bounded pagination and optional R2 offload (#59) - #308

Open
Yuvi1192p wants to merge 1 commit into
outerbase:mainfrom
Yuvi1192p:feat/stream-database-dumps
Open

Yuvi1192p wants to merge 1 commit into
outerbase:mainfrom
Yuvi1192p:feat/stream-database-dumps

Conversation

@Yuvi1192p

Copy link
Copy Markdown

Problem and Context

Closes #59.

Previously, requesting a database dump via /export/dump buffered the entire database in memory with SELECT * FROM ${table}; and new Blob([dumpContent]). For databases larger than a few megabytes or databases up to 10GB, this caused Cloudflare Workers to run out of memory (OOM) or trigger the 30-second execution wall time limit.


Solution & Architecture

This PR introduces an $O(1)$ memory streaming architecture for SQLite database dumps with two complementary execution paths:

1. Incremental Streaming Generator (generateDumpChunks)

  • Replaces unbounded full-table queries with paginated iteration using LIMIT 1000 OFFSET ${offset}.
  • SQLite file header (SQLite format 3\0) is yielded first, followed by each table's schema definition (SELECT sql FROM sqlite_master), and then row INSERT statements chunk by chunk.
  • Table data is never materialized in worker memory at once.

2. SQL Literal Escaping (escapeSqlValue)

  • Properly serializes all SQLite data types:
    • null / undefined -> NULL
    • number / bigint -> numeric string literals
    • boolean -> 1 / 0
    • ArrayBuffer / Uint8Array -> X'...' hex BLOB literals
    • string -> single-quote escaped '...'

3. Direct HTTP Streaming

  • When accessed directly via GET /export/dump, chunks are piped directly to the caller using a standard ReadableStream with headers:
    • Content-Type: application/x-sqlite3
    • Content-Disposition: attachment; filename="database_dump.sql"

4. Background R2 Multipart Offload

  • When ?location=r2 or ?r2=true is requested:
    • Validates presence of DATABASE_DUMP_BUCKET binding (R2Bucket), returning 400 with descriptive error if missing.
    • Generates ISO timestamped key dump_YYYYMMDD-HHMMSS.sql.
    • Buffers chunks until R2's 5 MiB part size threshold is reached and flushes parts incrementally via createMultipartUpload, uploadPart, and complete.
    • Aborts multipart upload if an error occurs to prevent orphaned parts in the bucket.
    • If executionContext.waitUntil is available (Cloudflare Worker runtime), offloads the work to the background, returning 202 Accepted ({ key, status: "accepted" }) immediately so large dumps are unaffected by HTTP timeouts.
    • Notifies an optional callback URL via POST once upload completes.

Verification & Test Matrix

  • Comprehensive unit test suite in src/export/dump.test.ts with 17 tests covering:
    • formatDumpTimestamp: UTC zero-padding and formatting.
    • escapeSqlValue: null, numbers, bigints, booleans, BLOBs, single quote escaping.
    • generateDumpChunks: pagination across multiple pages with LIMIT/OFFSET.
    • dumpDatabaseRoute: HTTP streaming, empty database, tables without data, 500 error propagation.
    • R2 offload: 400 error on missing bucket, multipart upload lifecycle, upload abort on failure, 202 accepted background execution with waitUntil, webhook callback dispatch.
  • Vitest Result: 17/17 passed (100%).
  • Codebase formatted via Prettier and zero TypeScript errors in implementation files.

/claim #59

Signed-off-by: Swapnil Jadhav yuvi.jadhav92@gmail.com

…nal R2 offload

- Fixes outerbase#59 by replacing full-table in-memory buffering with incremental LIMIT/OFFSET chunk streaming
- Added generateDumpChunks async generator streaming SQLite header, schema, and row INSERT statements in bounded pages
- Implemented escapeSqlValue supporting NULL, booleans, bigints, BLOB hex literals, and single-quote escaping
- Added direct HTTP streaming via ReadableStream avoiding execution memory spikes
- Added optional R2 multipart offload via ?location=r2 with DATABASE_DUMP_BUCKET binding and 5 MiB part buffering
- Added executionContext.waitUntil background processing returning 202 Accepted and optional webhook callback notification
- Added comprehensive Vitest suite covering streaming, pagination, multipart upload, and background execution (17/17 passing)
- Documented DATABASE_DUMP_BUCKET binding in wrangler.toml and README.md

Signed-off-by: Swapnil Jadhav <yuvi.jadhav92@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant