Repository navigation
Conversation
…nal R2 offload - Fixes outerbase#59 by replacing full-table in-memory buffering with incremental LIMIT/OFFSET chunk streaming - Added generateDumpChunks async generator streaming SQLite header, schema, and row INSERT statements in bounded pages - Implemented escapeSqlValue supporting NULL, booleans, bigints, BLOB hex literals, and single-quote escaping - Added direct HTTP streaming via ReadableStream avoiding execution memory spikes - Added optional R2 multipart offload via ?location=r2 with DATABASE_DUMP_BUCKET binding and 5 MiB part buffering - Added executionContext.waitUntil background processing returning 202 Accepted and optional webhook callback notification - Added comprehensive Vitest suite covering streaming, pagination, multipart upload, and background execution (17/17 passing) - Documented DATABASE_DUMP_BUCKET binding in wrangler.toml and README.md Signed-off-by: Swapnil Jadhav <yuvi.jadhav92@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and Context
Closes #59.
Previously, requesting a database dump via
/export/dumpbuffered the entire database in memory withSELECT * FROM ${table};andnew Blob([dumpContent]). For databases larger than a few megabytes or databases up to 10GB, this caused Cloudflare Workers to run out of memory (OOM) or trigger the 30-second execution wall time limit.Solution & Architecture
This PR introduces an$O(1)$ memory streaming architecture for SQLite database dumps with two complementary execution paths:
1. Incremental Streaming Generator (
generateDumpChunks)LIMIT 1000 OFFSET ${offset}.SQLite format 3\0) is yielded first, followed by each table's schema definition (SELECT sql FROM sqlite_master), and then rowINSERTstatements chunk by chunk.2. SQL Literal Escaping (
escapeSqlValue)null/undefined->NULLnumber/bigint-> numeric string literalsboolean->1/0ArrayBuffer/Uint8Array->X'...'hex BLOB literalsstring-> single-quote escaped'...'3. Direct HTTP Streaming
GET /export/dump, chunks are piped directly to the caller using a standardReadableStreamwith headers:Content-Type: application/x-sqlite3Content-Disposition: attachment; filename="database_dump.sql"4. Background R2 Multipart Offload
?location=r2or?r2=trueis requested:DATABASE_DUMP_BUCKETbinding (R2Bucket), returning400with descriptive error if missing.dump_YYYYMMDD-HHMMSS.sql.createMultipartUpload,uploadPart, andcomplete.executionContext.waitUntilis available (Cloudflare Worker runtime), offloads the work to the background, returning202 Accepted({ key, status: "accepted" }) immediately so large dumps are unaffected by HTTP timeouts.callbackURL viaPOSTonce upload completes.Verification & Test Matrix
src/export/dump.test.tswith 17 tests covering:formatDumpTimestamp: UTC zero-padding and formatting.escapeSqlValue: null, numbers, bigints, booleans, BLOBs, single quote escaping.generateDumpChunks: pagination across multiple pages with LIMIT/OFFSET.dumpDatabaseRoute: HTTP streaming, empty database, tables without data, 500 error propagation.R2 offload: 400 error on missing bucket, multipart upload lifecycle, upload abort on failure, 202 accepted background execution withwaitUntil, webhook callback dispatch./claim #59
Signed-off-by: Swapnil Jadhav yuvi.jadhav92@gmail.com