[MilindChatrabhuji]

Overview

Fixes #59

Currently, /export/dump loads the entire database into memory, formats it into SQL statements via string concatenation, and converts the whole dataset into a single Blob. On large databases (e.g. 100MB+ or up to 10GB) or inside Cloudflare Workers / Durable Objects with strict memory limits (128MB), this causes out-of-memory (OOM) errors and execution timeouts.

Solution

1. Incremental Streaming via ReadableStream:

   • Replaced monolithic string concatenation and Blob creation with ReadableStream<Uint8Array>.
   • Emits SQLite file header (SQLite format 3\0), table DDL statements, and row INSERT statements incrementally using TextEncoder.
   • Preserves attachment headers (Content-Type: application/x-sqlite3, Content-Disposition: attachment; filename="database_dump.sql").

2. Chunked Pagination (CHUNK_SIZE = 1000):

   • Fetches rows in batches using SELECT * FROM <table> LIMIT 1000 OFFSET <offset>; so peak memory consumption is strictly $O(1)$ per batch regardless of database size.

3. Type Serialization & SQL Quoting:

   • Accurately serializes null and undefined as NULL (preventing invalid SQL syntax).
   • Serializes boolean values as SQLite integer literals (1 and 0).
   • Escapes single quotes in string literals ('${val.replace(/'/g, "''")}').

4. Comprehensive Unit Tests:

   • Multi-batch chunk pagination across large datasets (>2,500 rows across 3 batches).
   • Accurate SQL serialization for NULL, boolean, and numeric types.
   • Stream consumption via ReadableStream reader (getReader()).
   • Stream-level error propagation (controller.error(err)).

Verification

• Unit test suite: npx vitest run src/export/dump.test.ts (9 of 9 tests passing).
• Prettier check: npx prettier --check "src/export/dump.ts" "src/export/dump.test.ts" (clean pass).

Bounty Claim

@algora-pbc /claim #59