Repository navigation
Conversation
- Refactor /export/dump, /export/csv/:tableName, and /export/json/:tableName to use ReadableStream - Page table data in chunks using LIMIT/OFFSET pagination (1000 rows/page) to prevent OOM - Update createExportResponse to support ReadableStream - Add unit and integration tests verifying incremental streaming and pagination - Fixes outerbase#59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #59
Database dumps on
/export/dump,/export/csv/:tableName, and/export/json/:tableNamepreviously failed on large databases because they loaded the entire dataset into memory at once as strings/buffers before creating aBlobresponse. On larger datasets or Cloudflare Workers with a 128MB memory budget, this caused out-of-memory errors and request timeouts.Changes
ReadableStream:/export/dump(SQL): Streams-- Table: ...,CREATE TABLE ..., and chunkedINSERT INTO ...statements via aReadableStream. Properly formatsNULLvalues./export/csv/:tableName(CSV): Streams headers followed by CSV records incrementally with RFC 4180 quote/comma/newline escaping./export/json/:tableName(JSON): Streams formatted JSON array entries ([\n { ... },\n ... \n]) incrementally over aReadableStream.LIMIT 1000 OFFSET <n>so peak memory consumption is bounded to O(chunk) rather than O(entire_database).src/export/index.ts: EnhancedcreateExportResponseto acceptReadableStreamasBodyInitwhile preserving attachment headers and chunked transfer.Verification
pnpm vitest run src/export/(all 31 tests passing).Bounty Claim
@algora-pbc /claim #59
Payout Information
0x20A0525b47f087f1Cc498cda257A0feB78D2320B