[Hackathon] Bound aggregate payload retention in buffered pagination (follow-up to #83) #517

Verified against (commit SHA or released version)

988c0355bb05631e9c6c1785e443937ff2a88c0e (current public main, checked October 18).

Discord Username / User ID

jerry_nee (display name: JerryNee)

What does this improvement do?

Follow up on the remaining pagination accumulation work explicitly separated from #83, not reopen the already-fixed single-response buffering issue. In the closing comment, the maintainer invited a new issue scoped to this remaining work. Credit for the original diagnosis belongs to #83's author.

paginate() still collects every page into one items array. The 64 MiB HTTP response cap bounds each individual response, and MAX_AUTO_PAGES bounds request count, but neither provides a practical aggregate payload budget for that array. An explicit --max-items helps callers who already know to limit their query; the default has no item/byte budget separate from the page-count guard.

On current main the relevant callers include project list, test list, and bare test steps <id>. test result --history is currently single-page, so I am not repeating the older report's claim that it automatically accumulates history pages.

This is a bounded-memory robustness improvement, not a report of a production crash. Larger accumulated responses also remain resident before the command renders its output. The request is to make these existing buffered paths fail predictably within a documented budget, or adopt a bounded streaming design agreed with maintainers.

Details / implementation notes

Current-code verification

• src/lib/pagination.ts: each page.items entry is appended to the same array. The default maxItems is undefined.
• src/lib/http.ts: the successful-response limit applies separately to each HTTP request.
• project.ts:153, test.ts:1015, and test.ts:6303 still use paginate() for their automatic-paging branches. An explicit page size without --max-items uses a single page instead.

Local reproduction

I exercised the built paginate and HttpClient with injected fetch, a dummy key, and no network access. The synthetic records deliberately contain large payloads to isolate aggregate retention; this does not claim that a production TestSprite endpoint returns this record shape or that a particular production field accepts a 256 KiB value.

The following runs from the repository root after npm ci && npm run build:

import assert from 'node:assert/strict';
import { HttpClient } from './dist/lib/http.js';
import { paginate } from './dist/lib/pagination.js';

globalThis.fetch = async () => { throw new Error('No network allowed'); };
const MiB = 1024 * 1024;
const totalRows = 300;
const payloadBytes = 256 * 1024;

async function scenario(name, flags = {}) {
  let requests = 0;
  let largestPageBytes = 0;
  const client = new HttpClient({
    baseUrl: 'https://mock.invalid',
    apiKey: 'sk-user-dummy',
    env: {},
    fetchImpl: async url => {
      const query = new URL(url).searchParams;
      const start = Number(query.get('cursor') ?? 0);
      const end = Math.min(start + Number(query.get('pageSize')), totalRows);
      const body = JSON.stringify({
        items: Array.from({ length: end - start }, (_, i) => ({
          id: start + i,
          payload: 'x'.repeat(payloadBytes),
        })),
        nextToken: end < totalRows ? String(end) : null,
      });
      const bytes = Buffer.byteLength(body);
      largestPageBytes = Math.max(largestPageBytes, bytes);
      requests++;
      assert.ok(bytes < 64 * MiB);
      return new Response(body, {
        headers: { 'content-type': 'application/json', 'content-length': String(bytes) },
      });
    },
  });
  const result = await paginate(({ pageSize, cursor }) =>
    client.get('/rows', { query: { pageSize, cursor } }), flags);
  const retainedPayloadBytes = result.items.reduce((sum, row) =>
    sum + Buffer.byteLength(row.payload), 0);
  const observation = {
    name, requests, largestPageBytes, retainedRows: result.items.length,
    retainedPayloadMiB: retainedPayloadBytes / MiB,
    firstId: result.items[0]?.id, nextToken: result.nextToken,
  };
  console.log(JSON.stringify(observation));
  return observation;
}

const all = await scenario('default accumulation');
assert.equal(all.requests, 12);
assert.equal(all.retainedRows, 300);
assert.equal(all.retainedPayloadMiB, 75);
assert.equal(all.nextToken, null);

const capped = await scenario('explicit maxItems control', { maxItems: 50 });
assert.equal(capped.requests, 2);
assert.equal(capped.retainedRows, 50);
assert.equal(capped.nextToken, '50');

const resumed = await scenario('cursor resume control', {
  maxItems: 50, startingToken: capped.nextToken,
});
assert.equal(resumed.firstId, 50);
assert.equal(resumed.nextToken, '100');

const oversize = new HttpClient({
  baseUrl: 'https://mock.invalid', apiKey: 'sk-user-dummy', env: {},
  fetchImpl: async () => new Response('{}', {
    headers: { 'content-length': String(65 * MiB) },
  }),
});
await assert.rejects(oversize.get('/rows'), error => error.code === 'PAYLOAD_TOO_LARGE');
console.log(JSON.stringify({ name: 'individual response cap control', code: 'PAYLOAD_TOO_LARGE' }));

Observed locally (all assertions passed):

Largest individual page: 6,554,229 bytes, below the existing 64 MiB response cap. The separate 65 MiB declared-response control correctly raised PAYLOAD_TOO_LARGE. These are deterministic retained string-content counts, not a process RSS/heap measurement. The diagnostic intentionally stays finite and does not attempt to exhaust memory.

Proposed scope, pending triage

For a focused implementation, I propose a documented, test-injectable aggregate byte budget for buffered pagination, with the default threshold agreed during triage. If streaming is preferred, I will follow that direction instead.

• Preserve current JSON/text success shapes, ordering, explicit --max-items, single-page selection, and cursor semantics below the budget.
• Count retained content incrementally; do not stringify the entire accumulated array on each page or introduce quadratic bookkeeping.
• On reaching the budget, stop fetching and emit a typed error with a stable reason and actionable --max-items / --page-size guidance. Do not silently truncate, claim success, or expose a cursor that skips undisplayed records.
• Keep repeated-cursor, empty-but-continuable-page, page-count, and interrupt behavior covered. Include boundary tests for the chosen budget and resume behavior under an explicit user limit.
• No changes to server pagination semantics, no new output formats, and no new dependency.

I checked the open/closed issue and PR searches for pagination, accumulation, and heap. #281 implements the already-merged individual HTTP response cap. #164 / PR #278 concern CSV/NDJSON output; this proposal does not take over those formats or their assigned work, and their current diff does not modify pagination.ts. #299 / PR #48 cover cursor-loop/page-count safety, not accumulated payload size.

AI assistance disclosure: Codex assisted with source inspection, duplicate checking, and the local diagnostic. No cloud tests, credits, real credentials, or production data were used. No fix implementation or PR has been submitted; I will wait for acceptance and assignment before proceeding.

Submitted via GitHub CLI using the Hackathon Improvement fields. Please add hackathon / needs-triage if the labels are not applied automatically.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.