Skip to content

test(e2e): model GraphQL in the fake gh and fail on unrecognised input - #51

Open
MiniGod wants to merge 2 commits into
masterfrom
feat/fake-gh-graphql
Open

test(e2e): model GraphQL in the fake gh and fail on unrecognised input#51
MiniGod wants to merge 2 commits into
masterfrom
feat/fake-gh-graphql

Conversation

@MiniGod

@MiniGod MiniGod commented Aug 21, 2026

Copy link
Copy Markdown
Owner

First of five PRs adding GitHub Projects support (from the review of #49). This
one is test infrastructure only — no src/ change, no new behaviour in the
app. It exists so the four PRs after it can be trusted when they go green.

Why this is its own PR

The fake gh in e2e/fakebin/gh ended with a bare process.exit(0) for
anything it did not implement. That is the most dangerous shape a fake can have,
because it is indistinguishable from success:

// src/github.ts:55
resolve(out.trim() ? JSON.parse(out) : null);

Empty stdout resolves to null rather than throwing. So a call the fake had
never heard of looked exactly like a successful call that returned nothing, and
the suite stayed green while the code under test received no data at all. Every
GraphQL query in PRs 2–5 would have landed on that path.

Fixing it first, on its own, means the next PR's tests prove something.

What changed

e2e/fakebin/gh — rewritten.

  • gh api graphql is modelled, keyed by the query's operation name, not
    by a substring of the query text. Substring matching picks the wrong entry
    when two queries share a fragment and answers nothing useful when a test fails.
    Naming the operation is a one-word requirement on production queries.
  • An array fixture is a sequence — successive calls to one operation return
    successive elements, which is how a paginated fetch is modelled. Running off
    the end fails rather than repeating the last page: repeating it is exactly
    the shape that spins a hasNextPage loop forever, and a hung test is a worse
    signal than a failed one.
  • An errors payload is served the way the real CLI serves one: body on
    stdout, gh: <first message> on stderr, exit 1 — including the partial-data
    case where data is populated and errors is present. Verified against the
    live API twice. That contract is what lets PRs 2–5 drive the read:project
    failure path without a token that has the scope.
  • Unrecognised input fails, naming itself — unknown subcommands, unknown
    flags, stray positionals, a missing --repo, a malformed -f pair, an
    anonymous query, a multi-operation document with no operationName.
    Flags are the subtle half: real gh api --paginate emits one JSON object per
    page
    (JSON.parse throws on that) and --jq/--template emit a transformed
    value. A fake that ignored those would answer with a body shape production
    never sees — the original bug moved down one level rather than fixed.
  • Both flag forms (--repo x and --repo=x) are parsed. Supporting only the
    separated form is how the old flag() returned undefined for --repo=slug
    and answered with an empty fixture, silently.

e2e/harness/mockEnv.tssetGhState clears the GraphQL sequence counter,
so a re-seed mid-test restarts a sequence instead of resuming the old call count.

e2e/fakebin.spec.ts — new, 32 tests. A test double does not normally get
its own spec; this one earns one because its failure mode is invisible. Each
test names the failure it prevents.

Two things worth reading the diff for

The counter is a mkdir ticket, not a counter. src/github.ts fans its
fetches out with Promise.all, so the same operation can genuinely be in flight
twice at once, in two separate processes. Increment-then-read-back does not
work — not a read-modify-write on a shared JSON file, and not an atomic append
followed by a stat either, which is what the first version of this PR did. The
append is atomic; the size read is a second syscall with nothing tying it to
the first, so A.append → B.append → A.stat → B.stat hands both processes the
same index. mkdir claims and reports in one syscall: it either creates the
directory or fails EEXIST.

Measured: the append-then-stat version loses a ticket in ~3 of 8 16-way rounds.
The test therefore runs eight rounds in one attempt — at one round it would
have surfaced as flaky, which Playwright retries away, rather than as broken.

Output is written with fs.writeSync, not process.stdout.write.
process.stdout.write is asynchronous when stdout is a pipe — which is what
every spawned gh gets — and process.exit() does not drain it. Anything past
what the reader has already taken is silently lost: measured at 8192 bytes
delivered out of 200017
. JSON.parse throws on the fragment, ghSafe()
swallows the throw, and the app sees [] — precisely the silent-empty failure
this PR exists to remove, reintroduced as a function of payload size. A GraphQL
page of 100 project items is well past the threshold, and every other fixture in
the suite is under 100 bytes, so nothing else could have seen it.

This one was pre-existing on master; PR 2 is what would have made it
load-bearing.

Limits, stated so a green suite does not imply more than it proves

  • ghSafe() swallows every rejection into [], and issue list / pr list
    both go through it. For those two, a rejection here still reaches the app as
    "succeeded, nothing there". The strictness is visible to the test (via the
    call log and stderr) but not to the app. Only gh api user, via gh(),
    propagates. New code that wants a failed fetch to be distinguishable from an
    empty one must call gh().
  • issue list --author <login> is accepted but ignored for fixture
    selection
    , so the owned/not-owned issue split (src/github.ts:359) cannot be
    expressed by a fixture. This PR does not close that gap.
  • issue list --search is rejected: production never sends one, and a
    whitelisted-but-unread flag is exactly how a fixture gets silently ignored.

Blocked on token scope

Nothing in this PR is blocked. Listing the one item here so it can be cleared in
a single pass before PR 2:

  • read:project (read-only; not the read-write project scope) is needed
    on the gh token to confirm that ProjectV2.items(query:) accepts the same
    filter syntax as ProjectV2View.filter
    . The whole planned selection set for
    PR 2 was pre-validated against the live schema without the scope — 33 errors,
    all INSUFFICIENT_SCOPES, zero schema errors, which proves every field
    exists and is correctly typed, because GraphQL validates the document before it
    authorises. That last hop is the one thing validation cannot answer. Per the
    brief it is dropped from PR 2 rather than built on as a maybe; if the scope
    arrives it goes back in.

One related thing already settled by testing rather than guessing: real
gh api graphql exits 1 on partial-data-with-errors. So the "alias user()
and organization() in one query and see which one comes back" owner-resolution
trick cannot work through gh — the owner type has to be resolved first.
PR 2 is written that way.

Adjacent, deliberately not fixed here

e2e/fakebin/tmux writes capture-pane output through the same async
process.stdout.write + process.exit() pattern, so a pane capture larger than
the pipe buffer would truncate the same way. Today's capture fixtures are far
under it, so nothing is broken — but it is the same latent bug, and
src/tmux.ts is being split by another session right now. Filing it rather than
widening this diff into a file someone else is holding.

Verification

bun run lint · bun run typecheck · bun run typecheck:e2e · bun run test
(31) · bun run test:e2e — all green. No oxlint threshold moved (nothing under
src/** changed).

Both fixes above were teeth-checked by reverting them and confirming the new
tests go red, not merely by confirming they pass.

Two adversarial review rounds by fresh sub-agents that did not write the code:
round 1 found 14 confirmed issues, round 2 found 8. All fixed.

The fake ended with `process.exit(0)` for any subcommand it did not
implement. That is indistinguishable from success: `gh()` in
src/github.ts resolves empty stdout to `null` rather than throwing, so an
unimplemented call returned "no data" and the backend carried on. A
GraphQL query the fake had never seen would have left the suite green
while the code under test received nothing at all.

Every unhandled path now exits non-zero naming what it did not
recognise, and `gh api graphql` is implemented: routed by operation name
(not substring, which picks the wrong entry when queries share a
fragment), with array fixtures acting as pagination sequences that fail
on overrun rather than re-serving the last page into an infinite
`hasNextPage` loop. An `errors` payload is served the way the real CLI
serves one — body on stdout, `gh: <message>` on stderr, exit 1 — which
is what makes the missing-`read:project` path testable without a token
that can reach the real API. That contract was verified against the live
API, including the partial-data-with-errors case.

No src/ changes. Only four `gh` argv shapes exist in the tree, all in
src/github.ts, so making the fallthrough strict breaks nothing; the new
spec pins all four so a regression there fails where the cause is
obvious.
… understand

The fake ended with a bare `process.exit(0)` for anything it did not
implement, which is indistinguishable from success: gh() in src/github.ts
resolves empty stdout to null rather than throwing, so an unimplemented
call looked like a successful call that returned nothing.

Model `gh api graphql`, routed by operation name, with array fixtures as
paginated sequences and real-CLI error semantics (body on stdout, `gh:`
on stderr, exit 1 — including the partial-data case). Make every
unrecognised input fail and name itself: unknown subcommands, unknown
flags, stray positionals, a missing --repo, a malformed -f pair, an
anonymous or ambiguous query.

Take the pagination ticket with an exclusive mkdir rather than an append
plus a stat: the append is atomic but the size read is a second syscall,
so two concurrent calls get the same index. Write output with fs.writeSync
rather than process.stdout.write, which is async on a pipe and loses
everything past ~8 KB when process.exit() follows it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant