test(e2e): model GraphQL in the fake gh and fail on unrecognised input - #51
Open
MiniGod wants to merge 2 commits into
Open
test(e2e): model GraphQL in the fake gh and fail on unrecognised input#51MiniGod wants to merge 2 commits into
MiniGod wants to merge 2 commits into
Conversation
The fake ended with `process.exit(0)` for any subcommand it did not implement. That is indistinguishable from success: `gh()` in src/github.ts resolves empty stdout to `null` rather than throwing, so an unimplemented call returned "no data" and the backend carried on. A GraphQL query the fake had never seen would have left the suite green while the code under test received nothing at all. Every unhandled path now exits non-zero naming what it did not recognise, and `gh api graphql` is implemented: routed by operation name (not substring, which picks the wrong entry when queries share a fragment), with array fixtures acting as pagination sequences that fail on overrun rather than re-serving the last page into an infinite `hasNextPage` loop. An `errors` payload is served the way the real CLI serves one — body on stdout, `gh: <message>` on stderr, exit 1 — which is what makes the missing-`read:project` path testable without a token that can reach the real API. That contract was verified against the live API, including the partial-data-with-errors case. No src/ changes. Only four `gh` argv shapes exist in the tree, all in src/github.ts, so making the fallthrough strict breaks nothing; the new spec pins all four so a regression there fails where the cause is obvious.
… understand The fake ended with a bare `process.exit(0)` for anything it did not implement, which is indistinguishable from success: gh() in src/github.ts resolves empty stdout to null rather than throwing, so an unimplemented call looked like a successful call that returned nothing. Model `gh api graphql`, routed by operation name, with array fixtures as paginated sequences and real-CLI error semantics (body on stdout, `gh:` on stderr, exit 1 — including the partial-data case). Make every unrecognised input fail and name itself: unknown subcommands, unknown flags, stray positionals, a missing --repo, a malformed -f pair, an anonymous or ambiguous query. Take the pagination ticket with an exclusive mkdir rather than an append plus a stat: the append is atomic but the size read is a second syscall, so two concurrent calls get the same index. Write output with fs.writeSync rather than process.stdout.write, which is async on a pipe and loses everything past ~8 KB when process.exit() follows it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of five PRs adding GitHub Projects support (from the review of #49). This
one is test infrastructure only — no
src/change, no new behaviour in theapp. It exists so the four PRs after it can be trusted when they go green.
Why this is its own PR
The fake
ghine2e/fakebin/ghended with a bareprocess.exit(0)foranything it did not implement. That is the most dangerous shape a fake can have,
because it is indistinguishable from success:
Empty stdout resolves to
nullrather than throwing. So a call the fake hadnever heard of looked exactly like a successful call that returned nothing, and
the suite stayed green while the code under test received no data at all. Every
GraphQL query in PRs 2–5 would have landed on that path.
Fixing it first, on its own, means the next PR's tests prove something.
What changed
e2e/fakebin/gh— rewritten.gh api graphqlis modelled, keyed by the query's operation name, notby a substring of the query text. Substring matching picks the wrong entry
when two queries share a fragment and answers nothing useful when a test fails.
Naming the operation is a one-word requirement on production queries.
successive elements, which is how a paginated fetch is modelled. Running off
the end fails rather than repeating the last page: repeating it is exactly
the shape that spins a
hasNextPageloop forever, and a hung test is a worsesignal than a failed one.
errorspayload is served the way the real CLI serves one: body onstdout,
gh: <first message>on stderr, exit 1 — including the partial-datacase where
datais populated anderrorsis present. Verified against thelive API twice. That contract is what lets PRs 2–5 drive the
read:projectfailure path without a token that has the scope.
flags, stray positionals, a missing
--repo, a malformed-fpair, ananonymous query, a multi-operation document with no
operationName.Flags are the subtle half: real
gh api --paginateemits one JSON object perpage (
JSON.parsethrows on that) and--jq/--templateemit a transformedvalue. A fake that ignored those would answer with a body shape production
never sees — the original bug moved down one level rather than fixed.
--repo xand--repo=x) are parsed. Supporting only theseparated form is how the old
flag()returnedundefinedfor--repo=slugand answered with an empty fixture, silently.
e2e/harness/mockEnv.ts—setGhStateclears the GraphQL sequence counter,so a re-seed mid-test restarts a sequence instead of resuming the old call count.
e2e/fakebin.spec.ts— new, 32 tests. A test double does not normally getits own spec; this one earns one because its failure mode is invisible. Each
test names the failure it prevents.
Two things worth reading the diff for
The counter is a
mkdirticket, not a counter.src/github.tsfans itsfetches out with
Promise.all, so the same operation can genuinely be in flighttwice at once, in two separate processes. Increment-then-read-back does not
work — not a read-modify-write on a shared JSON file, and not an atomic append
followed by a
stateither, which is what the first version of this PR did. Theappend is atomic; the size read is a second syscall with nothing tying it to
the first, so
A.append → B.append → A.stat → B.stathands both processes thesame index.
mkdirclaims and reports in one syscall: it either creates thedirectory or fails
EEXIST.Measured: the append-then-
statversion loses a ticket in ~3 of 8 16-way rounds.The test therefore runs eight rounds in one attempt — at one round it would
have surfaced as flaky, which Playwright retries away, rather than as broken.
Output is written with
fs.writeSync, notprocess.stdout.write.process.stdout.writeis asynchronous when stdout is a pipe — which is whatevery spawned
ghgets — andprocess.exit()does not drain it. Anything pastwhat the reader has already taken is silently lost: measured at 8192 bytes
delivered out of 200017.
JSON.parsethrows on the fragment,ghSafe()swallows the throw, and the app sees
[]— precisely the silent-empty failurethis PR exists to remove, reintroduced as a function of payload size. A GraphQL
page of 100 project items is well past the threshold, and every other fixture in
the suite is under 100 bytes, so nothing else could have seen it.
This one was pre-existing on
master; PR 2 is what would have made itload-bearing.
Limits, stated so a green suite does not imply more than it proves
ghSafe()swallows every rejection into[], andissue list/pr listboth go through it. For those two, a rejection here still reaches the app as
"succeeded, nothing there". The strictness is visible to the test (via the
call log and stderr) but not to the app. Only
gh api user, viagh(),propagates. New code that wants a failed fetch to be distinguishable from an
empty one must call
gh().issue list --author <login>is accepted but ignored for fixtureselection, so the owned/not-owned issue split (
src/github.ts:359) cannot beexpressed by a fixture. This PR does not close that gap.
issue list --searchis rejected: production never sends one, and awhitelisted-but-unread flag is exactly how a fixture gets silently ignored.
Blocked on token scope
Nothing in this PR is blocked. Listing the one item here so it can be cleared in
a single pass before PR 2:
read:project(read-only; not the read-writeprojectscope) is neededon the
ghtoken to confirm thatProjectV2.items(query:)accepts the samefilter syntax as
ProjectV2View.filter. The whole planned selection set forPR 2 was pre-validated against the live schema without the scope — 33 errors,
all
INSUFFICIENT_SCOPES, zero schema errors, which proves every fieldexists and is correctly typed, because GraphQL validates the document before it
authorises. That last hop is the one thing validation cannot answer. Per the
brief it is dropped from PR 2 rather than built on as a maybe; if the scope
arrives it goes back in.
One related thing already settled by testing rather than guessing: real
gh api graphqlexits 1 on partial-data-with-errors. So the "aliasuser()and
organization()in one query and see which one comes back" owner-resolutiontrick cannot work through
gh— the owner type has to be resolved first.PR 2 is written that way.
Adjacent, deliberately not fixed here
e2e/fakebin/tmuxwritescapture-paneoutput through the same asyncprocess.stdout.write+process.exit()pattern, so a pane capture larger thanthe pipe buffer would truncate the same way. Today's capture fixtures are far
under it, so nothing is broken — but it is the same latent bug, and
src/tmux.tsis being split by another session right now. Filing it rather thanwidening this diff into a file someone else is holding.
Verification
bun run lint·bun run typecheck·bun run typecheck:e2e·bun run test(31) ·
bun run test:e2e— all green. No oxlint threshold moved (nothing undersrc/**changed).Both fixes above were teeth-checked by reverting them and confirming the new
tests go red, not merely by confirming they pass.
Two adversarial review rounds by fresh sub-agents that did not write the code:
round 1 found 14 confirmed issues, round 2 found 8. All fixed.