fix(testing): bound the waits in the new-enemy e2e test

[vroldanbet]

What

Gives every setup write in TestNoNewEnemy a deadline of its own, bounds the
namespacesForNode search loop, and stops leaderFromRangeRow calling
log.Fatal.

Test-only. Skip-Changelog.

Why

E2E (22.1.5) on #3339 failed after 29m53s. It is not a regression from
that PR: a re-run of the identical commit (3f66c1ac) passed in 49.9s, and
the same 29-minute failure has already happened twice on main
(run 30120682812,
run 28831661566),
with the identical signature: --- FAIL: TestNoNewEnemy (1740.0s).

1740s is exactly the budget the test gives itself — the 30-minute go test
timeout minus the one-minute cleanup margin it subtracts in TestNoNewEnemy.
The test does not fail because something went wrong; it fails because it ran
out of clock while waiting, in silence, on a single gRPC call.

From the node logs of the failing run, where the time actually went:

phase	window	duration
CockroachDB start + migrate	18:03:56 – 18:04:30	34s
9 SpiceDB nodes to ready	18:04:32 – 18:04:40	8s
fillSchema, 4000 namespaces	18:04:41 – 18:04:45	4s
namespacesForNode, 57 attempts	18:04:45 – 18:04:49	4s
one stuck WriteSchema	18:04:49 – 18:33:29	~29m

The stall itself is a property of the cluster this test deliberately builds.
setSmallRanges configures 64KiB ranges, num_replicas = 1 and a 10s GC TTL;
after 4000 namespaces, namespace_config's full read inside SpiceDB's
WriteSchema transaction — 17ms on every one of the preceding 75 calls —
steps to 12.48s and stays pinned there:

 75 18:04:48.651      16.8ms
 76 18:04:49.399     687.9ms
 77 18:04:49.661     234.1ms
 78 18:05:02.675   12694.5ms
 79 18:05:15.188   12481.5ms
 ...
 93 18:21:22.268   12479.6ms

A read span that wide cannot be refreshed, so every commit comes back
RETRY_SERIALIZABLE - failed preemptive refresh and SpiceDB retries with
exponential backoff — 20 attempts, spacing out to six minutes apart — until
the client's gRPC connection hits MaxConnectionAge and the RPC dies with
received prior goaway: ... "max_age".

The test cannot prevent that. What it should not do is take 29 minutes of a
large runner to notice, and then report a single confusing line about a gRPC
GOAWAY rather than "this write never completed".

How

• fillWriteTimeout (90s) bounds each WriteSchema/WriteRelationships
  batch. On a healthy cluster these take tens of milliseconds and the slowest
  ever recorded in a passing run is under a second, so the budget is ~100x
  headroom; the point is that it is finite. fillSchema previously used
  context.Background(), honouring neither the suite deadline nor any bound
  of its own.
• namespaceSearchTimeout (5m) bounds namespacesForNode. The loop is
  guess-and-check against random range placement; passing runs have needed
  between 48 and 646 attempts, the latter taking a bit over two minutes.
  It now also fails the test when it gives up, instead of returning a zero
  NamespaceNames that the caller would happily fill relationships into.
• leaderFromRangeRow takes testing.TB and uses require.NoError instead
  of log.Fatal. log.Fatal exits the binary: no test output, no
  t.Cleanup, and so no Stop() for the three CockroachDB nodes and nine
  SpiceDB processes the suite started. Harmless while the only route here was
  a genuinely broken cluster; not harmless now that callers hold contexts that
  can expire.

Net effect: this failure mode becomes a ~2 minute job that says which write
stalled, instead of a 30 minute job that says nothing.

References

Two findings worth recording separately, neither addressed here:

• TestNoNewEnemy is flaky on main, independent of this stack. Three
  1740s timeouts in recent history.
• SpiceDB's WriteSchema reads all of namespace_config inside its
  read-write transaction. Once that table is large enough for the read span
  to exceed CockroachDB's refresh-span budget, a pushed transaction can never
  refresh, so WriteSchema retries forever rather than failing. This test
  reaches that state artificially, but a large real schema on a contended
  cluster is the same shape.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!