Skip to content

Add a deadline-aware poll helper and deprecate IntervalRetry - #1594

Draft
cristianrgreco wants to merge 4 commits into
mainfrom
claude/deadline-aware-poll-helper-e07af6
Draft

cristianrgreco wants to merge 4 commits into
mainfrom
claude/deadline-aware-poll-helper-e07af6

Conversation

@cristianrgreco

@cristianrgreco cristianrgreco commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

IntervalRetry only checked its timeout between attempts, so a hung attempt (for example container.exec in the shell wait strategy) blocked past the startup timeout. A throw from an attempt escaped the loop, so callers hand-rolled try/catch returning undefined and the timeout error never carried the last failure.

This replaces the Retry / AbstractRetry / Clock scaffolding with one exported function:

poll(fn, { until?, intervalMs, timeoutMs, timeoutMessage?, retryOnError? })
  • fn receives { attempt, signal }. signal aborts at the deadline.
  • The wait rejects at the deadline even when an attempt is still in flight.
  • The timeout error's cause is the last attempt's error, if that attempt threw.
  • A throw from fn is retried by default. retryOnError: false rethrows it immediately.

It is exported because modules can only import core through the testcontainers entry point. It is a helper for modules, so it has no page in docs/. Every IntervalRetry call site in core, Couchbase and the MongoDB Atlas Local test now uses it. IntervalRetry and Retry stay exported with their original loop, marked @deprecated.

The signal is threaded through the Docker calls the waits make, so nothing keeps running on our side after a timeout:

  • ContainerClient.exec takes signal in its options and destroys the exec stream on abort.
  • ContainerClient.inspect takes an optional { signal }.
  • StartedTestContainer.exec(cmd, { signal }) is supported and documented.
  • StartupCheckStrategy.checkStartupState is passed the signal as an optional third argument.
  • The shell, health check, startup check, host port and internal port waits, the ports-bound wait and the HTTP wait's abortOnContainerExit inspect all pass it on.

Other implementations this follows:

Behaviour changes

  • Timeouts are hard deadlines. An attempt still running at the deadline no longer gets to finish and succeed. This includes the internal 30s ports-bound wait.
  • Five core waits and the Couchbase KV readiness wait pass retryOnError: false (health check, host port, shell, startup check, ports-bound, mcstat) so a Docker error still fails fast. Measured: a container that exits after 2s fails the port or shell wait in about 3s with the 409 "is not running" error, before and after.
  • A throw from until propagates. Wait.forHttp(...).forResponsePredicate still fails immediately when the predicate throws.
  • HTTP wait: the poll interval is still readTimeoutMs. The in-flight request is cancelled at the deadline and the timeout error carries a cause.
  • Reaper: four connection attempts fit in the 4s window instead of five, and the socket is destroyed at the deadline.
  • Logs: the "log then throw" lines in the wait strategies, the Reaper and Couchbase are removed; waitForContainer already logs the same message with the container id. Aborted exec and inspect calls no longer log "Failed to ...".
  • MongoDB Atlas Local test: the index-ready wait now fails the test on timeout. Before, its returned Error was ignored. Measured locally: index ready in about 0.3s and the search result in about 1s, against 5s limits.
  • Timeouts above 2^31-1 ms are clamped, because setTimeout would otherwise fire after 1ms.

What still runs after a timeout

  • The process an aborted exec started inside the container. Docker has no API to kill an exec. Measured with exec(["sleep", "30"], { signal }) aborted after 1s: exec rejected at 1008ms, docker top still listed sleep 30 immediately after and 5s after, and it was gone at 32s. For wait strategies, waitForContainer stops and removes the container on failure, which ends it. For a user's own exec, it runs until it exits or the container stops. This is documented.
  • Couchbase's fetch calls. The Couchbase migration is mechanical (same intervals, timeouts and messages) and does not pass the signal to doHttpRequest. The KV readiness wait does pass it to container.exec.
  • A user's checkStartupState that ignores the signal.

Overlap with other PRs

Merged since this was opened, and resolved in the merges of main (7cc2a64, 9d68601):

Still open:

Verification

npm run format, npm run lint, npm run knip, npm run check-compiles and mkdocs build --strict pass. The first four were rerun after merging main.

Red-green, each test run before and after the change it covers:

Test Before After
poll rejects when an attempt never resolves (old loop under the new signature) timed out at 1000ms passes
inspectContainerUntilPortsExposed with an inspect that never resolves (untouched main) timed out at 1000ms passes
Wait.forSuccessfulCommand("sleep 30") with a 1000ms startup timeout (pre-change strategy) start resolved after 30.2s rejects at the timeout
poll with timeoutMs: Infinity (without the clamp) Timed out after Infinityms passes
client.exec rejects and destroys the stream on abort timed out at 5000ms passes
client.exec creates no exec for an already-aborted signal timed out at 5000ms passes
client.inspect rejects on abort timed out at 5000ms passes
container.exec(["sleep", "30"], { signal }) against Docker resolved after 30.7s rejects, 1.5s
Ports-bound wait aborts its inspect at the timeout TypeError, no signal passed passes
Health check, shell, listening ports and HTTP waits abort their Docker call TypeError / no signal received passes
checkStartupState receives a signal that aborts at the timeout expected false to be true passes

Targeted suites, run locally against Docker before merging main:

  • Wait strategies, reaper, retry, generic container, container client: 28 of 29 files pass (211 of 213 tests, 1 skipped). generic-container-restart.test.ts fails intermittently with fetch failed / ECONNRESET when run in this set. It fails the same way on untouched main in 2 of 3 runs of the same set, and passes alone 3 of 3 on this branch.
  • MongoDB Atlas Local: 6 of 6 pass.
  • Couchbase: flaky locally on both sides. 9 of 19 runs were fully green on this branch and 4 of 9 on main. The most common failure on both is Could not create primary index ... rebalance in progress (7 of 19 runs here, 4 of 9 on main). When the Docker VM (3.8 GB) was overloaded, this branch failed with URL /pools not accessible after 60000ms (4 runs) or the 30s ports-bound timeout (3 runs), while main failed with Could not perform request against couchbase HTTP endpoint (4 runs). I could not reproduce the ports-bound timeout under instrumentation in 6 further runs. A local-server comparison of the HTTP wait (connection resets, hangs, 503s, then 200) gave identical request counts and timings on both sides.

After merging main (7cc2a64), which brings in #1514:

  • Wait strategies, container client, generic-container.test.ts and Couchbase: 17 files, 145 passed, 1 skipped.
  • Couchbase again on its own: 11 of 11 pass.

After the second merge of main (9d68601), format, lint, knip and check-compiles pass, and one run of the retry, container client, ports-bound, generic-container.test.ts, wait strategy, reaper, Couchbase and MongoDB Atlas Local tests gave 21 files, 186 passed, 1 skipped.

Publish dry run for v12.2.1, succeeded on 6596e1d (before the merge of main): https://github.com/testcontainers/testcontainers-node/actions/runs/38295782401

Not breaking

  • IntervalRetry and Retry keep their signatures and loop. Their protected clock, hasTimedOut and wait members are gone; they were only reachable by subclassing.
  • poll, PollOptions, ExecOptions.signal, the inspect options and the third checkStartupState argument are additive.
  • Timeout error messages are unchanged.

Replace the IntervalRetry class hierarchy with one exported function,
poll(fn, { until, intervalMs, timeoutMs, timeoutMessage, retryOnError }).
It rejects at the deadline even when an attempt is still in flight, passes
fn an AbortSignal that aborts at the deadline, and attaches the last
attempt's error to the timeout error as its cause.

Migrate every IntervalRetry call site in core and in the Couchbase and
MongoDB modules. IntervalRetry and Retry stay exported with their original
behaviour, marked deprecated.

Thread the signal through ContainerClient.exec and ContainerClient.inspect
so wait strategies cancel their in-flight Docker calls at the startup
timeout, and expose it as a signal option on StartedTestContainer.exec.
@cristianrgreco cristianrgreco added enhancement New feature or request minor Backward compatible functionality labels Oct 12, 2026
Resolve conflicts with #1514 and #1566: keep the remaining-startup-budget timeouts from main on the Couchbase polls, drop the removed streamToString export, and move waitUntilKvAcceptsCredentials to poll, passing its signal to container.exec.
poll is exported so modules can import it, not as a user-facing feature.
Resolve conflicts with the Biome rule changes in #1585 and #1590: thread the signal through the extracted exec and HTTP helpers, and move the extracted Couchbase waits to poll.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request minor Backward compatible functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant