Skip to content

Measure retry delays with monotonic clocks in the Python and Ruby conformance runners - #530

Merged
jeremy merged 1 commit into
mainfrom
fix/runner-monotonic-delays
Jul 31, 2026
Merged

jeremy merged 1 commit into
mainfrom
fix/runner-monotonic-delays

Conversation

@jeremy

@jeremy jeremy commented Jul 31, 2026 •

Copy link
Copy Markdown
Member

The flake class

delayBetweenRequests assertions compare captured request timestamps against an exact minimum-delay bound. When the capture uses a wall clock, any NTP step or slew between two captures can make a real sleep of min_delay read marginally short, failing the assertion — a nondeterministic flake unrelated to SDK behavior.

#496 reported exactly this in the TypeScript runner; 058fd11 (#488) patched it there with a TIMER_SLACK_MS = 2 tolerance (a libuv early-fire workaround). The same class remained latent in two other runners:

  • Python (conformance/runner/python/runner.py): captured time.time(), asserted d < min_delay
  • Ruby (conformance/runner/ruby/runner.rb): captured Time.now, same exact bound

The fix: Go's approach, not TS's

The Go runner never had this problem — main.go subtracts time.Time values, which is monotonic in Go. This PR applies the same principle rather than copying the slack-constant hack:

  • Python captures time.monotonic()
  • Ruby captures Process.clock_gettime(Process::CLOCK_MONOTONIC)

Both SDKs' retry sleeps wait against the monotonic clock, so measuring on the same clock makes the elapsed delta a true lower bound on the sleep — no tolerance constant needed, and the assertion stays exact.

Consumer audit

The captured timestamp's only consumer in both runners is the delay-delta math (delays_between_requests). Nothing exports or reports it as an absolute time — other assertions read method/url/headers/body, and reports carry name/pass/fail only. So the field swaps wholesale, renamed time → monotonic_time so it can't be mistaken for a wall-clock reading. Ruby's now-unused require "time" is dropped.

Replay runners capture no timestamps; Kotlin/Swift runners don't implement delayBetweenRequests. Go and TS are untouched.

Evidence

Both runners green with the change, exit codes pinned:

  • make conformance-python → REAL_EXIT=0, 0 failed
  • make conformance-ruby → REAL_EXIT=0, 0 failed

The delay-assertion fixtures (GET operation retries on 503, GET operation retries on 429 with Retry-After) PASS through the monotonic path in both; the two DownloadURL delay fixtures remain pre-existing intentional skips in both runners.

Closes the Python/Ruby residual of #496 (TS half already fixed by 058fd11).


Summary by cubic

Switch Python and Ruby conformance runners to monotonic clocks for measuring retry delays to eliminate flaky failures from wall-clock adjustments. Keeps exact minimum-delay assertions without adding tolerance.

  • Bug Fixes
    • Python: replace time.time() with time.monotonic() and compute delays from monotonic_time.
    • Ruby: replace Time.now with Process.clock_gettime(Process::CLOCK_MONOTONIC) and compute delays from monotonic_time.
    • Rename captured field time → monotonic_time; remove unused require "time" in Ruby.

Written for commit 22b1241. Summary will update on new commits.

Review in cubic

…ners

The delayBetweenRequests assertions compared wall-clock captures
(time.time() / Time.now) against exact minimum-delay bounds. Wall clocks
can step or slew mid-test (NTP), so a real sleep of min_delay can read
marginally short and flake — the same class #496 hit in the TypeScript
runner, patched there with a TIMER_SLACK_MS tolerance (058fd11).

Fix it the way Go already avoids it (main.go uses time.Time subtraction,
which is monotonic in Go): capture time.monotonic() in Python and
Process.clock_gettime(Process::CLOCK_MONOTONIC) in Ruby. Both SDKs'
retry sleeps wait against the monotonic clock, so measuring on the same
clock makes the elapsed delta a true lower bound — no slack constant
needed.

The captured timestamp's only consumer in both runners is the
delay-delta math (nothing reports it as an absolute time), so the field
swaps wholesale and is renamed monotonic_time to keep it from being
mistaken for a wall-clock reading.
Copilot AI review requested due to automatic review settings July 31, 2026 19:54
@jeremy jeremy added the bug Something isn't working label Jul 31, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions github-actions Bot added the conformance Conformance test suite label Jul 31, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@jeremy

jeremy commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

Bot-review status: Copilot errored on its review attempt and a re-request could not be resolved via API; Codex is at its usage limit today. Zero review threads. Merging on green CI per the standing flaky-reviewer convention.

@jeremy
jeremy merged commit 78f8e94 into main Jul 31, 2026
41 of 42 checks passed
@jeremy
jeremy deleted the fix/runner-monotonic-delays branch July 31, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working conformance Conformance test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants