Measure retry delays with monotonic clocks in the Python and Ruby conformance runners - #530
Merged
Merged
Conversation
…ners The delayBetweenRequests assertions compared wall-clock captures (time.time() / Time.now) against exact minimum-delay bounds. Wall clocks can step or slew mid-test (NTP), so a real sleep of min_delay can read marginally short and flake — the same class #496 hit in the TypeScript runner, patched there with a TIMER_SLACK_MS tolerance (058fd11). Fix it the way Go already avoids it (main.go uses time.Time subtraction, which is monotonic in Go): capture time.monotonic() in Python and Process.clock_gettime(Process::CLOCK_MONOTONIC) in Ruby. Both SDKs' retry sleeps wait against the monotonic clock, so measuring on the same clock makes the elapsed delta a true lower bound — no slack constant needed. The captured timestamp's only consumer in both runners is the delay-delta math (nothing reports it as an absolute time), so the field swaps wholesale and is renamed monotonic_time to keep it from being mistaken for a wall-clock reading.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Member
Author
|
Bot-review status: Copilot errored on its review attempt and a re-request could not be resolved via API; Codex is at its usage limit today. Zero review threads. Merging on green CI per the standing flaky-reviewer convention. |
This was referenced Jul 31, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flake class
delayBetweenRequestsassertions compare captured request timestamps against an exact minimum-delay bound. When the capture uses a wall clock, any NTP step or slew between two captures can make a real sleep ofmin_delayread marginally short, failing the assertion — a nondeterministic flake unrelated to SDK behavior.#496 reported exactly this in the TypeScript runner; 058fd11 (#488) patched it there with a
TIMER_SLACK_MS = 2tolerance (a libuv early-fire workaround). The same class remained latent in two other runners:conformance/runner/python/runner.py): capturedtime.time(), assertedd < min_delayconformance/runner/ruby/runner.rb): capturedTime.now, same exact boundThe fix: Go's approach, not TS's
The Go runner never had this problem —
main.gosubtractstime.Timevalues, which is monotonic in Go. This PR applies the same principle rather than copying the slack-constant hack:time.monotonic()Process.clock_gettime(Process::CLOCK_MONOTONIC)Both SDKs' retry sleeps wait against the monotonic clock, so measuring on the same clock makes the elapsed delta a true lower bound on the sleep — no tolerance constant needed, and the assertion stays exact.
Consumer audit
The captured timestamp's only consumer in both runners is the delay-delta math (
delays_between_requests). Nothing exports or reports it as an absolute time — other assertions readmethod/url/headers/body, and reports carry name/pass/fail only. So the field swaps wholesale, renamedtime→monotonic_timeso it can't be mistaken for a wall-clock reading. Ruby's now-unusedrequire "time"is dropped.Replay runners capture no timestamps; Kotlin/Swift runners don't implement
delayBetweenRequests. Go and TS are untouched.Evidence
Both runners green with the change, exit codes pinned:
make conformance-python→REAL_EXIT=0, 0 failedmake conformance-ruby→REAL_EXIT=0, 0 failedThe delay-assertion fixtures (
GET operation retries on 503,GET operation retries on 429 with Retry-After) PASS through the monotonic path in both; the two DownloadURL delay fixtures remain pre-existing intentional skips in both runners.Closes the Python/Ruby residual of #496 (TS half already fixed by 058fd11).
Summary by cubic
Switch Python and Ruby conformance runners to monotonic clocks for measuring retry delays to eliminate flaky failures from wall-clock adjustments. Keeps exact minimum-delay assertions without adding tolerance.
time.time()withtime.monotonic()and compute delays frommonotonic_time.Time.nowwithProcess.clock_gettime(Process::CLOCK_MONOTONIC)and compute delays frommonotonic_time.time→monotonic_time; remove unusedrequire "time"in Ruby.Written for commit 22b1241. Summary will update on new commits.