fix(validate): retry a 429 in the link check instead of accepting it - #10
Conversation
c1d1eb6 to
862af68
Compare
7d52116 to
4b327eb
Compare
862af68 to
a1baab6
Compare
The link check reported two different verdicts for the same tree three seconds
apart this morning:
04:48:39 WARN ... xpu-installation.html was not reachable (HTTP 429) pass
04:48:42 FAIL ... xpu-installation.html is gone (HTTP 404) fail
Both runs were dependabot pull requests over the same skills. The first one's probe
was throttled by docs.vllm.ai, and rate limiting maps to 'unknown', which warns.
Mapping rate limiting to a warning is right — the alternative is a gate that fails a
contributor's pull request because someone else's CDN was busy. What is wrong is
accepting it on the first try. Every other 'unknown' is an answer *alongside* the
real one: a timeout or a 5xx says nothing about whether the page exists, and there
is nothing better to do than say so. A 429 is an answer given *instead of* the real
one, and here it stood in for HTTP 404 — the throttled run reported a dead link as
reachable and went green.
So a 429 is now retried with backoff before it is allowed to become a warning:
`Retry-After` when the host sends it in seconds, exponential otherwise, capped at 8s
per wait and 3 retries. The HTTP-date form of `Retry-After` is deliberately not
honoured — a host that says "come back in four minutes" is asking for longer than a
CI step should hold a pull request open, and the warning is the better answer there.
Worst case adds ~7s for one thoroughly throttled URL, in parallel across workers.
Why this host, and why now: eight workers over 99 URLs, a dozen of them on two
documentation domains, is enough to trip a per-IP limit on a shared runner. Nothing
about the repository changed for it to start happening, and nothing has to change
for it to happen again.
Checked against a stubbed opener, since a live 429 cannot be summoned on demand:
429 every time -> ('unknown', 'HTTP 429 after 4 attempts'), 4 attempts
429 then 404 -> ('dead', 'HTTP 404') <- the case that went green
HEAD 403 then GET 200 -> ('ok', '200') <- unchanged
Retry-After absent -> 1, 2, 4, 8, 8, 8 (capped)
Retry-After as a date -> falls back to exponential
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
a1baab6 to
6865eaf
Compare
xaleryb
left a comment
There was a problem hiding this comment.
Reviewed by testing rather than by reading: probe_url was driven through 15 scenarios
against a stubbed opener, and the same suite was run against main's version as a
baseline. Every non-429 row is identical between the two — 404, 410, 500, HEAD 403 → GET 200, HEAD 403 → GET 403, timeout and DNS failure all keep their previous verdict
and their previous number of requests — so the change is strictly scoped to 429.
The three claims that matter reproduce exactly as described:
| stub | verdict | requests | waits |
|---|---|---|---|
| 429 then 404 | dead, HTTP 404 |
2 | 1s |
| 429 forever | unknown, HTTP 429 after 4 attempts |
4 | 1, 2, 4 |
Retry-After: 30 |
capped | 4 | 8, 8, 8 |
Retry-After: HTTP-date |
exponential | 4 | 1, 2, 4 |
The retry can only turn a warning into a failure for a link this repository wrote itself:
a 429 that resolves to a 404 in an imported body still lands in the dead +
upstream_body branch of check_links, which warns. That property is what makes
retrying safe to do unconditionally.
One follow-up, not a blocker. retry_after accepts NaN: float("NaN") parses, and NaN
survives both max and min, so Retry-After: NaN reaches time.sleep(nan), which
raises ValueError: Invalid value NaN. It is uncaught, and it happens inside a
ThreadPoolExecutor worker, so pool.map re-raises it and the step ends in a traceback —
losing the warnings and errors the eight preceding checks had already collected, since
those are printed after check_links returns. No conforming server sends that header, but
the value is usually computed, and String(x) over an arithmetic result on undefined is
a well-known way to emit the literal string NaN. A math.isfinite check in the existing
except (TypeError, ValueError) fallback closes it with no behaviour change on any valid
input; verified against the same suite. Will be sent as a separate change.
3 of 5 in a split stack.
mainmain#9 and #12 are independent and both go straight to
main— they fix the two unrelated jobsthat are red there, and neither waits on the other. #10, #11 and #13 sit on #9 only so their
Validate runs are green while
main's link check is failing; GitHub retargets them tomainwhen #9 lands.
mainis failing two independent jobs, so any PR fixing one still displays the other.Nothing here introduces the failure it shows:
zizmor— the Pages workflow from #7validate— the dead docs.vllm.ai linkEverything else is green on every PR. All five test-merge into
maincleanly in any order,and the merged combination passes the full gate plus zizmor and actionlint.
What this changes
The link check reported two different verdicts for the same tree three seconds apart this
morning, on two dependabot pull requests over identical skills:
The first run's probe was throttled by docs.vllm.ai. Rate limiting maps to
unknown, whichwarns, so it went green — reporting a dead link as reachable.
Mapping rate limiting to a warning is right; the alternative is failing a contributor's pull
request because someone else's CDN was busy. What is wrong is accepting it on the first
try. Every other
unknownis an answer given alongside the real one — a timeout or a5xx tells you nothing about whether the page exists, and there is nothing better to do than
say so. A 429 is an answer given instead of the real one, and here it stood in for a 404.
So a 429 is now retried before it is allowed to become a warning:
Retry-Afterwhen thehost sends it in seconds, exponential otherwise, capped at 8s per wait and 3 retries. Worst
case adds ~7s for one thoroughly throttled URL, in parallel across workers.
The HTTP-date form of
Retry-Afteris deliberately not honoured. A host saying "comeback in four minutes" is asking for longer than a CI step should hold a pull request open,
and the warning is the better answer there.
Why this host and why now: 8 workers over 99 URLs, a dozen of them on two documentation
domains, is enough to trip a per-IP limit on a shared runner. Nothing about the repository
changed for it to start happening, and nothing has to change for it to happen again.
Verified
A live 429 cannot be summoned on demand, so this was checked against a stubbed opener:
('unknown', 'HTTP 429 after 4 attempts'), 4 attempts made('dead', 'HTTP 404')← the case that went green this morning('ok', '200')— existing fallback unchangedRetry-AfterRetry-Afteras an HTTP dateChecklist
descriptionagainst requests a user would really type — see CONTRIBUTING.md. (no skill text changed)python3 tools/validate_skills.pypasses locally.git commit -s(DCO).