Skip to content

fix(gatewayapi): keep an older translation from overwriting a newer GatewayClass status - #10408

Open
nobbs wants to merge 6 commits into
envoyproxy:mainfrom
nobbs:fix/runner-stale-gatewayclass-status
Open

nobbs wants to merge 6 commits into
envoyproxy:mainfrom
nobbs:fix/runner-stale-gatewayclass-status

Conversation

@nobbs

@nobbs nobbs commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

TestGatewayAPIConformance/GatewayClassObservedGenerationBump is flaky: 117 conformance jobs failed on it between 12 and 26 Oct. Each time, the GatewayClass kept Accepted at the old observedGeneration for the full 60s after its spec changed. Those runs logged no reconcile errors and no failed status writes.

Two writers store a GatewayClass status: the provider when it reconciles the class, and the Gateway API runner from each snapshot it translates. The runner can lag a snapshot behind:

spec patch, generation 1 -> 2
  provider reconcile: stores observedGeneration 2, the status updater writes it
  runner, still on the generation-1 snapshot: stores observedGeneration 1 again, written
  runner, generation-2 snapshot: stores observedGeneration 2
    status updater: its cache still shows the earlier write of 2, "status unchanged", skipped
  the GatewayClass stays at observedGeneration 1

This PR keeps the runner from storing a translated GatewayClass status when the stored status was already computed for a later generation of the same GatewayClass:

 if result.GatewayClass != nil {
-	key := utils.NamespacedName(result.GatewayClass)
-	r.ProviderResources.GatewayClassStatuses.Store(key, &result.GatewayClass.Status)
+	r.storeGatewayClassStatus(result.GatewayClass)
 }
storeGatewayClassStatus(gc)
  loop
    nothing stored:            LoadOrStore, check again if the provider stored first
    stored status is newer:    keep it, done
    CompareAndSwap to gc's:    done, or check again if the provider stored in between
  remember the stored status and gc's UID

stored status is newer
  not when the runner stored it for another UID: the GatewayClass was deleted and created again
  otherwise when a condition's observedGeneration is above gc's generation

The provider writes the same map concurrently, hence the compare-and-swap. A status the provider stored is trusted, since the provider only stores the status of a GatewayClass that exists.

The skipped write in the status updater is fixed separately in #10409, since it affects every kind. Either PR on its own closes this flake. #10028 targets a different path, transient errors aborting Reconcile, and the failing runs had none.

Evidence:

  • Before: TestTranslationOfOlderSnapshotKeepsNewerGatewayClassStatus translates a snapshot that still holds generation 1 after the generation-2 status is stored, and the stored status goes back to observedGeneration 1.
    After: it keeps observedGeneration 2.
  • Before: TestTranslationOfRecreatedGatewayClassStoresItsOwnStatus stores a late translation of a deleted GatewayClass at generation 5, then translates its replacement at generation 1. With the guard as first proposed here, the replacement stayed at observedGeneration 5.
    After: it stores its own status.
  • Before: TestStoreGatewayClassStatusRacingProviderStore races a provider store of generation 2N against the runner storing generation 2N−1, 20,000 rounds per run. With a separate check and store, it failed within the first ~100 rounds.
    After: 200,000 rounds pass, also under -race.
  • An envtest loop ran the provider and the runner through the conformance test's steps, with the interleaving forced by temporary delays in the runner and the cache (not part of this PR). Before, 8 of 8 iterations ended at observedGeneration 1. After, 0 of 8 did. Without the forced delays, the race showed up about twice in 400 iterations locally.

Merge danger: a two-way door. The blast radius is GatewayClass status. The runner no longer replaces a stored status with one computed for an older generation of the same GatewayClass, and it keeps one small entry per GatewayClass to recognize a recreated one. The provider still stores the status of every reconcile, and the runner still sets Accepted=False for an invalid EnvoyProxy at the current generation.

Which issue(s) this PR fixes:

Fixes #9691


PR Checklist

  • Authorship & ownership: Coding agents / AI assistants are welcome, but I have reviewed every change, understand how and why it works, can explain and maintain it, and take full responsibility for this PR. I have not submitted generated output I do not understand.
  • DCO: All commits are signed off (git commit -s). See DCO: Sign your work.
  • API agreed first: If this PR contains API changes (changes under /api), the API was discussed and agreed before the implementation. The API change can be in a separate PR, or in the same PR, but the API must be agreed before implementation. N/A if this PR does not contain API changes. (N/A: no API changes.)
  • Required checks pass: make generate gen-check, make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, but gen-check, lint, and coverage MUST pass.) (make lint and the runner package tests pass locally. gen-check and the full coverage run are left to CI.)
  • Tests added/updated: New/changed code is covered by appropriate tests. N/A if this PR does not contain code changes.
  • Docs: User-facing changes update the docs, either in this PR or a follow-up PR. N/A if this PR does not contain user-facing changes. (N/A: a status fix with no documented behavior to change.)
  • Release notes: For any non-trivial change, added a release-note fragment under release-notes/current/<section>/<pr-number>-<slug>.md (see release-notes/current/README.md for sections and naming). N/A if this PR does not contain non-trivial changes.
  • Generated files committed: Ran make gen-check and committed the result if API/helm charts/modules changed. (N/A: nothing generated changed.)
  • Scope & compatibility: The PR is reasonably scoped (no unrelated changes) and preserves backward compatibility, or any breaking change is called out above and documented in release-notes/current/breaking_changes/.
  • Codex review: Requested a Codex review and addressed all of its comments.
  • Copilot review: Requested a Copilot review and addressed all of its comments.

@netlify

netlify Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit f481847
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6ae2f9b34182f5c393c8458c
😎 Deploy Preview https://deploy-preview-10408--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T07:50:33.347370Z f481847 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75c04d67af

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/runner/runner.go Resolved
Comment thread internal/gatewayapi/runner/runner.go Resolved
@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.38%. Comparing base (e963b81) to head (f481847).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10408      +/-   ##
==========================================
+ Coverage   82.25%   82.38%   +0.13%     
==========================================
  Files         275      275              
  Lines       43515    43524       +9     
==========================================
+ Hits        35792    35856      +64     
+ Misses       7722     7667      -55     
  Partials        1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d2f7af1bf7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/runner/runner.go Resolved
Comment thread internal/gatewayapi/runner/runner.go Resolved
@nobbs
nobbs marked this pull request as draft October 26, 2026 15:15
@nobbs

nobbs commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: c56408a5c3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@nobbs
nobbs marked this pull request as ready for review October 26, 2026 15:26

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c56408a5c3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

return
}
accepted := meta.FindStatusCondition(stored.Conditions, string(gwapiv1.GatewayClassConditionStatusAccepted))
if accepted != nil && accepted.ObservedGeneration != gc.Generation {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check object identity for equal-generation GatewayClasses

When a GatewayClass is deleted and recreated under the same name at the same generation (commonly generation 1 for both), this condition is false even though the UIDs differ, so a delayed translation of the deleted object CASes its status over the replacement. If the two generation-1 specs produce different status (for example, an old invalid EnvoyProxy versus a new valid one), this publishes the old Accepted value and the later correction can be skipped by the cache-backed updater, leaving the replacement stale without another generation event. Fresh evidence in this revision is that the final guard removed UID tracking while its recreation test covers only generations 5 and 1; retain or verify object identity for the equal-generation case.

Useful? React with 👍 / 👎.

@nobbs
nobbs force-pushed the fix/runner-stale-gatewayclass-status branch from c56408a to 5c36831 Compare October 27, 2026 07:18
nobbs added 6 commits October 29, 2026 08:45
…atewayClass status

The provider stores a GatewayClass status as soon as it reconciles the
class, and the Gateway API runner stores the status again from each
snapshot it translates. When the runner was still translating a snapshot
with the previous generation after a spec change, it put that generation's
status back. The status updater then wrote it, and could skip the
following write of the newer status because its cache had not yet seen
its own write, leaving the GatewayClass at the old observedGeneration.
This made the GatewayClassObservedGenerationBump conformance test flaky.

Skip storing a translated GatewayClass status when the stored status was
already computed for a later generation.

Signed-off-by: Alexej Disterhoft <alexej@disterhoft.de>
Signed-off-by: Alexej Disterhoft <alexej@disterhoft.de>
…s in the status guard

The guard compared generations only. After a GatewayClass was deleted and
created again under the same name, a late translation of the old object
stored its higher generation and then kept every translation of the new
object from storing its status. The check and the store were also two map
operations, so a newer status the provider stored between them could
still be replaced.

Remember the status the runner stored for each GatewayClass and the UID it
was computed for, and do not treat a stored status the runner computed for
another object as newer. Store with LoadOrStore and CompareAndSwap, and
check again when the provider stored in between.

Signed-off-by: Alexej Disterhoft <alexej@disterhoft.de>
The runner translates the generations of a GatewayClass in order, so its
own last store can hold a later generation than the one it is translating
only when that store was computed for an object deleted and created again
under the same name. Comparing the stored status with the runner's last
store already identifies that case, so the UID check added nothing.

Remember only the last status the runner stored, fold the generation
check into storeGatewayClassStatus, and start the compare-and-swap loop
with LoadOrStore.

Signed-off-by: Alexej Disterhoft <alexej@disterhoft.de>
… runner

Only the older-snapshot test needs the whole runner, since it is the one
that covers the call site. The recreated-GatewayClass test now calls
storeGatewayClassStatus directly, like the race test, which removes the
helpers that started the runner, published snapshots with sentinels and
checked the stored generation.

Signed-off-by: Alexej Disterhoft <alexej@disterhoft.de>
…ation

The guard refused only a stored status of a later generation. When a
GatewayClass was deleted and created again under the same name, a late
translation of the old object still replaced the status the provider had
stored for the new one, and the status updater could leave that status on
the new object. The guard also kept the last status it stored for every
GatewayClass name and never dropped it.

The provider stores the status of a GatewayClass before it publishes a
snapshot with it, and generations only grow, so a stored status of any
other generation is either newer than the translation or belongs to
another object. Store the translated status only when the stored Accepted
condition has the same observedGeneration, which needs no state in the
runner.

Run the race test from the provider's status of the translated
generation, since the runner now leaves any other generation alone, and
test that a status the translator changed at the same generation, such
as Accepted=False for an invalid EnvoyProxy, still replaces the
provider's.

Signed-off-by: Alexej Disterhoft <alexej@disterhoft.de>
@nobbs
nobbs force-pushed the fix/runner-stale-gatewayclass-status branch from 5c36831 to f481847 Compare October 29, 2026 07:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestGatewayAPIConformance/GatewayClassObservedGenerationBump is flaky

1 participant