The gateway controller rewrites unchanged routes every few seconds #438

Activity

1. ecv

   added

   bugSomething isn't workingSomething isn't working

   on Sep 9, 2026
2. ecv commented on Sep 9, 2026

   

   ecv

   on Sep 9, 2026

   ContributorAuthor

   More actions

   Evidence from the production control plane on 9 September, all read-only.

   controller_runtime_reconcile_total{controller="gateway"} on the infrastructure control plane: 10 event-driven successes an hour on 25 August, about 5000 an hour now, with errors up from 45 to 464 an hour. gateway_resource_replicator went from 0 to about 1300 an hour, gateway_downstream_resources from 0 to 116, trafficprotectionpolicy at 475. The periodic requeue rate is unchanged, so this is all event-driven.

   Manager log, a 14 second window at 16:07 UTC, 185 lines across five gateways (cdp-preprod, my-first-edge-o0dcn1, smerwindotcom, apptest, swirls-k8s-3cqr6l). Every downstream httproute processed line for these routes carries operation_result: updated, four per gateway in that window, so about one write every 3.5 seconds per gateway. downstream gateway not yet accepted/programmed, requeueing follows with a five second requeue.

   The write happens in ensureDownstreamHTTPRoute. The CreateOrUpdate mutate function replaces Spec wholesale from ParentRefs, Hostnames and the rules built by processDownstreamHTTPRouteRules. A Service-kind backend reference is built with Kind and Namespace set and Group left nil, and the rule is assembled from the upstream rule's matches and filters. If the downstream API server defaults any of those fields (group, weight, path match), the stored object never equals the desired one and every pass is an update. That is the first thing to check, but it is inferred, not shown. All five looping gateways carry EndpointSlice-backed routes, which is the path #415 and #417 changed in v0.26.2.

   Downstream effect, per edge site: Karmada's execution controller applies about 1029 Works an hour at every site (61 on 25 August), Envoy Gateway reconciles 350 to 3350 times an hour (0 to 40), envoy_listener_manager_lds_update_success runs about 900 an hour (70 to 170), the proxies have sat at 10240 MiB since 2 to 3 September, and the edge controller's post-collection live heap peaks at 2.3 to 2.4 GiB per pass. Onset in two steps, 3 September 12:00 to 18:00 UTC and 4 September 12:00 to 24:00 UTC, with v0.26.3 on the control plane from 2 September 00:00 UTC.
3. ecv

   changed the title [-]Gateway controller rewrites downstream routes every few seconds for gateways that did not change[/-] [+]The gateway controller rewrites unchanged routes every few seconds[/+] on Sep 10, 2026
4. vvoytovych

   self-assigned this

   on Sep 17, 2026
5. vvoytovych

   closed this as completedin #474on Sep 17, 2026
6. vvoytovych commented on Sep 17, 2026

   

   vvoytovych

   on Sep 17, 2026

   Contributor

   More actions

   Shipped today

   network-services-operator v0.27.1 — one line, in production and verified on both estates.
7. vvoytovych commented on Oct 14, 2026

   

   vvoytovych

   on Oct 14, 2026

   Contributor

   More actions

   The second box is not met. Our #474 closed this issue through its closing keyword, but it fixed the route rewrite only.

   Your measure, successful reconciles of the gateway controller on the production infrastructure control plane, read 5,009 in the hour to 16:00 UTC on 14 October. You measured 10 an hour on 25 August and about 5,000 on 9 September. Production runs v0.32.23, so this is with #320, #568, #586 and #587, which each removed a cause of extra reconciles.

   sum by (result) (increase(controller_runtime_reconcile_total{cluster="prod-infrastructure-control-plane", controller="gateway"}[1h]))
   

   I'm reopening this issue for the second box.
8. vvoytovych

   reopened this on Oct 14, 2026
9. vvoytovych

   added a parent issue on Nov 3, 2026

   • Make network services settle and finish deleting #854
10. vvoytovych

    added the

    Bug

    issue type on Nov 3, 2026
11. vvoytovych commented on Nov 9, 2026

    

    vvoytovych

    on Nov 9, 2026

    Contributor

    More actions

    All four boxes hold now for Gateways that do not change. I read the hour to 16:00 UTC on 9 November in 5-minute steps.

    • Boxes 1 and 3 came with fix: stop rewriting unchanged downstream routes #474. In that hour, no Gateway was written downstream except while one was being created or deleted.
    • Box 2. Outside a few short bursts, the gateway controller's event-driven reconciles are as low as in August.
    • Box 4. Outside the same bursts, every edge location is near zero.

    Two things make the rest of the hour. Neither is the rewrite this issue is about:

    • A recurring test. One project creates a few Gateways and deletes them about two minutes later, several times an hour. Each create and delete is real work. Its Gateways go to three US locations, so those three rise in each burst.
    • A one-minute check. While a listener waits for its certificate, its Gateway is reconciled every minute (gateway_controller.go). These passes write nothing.

    I'm closing this.
12. vvoytovych

    closed this as completedon Nov 9, 2026