Skip to content

feat(BA-8444): export ingresses from the coordinator to Traefik etcd - #15893

Draft
jopemachine wants to merge 35 commits into
mainfrom
feat/BA-8444-traefik-ingress-export
Draft

jopemachine wants to merge 35 commits into
mainfrom
feat/BA-8444-traefik-ingress-export

Conversation

@jopemachine

@jopemachine jopemachine commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Resolves #15540 (BA-8444)

Summary

  • With enable_traefik, the coordinator /v2/ingress API sends every authority's ingresses to TraefikWorkerClient, which writes them to Traefik etcd, instead of pushing to workers. Without it, the push path from feat(BA-8443): push ingresses from the coordinator to api-delivery workers #15833 is unchanged.
  • TraefikWorkerClient and AppProxyWorkerClient share BaseWorkerClient, so IngressService picks an authority's clients in one place and waits for every client before reporting the failed ones in IngressPushFailedError (renamed from ApiDeliveryWorkerPushFailedError).
  • Each ingress is one set of subtrees under worker_{authority}/{http|tcp}/: routers/bai_ingress_{frontend_key}_router, services/..._service, and for HTTP middlewares/..._ipallowlist and ..._auth.
    • Every prefix of the ingress, including the other protocol's, is replaced in one transaction, so a protocol or allowlist it no longer has is cleared.
    • The revision is kept at ingress_revisions/worker_{authority}/{frontend_key}/revision, outside every Traefik root key. The replacement commits only if that key still holds the revision read before it (compare_and_replace_prefixes, feat(BA-8643): add compare_and_replace_prefixes to AbstractKVStore #15988).
    • As an api-delivery worker does, an upsert writes nothing when any of its ingresses is older than the held revision (409 TraefikIngressRevisionConflictError), and rewrites one of the same revision.
    • When another write to the same frontend commits between the revision check and the write, the write fails with 409 TraefikIngressConcurrentWriteError instead of retrying; resending the ingress applies it or reports a newer revision. An api-delivery worker serializes the check and the write under a lock, so it never fails this way.
    • Each ingress is written in its own transaction to stay within etcd's per-transaction operation limit.
    • Remove clears the set and its revision.
  • The appproxy-traefik-plugin middleware receives the ingress JSON as ingress in place of circuit.
  • TraefikWorkerClient takes TraefikRoutingConfigFactory (merged in refactor(BA-8675): add TraefikRoutingConfigFactory for ingresses #16018) through TraefikWorkerClientArgs, clears every key from all_routing_config_etcd_keys in the same transaction as writing the configs from routing_configs_by_etcd_key (or on remove), and keeps only the etcd transactions.
  • The worker values a router needs (hostname, wildcard_domain, tls_listen) come from service discovery: a worker now registers them as hostname, wildcard-domain and tls-listen labels next to authority, so the Traefik export does not read the coordinator DB. An authority with no worker registered with them is reported as 404 WorkerNotFound.
  • The coordinator builds the api-delivery worker ClientPool and the Traefik worker client in a new components stage (dependencies/components/, ComponentsComposer with WorkerClientPoolProvider and TraefikWorkerClientProvider). The server enters it in components_ctx, which replaces worker_client_pool_ctx, and ingress_service_ctx only assembles IngressService.
  • The circuit path and its reconcile are untouched. The reconcile only collects bai_service_{uuid} keys, so it never deletes ingress keys.

Not in this PR

  • The plugin repository reading ingress instead of circuit.
  • The usage-marker plugin (appproxy-traefik-plugin-go) for ingresses; it takes session ids, which an ingress does not carry (BA-8321 records usage by target id).
  • Target group weights: every target of a group with weight above 0 is served equally.
  • A removal drops the revision, so an upsert older than the removal that arrives later writes the ingress again.

Test plan

  • pants check / pants test on tests/unit/appproxy::
  • clients/worker/traefik/test_client.py: writes when the held revision is absent, older or the same; rejects a newer one; writes none of a batch holding an older ingress
  • services/test_ingress.py: a failed Traefik write is reported after every write finishes

🤖 Generated with Claude Code

@github-actions github-actions Bot added the size:L 100~500 LoC label Oct 20, 2026
jopemachine added a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added comp:common Related to Common component comp:app-proxy Related to App Proxy component labels Oct 20, 2026
@jopemachine
jopemachine force-pushed the feat/BA-8443-coordinator-ingress-push branch from 96cfd2b to 23c7172 Compare October 20, 2026 05:30
jopemachine added a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jopemachine
jopemachine force-pushed the feat/BA-8444-traefik-ingress-export branch from 1863133 to 812ac34 Compare October 20, 2026 05:31
@jopemachine
jopemachine force-pushed the feat/BA-8443-coordinator-ingress-push branch from 23c7172 to 5db6471 Compare October 20, 2026 05:34
jopemachine added a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jopemachine
jopemachine force-pushed the feat/BA-8444-traefik-ingress-export branch from 812ac34 to 70a4a7f Compare October 20, 2026 05:34
@jopemachine
jopemachine force-pushed the feat/BA-8443-coordinator-ingress-push branch from 5db6471 to be45681 Compare October 20, 2026 05:44
jopemachine added a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jopemachine
jopemachine force-pushed the feat/BA-8444-traefik-ingress-export branch from 70a4a7f to 3272c7c Compare October 20, 2026 05:45
@jopemachine
jopemachine force-pushed the feat/BA-8443-coordinator-ingress-push branch from be45681 to bacd9c7 Compare October 20, 2026 05:47
@jopemachine
jopemachine force-pushed the feat/BA-8444-traefik-ingress-export branch from 3272c7c to fa67cfa Compare October 20, 2026 05:47
jopemachine added a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jopemachine
jopemachine force-pushed the feat/BA-8443-coordinator-ingress-push branch from bacd9c7 to ada3d50 Compare October 20, 2026 05:49
@github-actions github-actions Bot added the size:XL 500~ LoC label Oct 20, 2026
jopemachine added a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot removed the size:L 100~500 LoC label Oct 20, 2026
@jopemachine
jopemachine force-pushed the feat/BA-8444-traefik-ingress-export branch from 33a9c3e to e986316 Compare October 20, 2026 05:49
@github-actions github-actions Bot added size:L 100~500 LoC and removed size:XL 500~ LoC labels Oct 20, 2026
@jopemachine
jopemachine force-pushed the feat/BA-8443-coordinator-ingress-push branch from ada3d50 to 3b02de2 Compare October 20, 2026 05:52
jopemachine added a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jopemachine
jopemachine force-pushed the feat/BA-8444-traefik-ingress-export branch from e986316 to 55346e3 Compare October 20, 2026 05:52
@jopemachine
jopemachine force-pushed the feat/BA-8443-coordinator-ingress-push branch from 3b02de2 to d22905f Compare October 20, 2026 05:58
jopemachine and others added 7 commits October 23, 2026 13:09
`_revision_prefix` and `_revision_key` become methods of
`TraefikWorkerClient`, and the older-revision check is written where the
held revision is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_Names` becomes `_IngressTraefikObjectNames` with `router_name`,
`service_name`, `ip_allowlist_middleware_name` and
`auth_plugin_middleware_name`, `_ingress_prefixes` becomes
`_ingress_etcd_prefixes`, and the locals spell out the held revision, the
ingress key, the protocol etcd prefix, the router rule and the worker
clients by authority.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_write` makes one guarded replacement. When another write to the same
frontend commits between the revision check and the write, it raises
`TraefikIngressConcurrentWriteError` (409) instead of re-reading and
retrying; resending the ingress applies it or reports a newer revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nfigFactory

`etcd_prefix` becomes `router_etcd_key`, `service_etcd_key` and
`http_middleware_etcd_key`, which return the whole key of one object, so
`TraefikWorkerClient` no longer joins a protocol prefix with
`routers`/`services`/`middlewares` itself. `_ingress_etcd_prefixes` becomes
`_ingress_object_etcd_keys`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onfigFactory

The factory builds the object names, etcd keys and configs of an ingress
from the ingress itself: `object_etcd_keys`, `router_etcd_key`,
`service_etcd_key`, `http_router_config(ingress, worker)`,
`tcp_router_config(ingress)` and `http_middleware_configs(ingress)`, which
also decides whether the ipAllowList middleware is written. This drops
`_IngressTraefikObjectNames` and `_ingress_object_etcd_keys` from
`TraefikWorkerClient`, which now keeps only the etcd transactions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se them

The object name, etcd key, rule, entrypoint and target helpers and the
CORS middleware name become private members of
`TraefikRoutingConfigFactory`, and `_REVISION_ROOT` a private constant of
`TraefikWorkerClient`, since nothing outside those classes uses them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Traefik v3 rule matchers take one value each, so a rule such as
ClientIP(`a`, `b`) is rejected and the router is disabled. An ingress with
several allowed client IPs now gets ClientIP(`a`) || ClientIP(`b`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jopemachine and others added 7 commits October 23, 2026 15:32
A worker registers its hostname, wildcard domain and tls_listen as
service discovery labels next to its authority, and `TraefikWorkerClient`
reads them from the appproxy-worker group instead of the coordinator
`workers` table, so the Traefik ingress export does not depend on the
coordinator DB. `WorkerRepository` is removed; a worker registered without
these labels is reported as `ObjectNotFound` as a missing row was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e factory

`TraefikRoutingConfigFactory` keeps two public methods,
`object_etcd_keys` and `object_configs_by_etcd_key`, which returns the
config of every Traefik object an ingress uses by its etcd key. The etcd
key, router, service and middleware builders become private, and
`TraefikWorkerClient._generate_ingress_subtree` only clears the object
keys and converts the configs for etcd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator holds a core and a worker service discovery, and
`TraefikWorkerClient` and `IngressService` read workers from the latter,
so their field, argument and test fixture names say
`worker_service_discovery`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uting configs

"Object" was a term made up for them; Traefik calls them its routing
configuration, as the factory name does. `object_etcd_keys` becomes
`routing_config_etcd_keys` and `object_configs_by_etcd_key` becomes
`routing_configs_by_etcd_key`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_CORS_HEADERS_MIDDLEWARE_NAME` and `_REVISION_ROOT` move out of
`TraefikRoutingConfigFactory` and `TraefikWorkerClient` to their modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…omponents composer

`ComponentsComposer` provides the api-delivery worker `ClientPool` and the
Traefik worker client (or None without Traefik etcd) through
`WorkerClientPoolProvider` and `TraefikWorkerClientProvider`. The server
enters it in `components_ctx`, which replaces `worker_client_pool_ctx`, and
`ingress_service_ctx` only assembles `IngressService` from the results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`TraefikWorkerClient` raises the new `WorkerNotFound` (404) with the
authority when service discovery holds no worker of it, in place of the
generic `ObjectNotFound`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jopemachine and others added 13 commits October 23, 2026 16:24
The factory keeps the `TraefikConfig` it is given and reads
`client_ip_strategy` from it when building an ipAllowList middleware,
instead of keeping only the `ClientIPStrategyConfig`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`TraefikWorker` is the value the Traefik worker client builds from
service discovery and the routing config factory reads, so it lives in
its own module next to both instead of in the factory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It holds the worker's frontend settings a Traefik router needs (hostname,
wildcard domain, tls_listen), not a worker, so the type and the arguments
carrying it say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…listen separately

`WorkerFrontendSettings` grouped two worker API settings with the Traefik
wildcard domain under a made-up name. The factory now takes the three
values as keyword arguments and hands each private builder only the ones
it uses, and `clients/worker/types.py` is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tingConfigFactory

The factory receives the Traefik, secret and permit hash config sections
as one `TraefikRoutingConfigFactoryArgs` and reads the auth plugin
secrets from them, instead of taking the two secret strings separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fig_etcd_keys

It returns every routing config key an ingress can own, whether or not the
ingress uses it now, for clearing the ones it no longer has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The router, service and middleware configs the factory returns are
`TypedDict`s (`TraefikHttpRouterConfig`, `TraefikTcpServiceConfig`,
`TraefikIpAllowListMiddlewareConfig`, ...) instead of `dict[str, Any]`,
with the keys Traefik reads. The written etcd keys and values are
unchanged. The tests compare each config as a whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every TypedDict in traefik_routing_config describes a Traefik config, so
the module name already carries the prefix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… into a traefik package

Move clients/worker/traefik.py to clients/worker/traefik/client.py and the routing
config module to clients/worker/traefik/routing_config.py with its TypedDicts in
clients/worker/traefik/types.py, matching #16018.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ances

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ingress-export

# Conflicts:
#	src/ai/backend/appproxy/coordinator/errors/ingress.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:app-proxy Related to App Proxy component comp:common Related to Common component size:XL 500~ LoC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export Ingress from the coordinator to Traefik etcd

1 participant