gateway-controller: LlmProxy finalizer never removed when internal delete-retry gets 404 (already deleted)

[Analect]

Summary

gateway-controller's LlmProxy reconciler never removes its own finalizer
after a LlmProxy custom resource is deleted, if that resource's internal
management-API representation is already gone by the time the delete-retry
loop runs. This leaves the Kubernetes object permanently stuck
Terminating, and — for any consumer (like OpenChoreo's own
RenderedRelease controller) that owns the LlmProxy alongside other
resources and waits for it to actually finalize before considering cleanup
complete — blocks that consumer's own cleanup indefinitely.

Reproduced independently five times in one afternoon, always the same
pattern, always requiring a manual kubectl patch ... --type merge -p '{"metadata":{"finalizers":[]}}' to unblock.

Environment

• gateway-controller image: (same image family as the one covered in our
  earlier report on this repo's llm_validator.go — happy to confirm the
  exact tag if useful; this instance is deployed via the
  ai-gateway-wso2-api-platform OpenChoreo community module, one
  APIGateway per gateway instance, wso2-gateway-operator-managed)
• Trigger: deleting a LlmProxy custom resource (gateway.api-platform.wso2.com/v1alpha1)
  that has an active reconciled configuration in the controller's internal
  store

What we observed, with exact log lines

Deleting the LlmProxy CR (via kubectl delete, indirectly through an
owning controller's own cleanup) sets its deletionTimestamp and leaves its
finalizer in place, as expected initially. The controller then repeatedly
attempts to delete its own internal representation of the same resource,
and gets a 404 every time:

{"level":"WARN","source":"/build/pkg/api/handlers/llm_proxy_handler.go:248","msg":"LLM proxy configuration not found for deletion","handle":"insurance-agent-oc-development-1696de50-llm-llm"}
{"level":"INFO","source":"/httpkit/middleware/logging.go:44","msg":"request","method":"DELETE","path":"/api/management/v1/llm-proxies/insurance-agent-oc-development-1696de50-llm-llm","status":404,...}

This repeated six times over roughly four minutes in our capture (timestamps
17:08:48 through 17:11:30), each attempt logging the identical 404 and
warning, with no change in outcome between attempts. At no point does the
controller treat a 404 on its own delete-retry as "already gone, proceed
to remove the finalizer." It just keeps retrying the same call.

The LlmProxy Kubernetes object itself remained Terminating (finalizer
gateway.api-platform.wso2.com/llmproxy-finalizer still present) for the
entire duration we observed — anywhere from several minutes to over 100
minutes across our five reproductions, never self-resolving on its own
within that window.

How we unblocked it each time

kubectl patch llmproxy <name> -n <namespace> --type merge -p '{"metadata":{"finalizers":[]}}'

This is safe in our case because the controller's own logs already confirm
the real cleanup work (removing the runtime deploy config) succeeded before
the retry loop began — the finalizer was the only thing left stuck, not any
actual undone work. Every time, garbage-collecting the CR this way let a
freshly-rendered replacement LlmProxy (same handle, from our own
platform's regenerated release) get created cleanly moments later:

{"level":"INFO","source":"/build/pkg/utils/llm_deployment.go:597","msg":"LLM proxy configuration created","handle":"insurance-agent-oc-development-1696de50-llm-llm",...}
{"level":"INFO","source":"/build/pkg/api/handlers/handlers.go:198","msg":"Configuration deployed successfully",...}
{"level":"INFO","source":"/build/pkg/policyxds/manager.go:94","msg":"Runtime deploy config added",...}

Why this matters beyond an annoyance

Our own platform (OpenChoreo) models the LlmProxy as one of several
resources a higher-level RenderedRelease owns and waits on for a clean
teardown+recreate cycle (needed on every rebuild, since our own release
controller doesn't recreate a running pod from a ComponentRelease update
alone — a separate, unrelated gap). Because this finalizer never clears
itself, that higher-level release gets stuck in "cleaning up" indefinitely,
and the actual application pod never recreates, every single time we need
to redeploy a new build of a component using the ai-llm-proxy trait. This
has now happened on every rebuild of the affected component across a full
day of iterative development.

Possible fix directions (not prescribing one)

• Treat a 404 from the internal management API on a delete-retry as
  success (nothing to delete, proceed to strip the finalizer), the standard
  idempotent-delete pattern.
• Or: if the internal store and the finalizer-removal step are meant to be
  transactionally linked, investigate why the store's own record can go
  missing (a prior partial delete? a race with garbage collection?) before
  the finalizer-removal step runs against it.

Happy to provide the full controller log capture, our own LlmProxy
manifest, or reproduce again with any additional debug logging enabled if
useful.