diff --git a/docs/how-to/manage-ai-gateway-keys.md b/docs/how-to/manage-ai-gateway-keys.md index c72343ab..2871b84f 100644 --- a/docs/how-to/manage-ai-gateway-keys.md +++ b/docs/how-to/manage-ai-gateway-keys.md @@ -151,10 +151,19 @@ Before each key create, the operator writes a cleanup record: a ConfigMap named When the control plane later reaches another Coder deployment (for example after its database Secret changed), the record still locates the keys in the old one. The operator then stalls the object with `CoderDeploymentChanged` at its next key create, keeps the record and creates no key in the new deployment. A delete of the object then removes the finalizer without a delete in Coder. Delete the keys in the old deployment, then delete the ConfigMap. -The operator removes the record only after a list in the record's deployment shows no key of the object: at the end of a normal delete, or after the one bounded pass when the last create was more than 5 minutes before it. In every other release without a delete, the record stays and names where the keys are. Find the records with `kubectl -n coder-system get configmaps -l coder.com/cleanup-record=CoderAIGatewayKey`. The operator gets write access to ConfigMaps only in that namespace. Do not give other users write access to ConfigMaps there. +The operator removes the record only after a list in the record's deployment shows no key of the object: at the end of a normal delete, or after the one bounded pass when the last create was more than 5 minutes before it. In every other release without a delete, for example in a namespace deletion, the record stays, and the operator sweeps it: + +1. It waits until the object with that UID is gone and 5 minutes have passed since the last create, so that a create still in flight commits first. +2. It looks for a ready `CoderControlPlane` in the object's namespace whose Coder answers the record's deployment ID now. It never matches by URL or name: a new Coder database behind the same URL has another deployment ID and is never swept. A control plane in another namespace never sweeps the record. +3. Through that control plane, whose operator user must be a site owner, it lists the keys and deletes every key whose name is the record's prefix `ck8s--` followed by a serial number (the names that the operator gives, such as `ck8s--3`). It never deletes any other key. +4. It removes the record when a list shows no such key. A failed list or delete, or a `404` that is not Coder's own, keeps the record, and the operator retries every minute. + +So after a namespace deletion with an external database, the leftover keys go once you create the namespace again with a control plane that uses the same database. The metric `coder_k8s_cleanup_records_swept_total` (label `kind`) counts the swept records. + +When no ready control plane reaches the record's deployment, the record stays. The gauge `coder_k8s_cleanup_records_unmatched` counts these records, and the operator writes a `NoMatchingDeployment` Warning Event on the ConfigMap every 10 minutes. When that Coder deployment never comes back, delete the keys with the record's prefix in Coder by hand, then delete the ConfigMap. Find the records with `kubectl -n coder-system get configmaps -l coder.com/cleanup-record=CoderAIGatewayKey`. The operator gets write access to ConfigMaps only in that namespace. Do not give other users write access to ConfigMaps there: a record names the keys that a sweep deletes. !!! warning "Delete the keys before the namespace" - When you delete the namespace, Kubernetes deletes the control plane and the `CoderAIGatewayKey` objects at the same time. The control plane is usually gone before the operator handles the objects, so the finalizers go without a delete, and the keys stay valid in Coder. The namespace deletion finishes, and the operator logs the key IDs with reason `ControlPlaneGone`. + When you delete the namespace, Kubernetes deletes the control plane and the `CoderAIGatewayKey` objects at the same time. The control plane is usually gone before the operator handles the objects, so the finalizers go without a delete, and the keys stay valid in Coder. The namespace deletion finishes, and the operator logs the key IDs with reason `ControlPlaneGone`. A [sweep](#cleanup-records) deletes the keys only when the namespace comes back with a control plane on the same Coder database. Until then they stay valid. Delete the `CoderAIGatewayKey` objects first and wait until they are gone, then delete the control plane or the namespace: @@ -191,7 +200,7 @@ A deletion that keeps failing never ends by itself. Leave it in one of these way ## Known limits -- **Leftover keys.** A key that commits more than 5 minutes after a lost create answer, or after the object is gone, stays in Coder. So do the keys of objects that go together with their control plane, for example in a namespace deletion. Find keys of deleted objects with `coder ai-gateway keys list`: their names start with `ck8s-` and a UID that no `CoderAIGatewayKey` has. +- **Leftover keys.** A key that commits more than 5 minutes after a lost create answer, or after the object is gone, stays in Coder. So do the keys of objects that go together with their control plane, for example in a namespace deletion, until a [sweep](#cleanup-records) deletes them. Find keys of deleted objects with `coder ai-gateway keys list`: their names start with `ck8s-` and a UID that no `CoderAIGatewayKey` has. - **A short key list.** The operator reads all keys with one list call. When a list answer names the current key but leaves out a replaced key, the operator treats the replaced key as deleted. If the key appears again, the operator deletes it as an orphan, without the rest of its grace period. Coder lists all keys in one answer, so only a proxy between the operator and Coder that changes the answer can cause this. ## Grant access diff --git a/internal/controller/cleanup_sweep.go b/internal/controller/cleanup_sweep.go index aebeec4e..accc677d 100644 --- a/internal/controller/cleanup_sweep.go +++ b/internal/controller/cleanup_sweep.go @@ -63,8 +63,8 @@ func init() { // CleanupRecordReconciler sweeps the cleanup records of objects that went // without a confirmed delete in Coder (#353). It acts only through a ready // control plane whose current Coder deployment ID equals the record's, and -// it deletes only the recorded provider ID or the provider with the -// recorded nonce. +// it deletes only the recorded provider ID, the provider with the recorded +// nonce, or the AI Gateway keys whose names start with the recorded prefix. type CleanupRecordReconciler struct { client.Client // APIReader reads the object of a record live, so a stale cache never @@ -78,8 +78,9 @@ type CleanupRecordReconciler struct { } // SetupWithManager watches the records in the operator namespace. It wakes a -// record when its CoderAIProvider is deleted, and every record when a control -// plane gets operator access, since that control plane can match them. +// record when its CoderAIProvider or CoderAIGatewayKey is deleted, and every +// record when a control plane gets operator access, since that control plane +// can match them. func (r *CleanupRecordReconciler) SetupWithManager(mgr ctrl.Manager) error { if mgr == nil || r.Client == nil || r.APIReader == nil || r.OperatorNamespace == "" || r.Clock == nil || r.Recorder == nil { return errors.New("assertion failed: the cleanup record reconciler needs a manager, a client, an API reader, the operator namespace, a clock and an event recorder") @@ -87,11 +88,13 @@ func (r *CleanupRecordReconciler) SetupWithManager(mgr ctrl.Manager) error { isRecord := predicate.NewPredicateFuncs(func(obj client.Object) bool { return obj.GetNamespace() == r.OperatorNamespace && obj.GetLabels()[cleanupRecordLabel] != "" }) + recordOf := handler.EnqueueRequestsFromMapFunc(func(_ context.Context, obj client.Object) []reconcile.Request { + return []reconcile.Request{{NamespacedName: types.NamespacedName{Namespace: r.OperatorNamespace, Name: cleanupRecordObjectName(obj.GetUID())}}} + }) return ctrl.NewControllerManagedBy(mgr). For(&corev1.ConfigMap{}, builder.WithPredicates(isRecord)). - Watches(&coderv1alpha1.CoderAIProvider{}, handler.EnqueueRequestsFromMapFunc(func(_ context.Context, obj client.Object) []reconcile.Request { - return []reconcile.Request{{NamespacedName: types.NamespacedName{Namespace: r.OperatorNamespace, Name: cleanupRecordObjectName(obj.GetUID())}}} - }), builder.WithPredicates(dependentRemoved)). + Watches(&coderv1alpha1.CoderAIProvider{}, recordOf, builder.WithPredicates(dependentRemoved)). + Watches(&coderv1alpha1.CoderAIGatewayKey{}, recordOf, builder.WithPredicates(dependentRemoved)). Watches(&coderv1alpha1.CoderControlPlane{}, handler.EnqueueRequestsFromMapFunc(r.allRecords), builder.WithPredicates(operatorAccessGained)). Named("cleanuprecord"). Complete(r) @@ -149,13 +152,14 @@ func (r *CleanupRecordReconciler) Reconcile(ctx context.Context, req ctrl.Reques log.FromContext(ctx).Error(err, "Ignoring a ConfigMap that is not a valid cleanup record") return ctrl.Result{}, nil } - if record.Kind != cleanupRecordKindAIProvider { - return ctrl.Result{}, nil // Only provider records are swept so far. + kind, ok := cleanupRecordKinds[record.Kind] + if !ok { + return ctrl.Result{}, nil // A kind that this operator does not sweep. } // Act only when the object with that UID no longer exists (invariant 6). - provider := &coderv1alpha1.CoderAIProvider{} - if err := r.APIReader.Get(ctx, types.NamespacedName{Namespace: record.Namespace, Name: record.Name}, provider); err == nil && provider.UID == record.UID { + object := kind.object() + if err := r.APIReader.Get(ctx, types.NamespacedName{Namespace: record.Namespace, Name: record.Name}, object); err == nil && object.GetUID() == record.UID { r.setUnmatched(record.UID, record.Kind, false) return ctrl.Result{RequeueAfter: cleanupObjectRecheck}, nil } else if err != nil && !apierrors.IsNotFound(err) { @@ -181,23 +185,23 @@ func (r *CleanupRecordReconciler) Reconcile(ctx context.Context, req ctrl.Reques if len(matches) == 0 { r.setUnmatched(record.UID, record.Kind, true) r.Recorder.Eventf(configMap, nil, corev1.EventTypeWarning, "NoMatchingDeployment", "Sweep", - "No ready CoderControlPlane in namespace %s reaches Coder deployment %s, which holds the provider of deleted %s %s/%s. Delete the provider in Coder by hand, then delete this ConfigMap.", - record.Namespace, record.DeploymentID, record.Kind, record.Namespace, record.Name) + "No ready CoderControlPlane in namespace %s reaches Coder deployment %s, which holds %s of deleted %s %s/%s. Delete %s in Coder by hand, then delete this ConfigMap.", + record.Namespace, record.DeploymentID, kind.what, record.Kind, record.Namespace, record.Name, kind.what) return ctrl.Result{RequeueAfter: cleanupUnmatchedRecheck}, nil } r.setUnmatched(record.UID, record.Kind, false) - logger := log.FromContext(ctx).WithValues("record", configMap.Name, "deploymentID", record.DeploymentID, "provider", record.Namespace+"/"+record.Name) + logger := log.FromContext(ctx).WithValues("record", configMap.Name, "deploymentID", record.DeploymentID, "object", record.Kind+" "+record.Namespace+"/"+record.Name) // Each matching control plane has its own operator token, so a failure // through one tries the next. gone, why, through := false, "", logger for _, m := range matches { through = logger.WithValues("controlPlane", types.NamespacedName{Namespace: m.controlPlane.Namespace, Name: m.controlPlane.Name}) - if gone, why, err = sweepAIProvider(ctx, m.sdk, record); err != nil { + if gone, why, err = kind.sweep(ctx, m.sdk, record); err != nil { return ctrl.Result{}, err } else if gone { break } - through.Info("The cleanup of a deleted provider in Coder failed", "why", why) + through.Info("The cleanup of a deleted object in Coder failed", "why", why) } if !gone { return ctrl.Result{RequeueAfter: cleanupRetry}, nil @@ -207,10 +211,26 @@ func (r *CleanupRecordReconciler) Reconcile(ctx context.Context, req ctrl.Reques return ctrl.Result{}, err } cleanupRecordsSwept.WithLabelValues(record.Kind).Inc() - through.Info("Swept the provider of a deleted CoderAIProvider in Coder", "result", why, "providerID", record.CoderID) + through.Info("Swept the Coder resources of a deleted object", "result", why, "coderID", record.CoderID) return ctrl.Result{}, nil } +// cleanupSweepKind is how a sweep handles the records of one object kind. +type cleanupSweepKind struct { + object func() client.Object + what string // What the object holds in Coder, for the Event. + sweep func(context.Context, *codersdk.Client, *cleanupRecord) (gone bool, why string, err error) +} + +var cleanupRecordKinds = map[string]cleanupSweepKind{ + cleanupRecordKindAIProvider: { + object: func() client.Object { return &coderv1alpha1.CoderAIProvider{} }, what: "the provider", sweep: sweepAIProvider, + }, + cleanupRecordKindAIGatewayKey: { + object: func() client.Object { return &coderv1alpha1.CoderAIGatewayKey{} }, what: "the AI Gateway keys", sweep: sweepAIGatewayKeys, + }, +} + // controlPlaneMatch is a control plane whose Coder deployment is a record's. type controlPlaneMatch struct { sdk *codersdk.Client @@ -309,6 +329,44 @@ func sweepAIProvider(ctx context.Context, sdk *codersdk.Client, record *cleanupR return true, "deleted provider " + id, nil } +// sweepAIGatewayKeys deletes the keys of record in Coder: every key whose +// name is the record's prefix ck8s-- and a serial (invariant 5). The +// object is gone, so no serial bound applies: the settle window has passed, +// and no create of the object can still commit. It returns gone only when a +// LIST, read by a site owner, shows no such key. +func sweepAIGatewayKeys(ctx context.Context, sdk *codersdk.Client, record *cleanupRecord) (gone bool, why string, err error) { + prefix, err := aiGatewayKeyNamePrefix(record.UID) + if err != nil { + return false, "", err + } + if record.Nonce != prefix { + return false, "", fmt.Errorf("assertion failed: cleanup record of %s has key prefix %q, not the prefix of its UID", record.UID, record.Nonce) + } + if step := requireSiteOwner(ctx, sdk); step != nil { + // Only a site owner's LIST shows every key. + return false, step.message, nil + } + deleted := 0 + for range 2 { + listed, err := sdk.ListAIGatewayKeys(ctx) + if err != nil { + return false, "list AI Gateway keys: " + coderErrorSummary(err), nil + } + own := ownAIGatewayKeys(prefix, listed) + if len(own) == 0 { + return true, fmt.Sprintf("deleted %d keys, and a list shows none left", deleted), nil + } + for _, k := range own { + // Only Coder's own 404 proves that the key is gone (#325). + if err := sdk.DeleteAIGatewayKey(ctx, k.ID); err != nil && !isCoderResourceNotFound(err) { + return false, "delete key " + k.Name + ": " + coderErrorSummary(err), nil + } + deleted++ + } + } + return false, "Coder still lists keys of the object after their delete", nil +} + // setUnmatched keeps the unmatched gauge equal to the records that no // deployment reaches. func (r *CleanupRecordReconciler) setUnmatched(uid types.UID, kind string, unmatched bool) { @@ -317,7 +375,7 @@ func (r *CleanupRecordReconciler) setUnmatched(uid types.UID, kind string, unmat } else { r.unmatched.Delete(uid) } - counts := map[string]float64{cleanupRecordKindAIProvider: 0} + counts := map[string]float64{cleanupRecordKindAIProvider: 0, cleanupRecordKindAIGatewayKey: 0} r.unmatched.Range(func(_, v any) bool { counts[v.(string)]++ return true diff --git a/internal/controller/cleanup_sweep_key_test.go b/internal/controller/cleanup_sweep_key_test.go new file mode 100644 index 00000000..f8e46add --- /dev/null +++ b/internal/controller/cleanup_sweep_key_test.go @@ -0,0 +1,224 @@ +package controller_test + +import ( + "context" + "net/http" + "strings" + "testing" + "time" + + "github.com/coder/coder/v2/codersdk" + "github.com/google/uuid" + "github.com/stretchr/testify/require" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/tools/events" + clocktesting "k8s.io/utils/clock/testing" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/metrics" + + coderv1alpha1 "github.com/coder/coder-k8s/api/v1alpha1" + "github.com/coder/coder-k8s/internal/controller" +) + +// keySweepEnv is a CoderAIGatewayKey that went with its control plane, as in +// a namespace deletion: Coder keeps its keys and the record stays. Coder also +// holds keys that are not the object's: a person's key, and a key of another +// object. +type keySweepEnv struct { + aiGatewayKeyEnv + key *coderv1alpha1.CoderAIGatewayKey + clock *clocktesting.FakeClock + recorder *events.FakeRecorder + sweeper *controller.CleanupRecordReconciler +} + +func newKeySweepEnv(ctx context.Context, t *testing.T) keySweepEnv { + t.Helper() + env := newAIGatewayKeyEnv(ctx, t, ptrTo(true)) + got := rotatedKey(ctx, t, &env) + sdk := env.fake.client(t, 5*time.Second) + for _, name := range []string{ + aiGatewayKeyPrefix(got) + "9", // An orphan of a create whose answer was lost. + "personal-key", + "ck8s-" + strings.ReplaceAll(uuid.NewString(), "-", "") + "-0", + } { + _, err := sdk.CreateAIGatewayKey(ctx, codersdk.CreateAIGatewayKeyRequest{Name: name}) + require.NoError(t, err) + } + require.NoError(t, k8sClient.Delete(ctx, &coderv1alpha1.CoderControlPlane{ObjectMeta: metav1.ObjectMeta{Namespace: env.namespace, Name: "coder"}})) + require.NoError(t, k8sClient.Delete(ctx, got)) + gone, _ := env.reconcile(ctx, t, got) + require.Nil(t, gone, "ControlPlaneGone releases the finalizer") + require.NotNil(t, env.aiGatewayKeyRecord(ctx, t, got.UID)) + require.Len(t, env.fake.aiGatewayKeys(), 5) + return newKeySweeper(t, env, got) +} + +func newKeySweeper(t *testing.T, env aiGatewayKeyEnv, key *coderv1alpha1.CoderAIGatewayKey) keySweepEnv { + t.Helper() + s := keySweepEnv{aiGatewayKeyEnv: env, key: key, clock: clocktesting.NewFakeClock(env.clock.Now()), recorder: events.NewFakeRecorder(10)} + s.sweeper = &controller.CleanupRecordReconciler{Client: watchClient(t), APIReader: k8sClient, OperatorNamespace: env.records, Clock: s.clock, Recorder: s.recorder} + return s +} + +func (s keySweepEnv) sweep(ctx context.Context, t *testing.T) ctrl.Result { + t.Helper() + result, err := s.sweeper.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKey{Namespace: s.records, Name: "ck8s-cleanup-" + string(s.key.UID)}}) + require.NoError(t, err) + return result +} + +// keysLeft returns the names of the keys that Coder holds. +func (s keySweepEnv) keysLeft() []string { + var names []string + for _, k := range s.fake.aiGatewayKeys() { + names = append(names, k.name) + } + return names +} + +// kindMetric reads a controller metric with the label kind. +func kindMetric(t *testing.T, name, kind string) float64 { + t.Helper() + families, err := metrics.Registry.Gather() + require.NoError(t, err) + for _, f := range families { + for _, m := range f.GetMetric() { + for _, l := range m.GetLabel() { + if f.GetName() == name && l.GetName() == "kind" && l.GetValue() == kind { + if m.GetGauge() != nil { + return m.GetGauge().GetValue() + } + return m.GetCounter().GetValue() + } + } + } + } + return 0 +} + +// #353, key half: after a namespace deletion, a new control plane that +// reaches the same Coder deployment deletes every key with the object's +// prefix, once the settle window has passed, and leaves every other key. A +// different deployment at the same URL never does. Not parallel: it reads +// process-wide metrics. +func TestCleanupSweepAIGatewayKeys(t *testing.T) { + ctx := context.Background() + s := newKeySweepEnv(ctx, t) + const kind = "CoderAIGatewayKey" + swept := kindMetric(t, "coder_k8s_cleanup_records_swept_total", kind) + + requests := s.fake.totalRequests() + result := s.sweep(ctx, t) + require.Equal(t, requests, s.fake.totalRequests(), "no Coder call inside the settle window") + require.Greater(t, result.RequeueAfter, 4*time.Minute) + require.LessOrEqual(t, result.RequeueAfter, 5*time.Minute, "it waits for the end of the settle window after the last create") + s.clock.Step(6 * time.Minute) + require.Equal(t, 10*time.Minute, s.sweep(ctx, t).RequeueAfter, "no control plane reaches the deployment") + event := <-s.recorder.Events + require.Contains(t, event, "NoMatchingDeployment") + require.Contains(t, event, "the AI Gateway keys") + require.Equal(t, 1.0, kindMetric(t, "coder_k8s_cleanup_records_unmatched", kind)) + + // A new control plane at the same URL, but a new Coder database answers. + createTestControlPlane(ctx, t, s.namespace, "coder-b", s.fake.server.URL) + recorded := s.fake.deploymentID + s.fake.mu.Lock() + s.fake.deploymentID = uuid.NewString() + s.fake.mu.Unlock() + require.Equal(t, 10*time.Minute, s.sweep(ctx, t).RequeueAfter) + require.Zero(t, s.fake.requestCount(routeDeleteAIGatewayKey), "a different deployment is never swept") + require.Len(t, s.fake.aiGatewayKeys(), 5) + + // The same deployment comes back. + s.fake.mu.Lock() + s.fake.deploymentID = recorded + s.fake.mu.Unlock() + require.Zero(t, s.sweep(ctx, t).RequeueAfter) + left := s.keysLeft() + require.Len(t, left, 2, "the person's key and the other object's key stay") + require.Contains(t, left, "personal-key") + for _, name := range left { + require.False(t, strings.HasPrefix(name, aiGatewayKeyPrefix(s.key)), "every key of the object is gone: %s", name) + } + require.Equal(t, 3, s.fake.requestCount(routeDeleteAIGatewayKey), "the current, the retiring and the orphan key") + require.Nil(t, s.aiGatewayKeyRecord(ctx, t, s.key.UID), "a LIST confirmed the delete, so the record goes") + require.Equal(t, swept+1, kindMetric(t, "coder_k8s_cleanup_records_swept_total", kind)) + require.Zero(t, kindMetric(t, "coder_k8s_cleanup_records_unmatched", kind)) +} + +// #353 invariants 5 to 7 for keys: a sweep never touches the keys of a +// living object, keeps the record until a site owner's LIST shows no key of +// the object, and decides by UID that the object is gone. +func TestCleanupSweepAIGatewayKeyCases(t *testing.T) { + t.Parallel() + ctx := context.Background() + t.Run("living object", func(t *testing.T) { + t.Parallel() + env := newAIGatewayKeyEnv(ctx, t, ptrTo(true)) + got, _ := env.reconcile(ctx, t, env.createKey(ctx, t, "coder", testAIGatewaySecret)) + s := newKeySweeper(t, env, got) + s.clock.Step(time.Hour) + requests := s.fake.totalRequests() + require.Equal(t, 30*time.Minute, s.sweep(ctx, t).RequeueAfter) + require.Equal(t, requests, s.fake.totalRequests(), "no Coder call for a living object") + require.NotNil(t, s.aiGatewayKeyRecord(ctx, t, got.UID)) + }) + t.Run("same name, another object", func(t *testing.T) { + t.Parallel() + s := newKeySweepEnv(ctx, t) + createTestControlPlane(ctx, t, s.namespace, "coder-b", s.fake.server.URL) + s.createKey(ctx, t, "coder", testAIGatewaySecret) // The old name, a new UID. + s.clock.Step(6 * time.Minute) + require.Zero(t, s.sweep(ctx, t).RequeueAfter, "only the UID decides that the object is gone") + require.Len(t, s.keysLeft(), 2) + }) + t.Run("delete fails", func(t *testing.T) { + t.Parallel() + s := newKeySweepEnv(ctx, t) + createTestControlPlane(ctx, t, s.namespace, "coder-b", s.fake.server.URL) + s.fake.failNext(routeDeleteAIGatewayKey, fakeFault{Status: http.StatusNotFound, HTML: "not found"}) + s.clock.Step(6 * time.Minute) + require.Equal(t, time.Minute, s.sweep(ctx, t).RequeueAfter, "only Coder's own 404 confirms a delete") + require.NotNil(t, s.aiGatewayKeyRecord(ctx, t, s.key.UID), "the record stays until Coder confirms") + require.Len(t, s.keysLeft(), 5, "the sweep stops at the failed DELETE") + require.Zero(t, s.sweep(ctx, t).RequeueAfter) + require.Nil(t, s.aiGatewayKeyRecord(ctx, t, s.key.UID)) + require.Len(t, s.keysLeft(), 2) + }) + t.Run("LIST still shows a key", func(t *testing.T) { + t.Parallel() + s := newKeySweepEnv(ctx, t) + createTestControlPlane(ctx, t, s.namespace, "coder-b", s.fake.server.URL) + // Both LISTs of the pass show a key that each DELETE answers 404 for, + // as a Coder that lists a key it cannot delete would. + ghost := codersdk.AIGatewayKey{ID: uuid.New(), Name: aiGatewayKeyPrefix(s.key) + "1"} + for range 2 { + s.fake.failNext(routeAIGatewayKeys, fakeFault{Rewrite: func(a any) any { return append(a.([]codersdk.AIGatewayKey), ghost) }}) + } + s.clock.Step(6 * time.Minute) + require.Equal(t, time.Minute, s.sweep(ctx, t).RequeueAfter, "only a LIST that shows no key of the object confirms") + require.NotNil(t, s.aiGatewayKeyRecord(ctx, t, s.key.UID)) + require.Zero(t, s.sweep(ctx, t).RequeueAfter) + require.Nil(t, s.aiGatewayKeyRecord(ctx, t, s.key.UID)) + }) + t.Run("operator not a site owner", func(t *testing.T) { + t.Parallel() + s := newKeySweepEnv(ctx, t) + createTestControlPlane(ctx, t, s.namespace, "coder-b", s.fake.server.URL) + s.fake.updateUser(s.fake.operatorID, func(u *codersdk.User) { u.Roles = []codersdk.SlimRole{} }) + s.clock.Step(6 * time.Minute) + require.Equal(t, time.Minute, s.sweep(ctx, t).RequeueAfter) + require.Zero(t, s.fake.requestCount(routeDeleteAIGatewayKey), "only a site owner's LIST shows every key") + require.NotNil(t, s.aiGatewayKeyRecord(ctx, t, s.key.UID)) + }) + t.Run("control plane in another namespace", func(t *testing.T) { + t.Parallel() + s := newKeySweepEnv(ctx, t) + createTestControlPlane(ctx, t, createTestNamespace(ctx, t, "aigk-other"), "coder", s.fake.server.URL) + s.clock.Step(6 * time.Minute) + require.Equal(t, 10*time.Minute, s.sweep(ctx, t).RequeueAfter) + require.Zero(t, s.fake.requestCount(routeDeleteAIGatewayKey), "only a control plane in the object's namespace sweeps") + }) +}