diff --git a/.coverage-baseline b/.coverage-baseline index 18a2c6385..f32b65c84 100644 --- a/.coverage-baseline +++ b/.coverage-baseline @@ -1 +1 @@ -80.2 +80.3 diff --git a/.gitignore b/.gitignore index c7dd73d12..d2f1bdfde 100644 --- a/.gitignore +++ b/.gitignore @@ -14,6 +14,10 @@ Dockerfile.cross # The GitOps layout fixtures check in a kustomize secretGenerator env file on purpose. # Its contents are placeholder text, never a real credential. !test/fixtures/gitops-layouts/**/*.env +# So do the configMapGenerator write-through fixtures: their env files are the input a live +# ConfigMap edit is written into. +!test/fixtures/layout-corpus/**/*.env +!test/e2e/fixtures/**/*.env # Test binary, built with `go test -c` *.test diff --git a/README.md b/README.md index ae7771500..715828459 100644 --- a/README.md +++ b/README.md @@ -76,13 +76,16 @@ repos, branches, and folders. See [Configuration](docs/configuration.md). | Your source in Git | What you can do | |---|---| | Plain Kubernetes manifests | Capture resources and update their existing YAML documents | -| Supported Kustomize layouts | Update source manifests or image/replica declarations; add and remove resources | +| Supported Kustomize layouts | Update source manifests, image/replica declarations, and `configMapGenerator` input files; add and remove resources | | A Flux `HelmRelease` or Argo CD `Application` | Capture the declaration, including chart versions and inline values | | Helm templates or standalone `values.yaml` | No writeback from rendered workloads | -Local Kustomize bases and overlays work, with the base kept read-only. Generators, components, and -remote bases are unsupported; such a folder is refused with `Stalled=True` before anything is -written. See the [supported subset](docs/configuration.md#kustomize-support-in-the-target-path). +Local Kustomize bases and overlays work, with the base kept read-only. A `configMapGenerator` with +`disableNameSuffixHash: true` is writable: a ConfigMap edit lands in the file, `.env` line or +literal it came from, and deleting the ConfigMap removes its generator entry. `secretGenerator`, +hash-suffixed generators, components, and remote bases are unsupported; such a folder is refused +with `Stalled=True` before anything is written. See the [supported +subset](docs/configuration.md#kustomize-support-in-the-target-path). Watch the resources that express your intent, such as a `HelmRelease` rather than the workloads it renders: the operator cannot tell authored resources from generated ones. See diff --git a/docs/architecture.md b/docs/architecture.md index 779876c03..50c1e6879 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1391,7 +1391,8 @@ write side is shared with live writes): - before anything is planned, a **structure-only acceptance gate** runs over the scanned subtree; if it finds content the operator cannot safely manage: a kustomization using an unsupported feature - (generators / inline or JSON6902 patches / components / helm / replacements / transformers / + (`secretGenerator` or a hash-suffixed or merging `configMapGenerator` / inline or JSON6902 patches / + components / helm / replacements / transformers / name(pre|suf)fix / remote bases) or malformed `images:`/`replicas:` overrides, a duplicate manifest identity, an impure managed file, or a standalone non-KRM / invalid YAML; (a path-based strategic-merge `patches:` entry is tolerated as read-only build context, and an diff --git a/docs/bi-directional.md b/docs/bi-directional.md index 724aafc57..81ec57ba3 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -408,7 +408,9 @@ Argo CD stamps `argocd.argoproj.io/tracking-id` on every non-CRD object it appli the live object would stop matching the committed file, and every apply would produce a commit. GitOps Reverser strips all of it before writing, along with `argocd.argoproj.io/installation-id`, -`kro.run/`, `applyset.kubernetes.io/`, and `kcp.io/cluster`; the rules and reasons are in +`kro.run/`, `applyset.kubernetes.io/`, `kcp.io/cluster`, and kustomize's build provenance +(`config.kubernetes.io/origin`, `alpha.config.kubernetes.io/transformations`), which Flux requests +for a `kustomization.yaml` with no `resources:`; the rules and reasons are in [`internal/sanitize/types.go`](../internal/sanitize/types.go). Under `argocd.argoproj.io/` only the exact tracking keys go: `sync-wave`, `sync-options` and `hook` are user intent that belongs in Git. That is also why the loop settles: an applied commit GitOps Reverser wrote matches the live object, diff --git a/docs/configuration.md b/docs/configuration.md index d75d269a3..821f15921 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -2650,6 +2650,17 @@ A target path may contain `kustomization.yaml` files. The operator retains them keeps its bytes. Only fields the entry already declares are updated; the operator never adds or removes entries. Note that one entry is a shared knob, exactly as in kustomize itself: updating it affects every resource in the build whose image matches. +- **`configMapGenerator` with `disableNameSuffixHash: true`** (in `generatorOptions:` or on the + entry): a live change to the generated ConfigMap is written into the generator's inputs. A + `files:` key's file gets the live value as its content; an `envs:` key's line and a `literals:` + key's `KEY=value` item are edited in place, and a removed key's line or item is removed. A new key + is appended to the entry's `envs:` file when it has exactly one, and otherwise to its `literals:` + when it has some. Deleting the ConfigMap removes the entry from `configMapGenerator:`, and the + input files no other entry, resource or patch reads; under `spec.prune.mode: Always` a resync + sweeps the entry of a generated ConfigMap the cluster no longer has, as it sweeps a document. A + change with no single input to land in is refused per object: `binaryData`, a label or annotation, + a new key with several `envs:` files or only `files:` sources, a removed `files:` key, and a + generator in a read-only base. The input files are build inputs, never foreign content. Two kustomize shapes beyond that subset are **supported without being authored**: @@ -2662,7 +2673,9 @@ Two kustomize shapes beyond that subset are **supported without being authored** Everything else outside the modeled subset **refuses the whole target path before anything is written**: inline or JSON6902 patches and the deprecated `patchesStrategicMerge`/`patchesJson6902` -spellings, generators, `components`, Helm fields, `replacements`, `transformers`, +spellings, `secretGenerator`, a `configMapGenerator` whose name hash is not disabled (the message +says to set `disableNameSuffixHash: true`) or whose `behavior:` is `merge` or `replace`, +`components`, Helm fields, `replacements`, `transformers`, `namePrefix`/`nameSuffix`, remote bases, and `images:`/`replicas:` values that do not parse (those would fail `kustomize build` too). A refusal is loud: the target reports `GitPathAccepted=False`, `Stalled=True`, and `Ready=False` with reason `UnsupportedContent` until the path is cleaned up. diff --git a/docs/design/support-boundary/kustomize-support-boundary.md b/docs/design/support-boundary/kustomize-support-boundary.md index 11dd5b728..66c91df07 100644 --- a/docs/design/support-boundary/kustomize-support-boundary.md +++ b/docs/design/support-boundary/kustomize-support-boundary.md @@ -13,9 +13,10 @@ language. | Class | Fields | Boundary | |---|---|---| | Editable declarations | local `resources`/`bases`, `namespace`, `images`, `replicas` | supported where the layout provides one write home | +| Editable inputs | `configMapGenerator` with `disableNameSuffixHash: true` and no `behavior: merge`/`replace` | an edit lands in the `files:` file, on the `envs:` line or in the `literals:` item it came from; a delete removes the entry | | Tolerated build context | `patches:` with local strategic-merge `path:` documents | rendered and retained, but neither authored nor edited | | Planned authoring surface | scalar strategic-merge patch for an inherited field | see [patch authoring](patch-authoring.md) | -| Refused | inline/JSON6902/deprecated patches, generators, `components`, `namePrefix`/`nameSuffix`, remote bases, `helmCharts`, plugins, `replacements`, `vars` | no safe or owned inverse in the current model | +| Refused | inline/JSON6902/deprecated patches, `secretGenerator`, hash-suffixed or merging `configMapGenerator`, `components`, `namePrefix`/`nameSuffix`, remote bases, `helmCharts`, plugins, `replacements`, `vars` | no safe or owned inverse in the current model | | Refused for editor safety | `configurations`, `openapi`, `crds` | they alter merge semantics that the writer must not guess | `labels`, `commonLabels`, and `commonAnnotations` are not field-authoring channels. @@ -92,7 +93,8 @@ described in [patch authoring](patch-authoring.md) exist. - No chart inflation or remote-base resolution. - No plugin execution or controller-side transform emulation. - No write-through to a shared base or another environment. -- No generic inversion of patches, generators, name references, or merge keys. +- No generic inversion of patches, name references, or merge keys, and no generator beyond a + hash-disabled `configMapGenerator`. The fixture corpus under [`test/fixtures/gitops-layouts/`](../../../test/fixtures/gitops-layouts/) is the executable evidence for these verdicts. The more detailed history of the renderer and diff --git a/docs/design/support-boundary/support-contract.md b/docs/design/support-boundary/support-contract.md index 608be2009..dfe5206cb 100644 --- a/docs/design/support-boundary/support-contract.md +++ b/docs/design/support-boundary/support-contract.md @@ -86,7 +86,8 @@ templates, and multi-input ResourceSet templates alike. It is specified once, in | kustomize `patches:` — a strategic-merge document named by `path:` | **Tolerated, not authored** | the folder is accepted and the render is mirrored; the patch is read-only build context. An edit to a field the patch OWNS is refused per object, not per folder | | kustomize `patches:` — inline, JSON6902, or a path outside the tree | **Refused** | not a sparse KRM document we can read; refused by name | | kustomize `patchesStrategicMerge`, `patchesJson6902` (deprecated spellings) | **Refused** | kustomize does not fold them into `patches:` (measured), so they refuse under their own names | -| kustomize generators, `components`, `namePrefix`/`nameSuffix`, remote bases | **Refused** | non-invertible | +| kustomize `configMapGenerator` with `disableNameSuffixHash: true` | **Editable** | the edit lands in the `files:` file, on the `envs:` line or in the `literals:` item it came from, and a delete removes the entry, checked by re-render; a change with no single input is refused per object | +| kustomize `secretGenerator`, hash-suffixed or merging `configMapGenerator`, `components`, `namePrefix`/`nameSuffix`, remote bases | **Refused** | non-invertible | | kustomize `helmCharts:` (inflation) | **Refused** | we never render a chart | | A Helm chart (`Chart.yaml` + `templates/` + `values.yaml` + `crds/`) | **Planned: skipped as a unit** | this needs chart-folder detection; today `scan-repo` can report an incidental `crds/` directory as accepted | | Helm knobs on a `HelmRelease` / `Application` (chart version, inline values, parameters) | **Editable** | the Helm surface people actually use — see below | diff --git a/docs/design/support-boundary/values-file-projection.md b/docs/design/support-boundary/values-file-projection.md index 50f4e0cc5..dac12ac71 100644 --- a/docs/design/support-boundary/values-file-projection.md +++ b/docs/design/support-boundary/values-file-projection.md @@ -40,11 +40,11 @@ The same Helm values, expressed six ways in the corpus, get five different verdi | `HelmRelease.spec.values` inline, `valuesFrom` → a hand-written `ConfigMap` | `flux-helmrelease/infrastructure/controllers/ingress-nginx/` | **Editable** | | a `values.yaml` alone in a `values/` directory | `argocd-external-helm/values/**` | **Not seen at all** — no KRM in the dir, so never a candidate | | a `values.yaml` **next to** KRM | `argocd-external-helm/platform/cert-manager/values.yaml` | **Refused** — and it takes the folder with it | -| a `values.yaml` wrapped by a `configMapGenerator` | `flux-helmrelease/apps/frontend/` | **Refused** — the generator is on the deny-list | +| a `values.yaml` wrapped by a `configMapGenerator` | `flux-helmrelease/apps/frontend/` | **Editable** — with `disableNameSuffixHash: true`, the file is the generator's input | -Read that column again. Values are editable when a human wraps them in a `ConfigMap`, and -refused when `configMapGenerator` does the identical wrapping — with -`disableNameSuffixHash: true`, so the result is deterministic. They are editable as a *string* +Read that column again. Values are editable when a human wraps them in a `ConfigMap`, and, since +the generator write-through, when a `configMapGenerator` with `disableNameSuffixHash: true` does +the identical wrapping: the live edit lands in the file. They are editable as a *string* inside an Application, and refused as a *file* on disk. They are invisible when alone and fatal when adjacent. diff --git a/docs/facts/expansion-provenance-markers.md b/docs/facts/expansion-provenance-markers.md index c5c91958c..480c601ae 100644 --- a/docs/facts/expansion-provenance-markers.md +++ b/docs/facts/expansion-provenance-markers.md @@ -88,7 +88,8 @@ belongs in the evidence table, not as a replacement for it. [`internal/sanitize/types.go`](../../internal/sanitize/types.go) strips `kustomize.toolkit.fluxcd.io/` (labels + annotations), `kro.run/` (labels only), `applyset.kubernetes.io/`, and — by exact key — `argocd.argoproj.io/tracking-id`, -`argocd.argoproj.io/installation-id`, and `kcp.io/cluster`. +`argocd.argoproj.io/installation-id`, `kcp.io/cluster`, `config.kubernetes.io/origin`, and +`alpha.config.kubernetes.io/transformations`. It does **not** strip, and these are all controller bookkeeping observed on real objects: diff --git a/internal/git/generator_write.go b/internal/git/generator_write.go new file mode 100644 index 000000000..9d243e906 --- /dev/null +++ b/internal/git/generator_write.go @@ -0,0 +1,184 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +import ( + "context" + "errors" + "fmt" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "sigs.k8s.io/controller-runtime/pkg/log" + + "github.com/ConfigButler/gitops-reverser/internal/git/manifestedit" + "github.com/ConfigButler/gitops-reverser/internal/manifestanalyzer" + "github.com/ConfigButler/gitops-reverser/internal/manifestreport" + "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// writeGenerated writes a live ConfigMap that a configMapGenerator renders back into the +// generator's input files: a files: source gets the live value as its content, an envs: line is +// edited in place. manifestanalyzer.PlanGeneratedWrite decides where each key goes; the oracle +// then re-renders the folder and refuses the flush unless the generator produces the live object. +// +// A change with no destination (binaryData, a label, a key with no single input to land in) is +// refused rather than written somewhere else: a new ConfigMap document beside the generator would +// collide with it in the build. +func (wb *writeBatch) writeGenerated( + ctx context.Context, + event Event, + generated []*manifestanalyzer.GeneratedObject, +) (upsertOutcome, error) { + live := event.Object + gen := generated[0] + if len(generated) > 1 { + return upsertNoChange, generatedRefusal(live, gen.GeneratedIn, fmt.Errorf( + "%d configMapGenerator entries render an object of this name", len(generated))) + } + if wb.writer.isSensitiveIdentifier(event.Identifier) { + // A generator input is plaintext; a resource the target encrypts must never land in one. + return upsertNoChange, generatedRefusal(live, gen.GeneratedIn, errors.New( + "the resource is sensitive, and a generator input is a plaintext file")) + } + + desired := manifestreport.Project(live) + // The same render-fidelity fence a document's edit passes: a ${...} token the input holds that + // the live object carries expanded was substituted downstream (Flux postBuild), and writing the + // expansion back into the input is the corruption no re-render can see, since it then matches. + divergences := manifestanalyzer.RenderTokenDivergences(gen.Rendered.Object, desired.Object) + if len(divergences) > 0 { + id := manifestedit.Identity{Kind: desired.GetKind(), Name: desired.GetName()} + return upsertNoChange, renderFidelityRefusal( + gen.GeneratedIn, id, &renderFidelityRefusedError{Divergences: divergences}) + } + writes, err := manifestanalyzer.PlanGeneratedWrite(gen, desired, wb.writeSubdir, wb.currentContent) + if err != nil { + return upsertNoChange, generatedRefusal(live, gen.GeneratedIn, err) + } + for _, w := range writes { + wb.buffer(w.Path).current = w.Content + log.FromContext(ctx).Info("Routed live change to configMapGenerator input", + "kustomization", gen.GeneratedIn, "file", w.Path, "resource", event.Identifier.String()) + } + + // Declared whether or not this event wrote anything, as for a governed document: an input + // may be shared with another entry an earlier event already wrote, and only the intent tells + // the oracle that this object's move is convergence. Governed, because a generated object no + // render root produces after the write is a contradiction, not a skip. + wb.intend(manifestanalyzer.WriteIntent{ + SourcePath: gen.GeneratedIn, + Kind: desired.GetKind(), + Name: desired.GetName(), + Desired: desired, + Governed: true, + }) + if len(writes) == 0 { + return upsertNoChange, nil + } + wb.putToKustomize = true + return upsertUpdated, nil +} + +// deleteGenerated mirrors a live delete of an object a configMapGenerator renders; see +// removeGenerated. A delete no generator renders is a no-op. +func (wb *writeBatch) deleteGenerated(ctx context.Context, event Event) error { + obj := event.Object + if obj == nil { + id := event.Identifier + if id.Group != "" || id.Resource != "configmaps" { + return nil + } + obj = objectNamed(manifestedit.Identity{ + APIVersion: "v1", Kind: "ConfigMap", Namespace: id.Namespace, Name: id.Name, + }) + } + _, err := wb.removeGenerated(ctx, event.Identifier, obj, documentDeletedLive) + return err +} + +// removeGenerated takes an object a configMapGenerator renders out of the folder: the entry leaves +// configMapGenerator:, and the input files only it reads are deleted, so the next render does not +// bring the object back. The oracle proves the object left the render and nothing else moved. It +// is the one removal the live delete and the resync sweep share, as removeDocument is for a +// document, and it reports whether it removed anything: an object no generator renders, or one +// this batch already removed, is a no-op. +func (wb *writeBatch) removeGenerated( + ctx context.Context, + identifier types.ResourceIdentifier, + obj *unstructured.Unstructured, + outcome string, +) (bool, error) { + generated := wb.store.GeneratedObjectsFor(obj) + switch { + case len(generated) == 0: + return false, nil + case len(generated) > 1: + return false, generatedRefusal(obj, generated[0].GeneratedIn, fmt.Errorf( + "%d configMapGenerator entries render an object of this name", len(generated))) + } + gen := generated[0] + if wb.removedGenerated[gen] { + return false, nil + } + writes, err := manifestanalyzer.PlanGeneratedDelete(gen, wb.writeSubdir, wb.currentContent) + if err != nil { + return false, generatedRefusal(obj, gen.GeneratedIn, err) + } + for _, w := range writes { + buf := wb.buffer(w.Path) + buf.current = w.Content + if w.Path == gen.GeneratedIn { + wb.recordDocumentRemoval(buf) + } + } + if wb.removedGenerated == nil { + wb.removedGenerated = map[*manifestanalyzer.GeneratedObject]bool{} + } + wb.removedGenerated[gen] = true + wb.intend(manifestanalyzer.WriteIntent{ + SourcePath: gen.GeneratedIn, + Kind: obj.GetKind(), + Name: obj.GetName(), + Removed: true, + }) + wb.putToKustomize = true + wb.tallyDocument(identifier, outcome) + log.FromContext(ctx).Info("Removed configMapGenerator entry for a deleted ConfigMap", + "kustomization", gen.GeneratedIn, "files", len(writes)-1, "resource", identifier.String()) + return true, nil +} + +// objectNamed is a bodiless object with the given identity, enough to look a generated object up. +func objectNamed(id manifestedit.Identity) *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetAPIVersion(id.APIVersion) + obj.SetKind(id.Kind) + obj.SetNamespace(id.Namespace) + obj.SetName(id.Name) + return obj +} + +// generatedRefusal reports a generated object the writer cannot write back, as the write-boundary +// refusal every unplaceable edit surfaces as, naming the object and the kustomization. +func generatedRefusal(obj *unstructured.Unstructured, kustomization string, err error) error { + return &manifestanalyzer.AcceptanceRefusedError{ + Issues: []manifestanalyzer.AcceptanceIssue{{ + Kind: manifestanalyzer.IssueUnplaceableEdit, + Path: kustomization, + // Not solvable from either side: the change itself has nowhere to land. + Solvable: false, + Message: fmt.Sprintf("%s/%s generated in %s: %v", + obj.GetKind(), obj.GetName(), kustomization, err), + }}, + } +} + +// currentContent is a file's bytes as the batch has left them so far, without hydrating a buffer +// for it. +func (wb *writeBatch) currentContent(rel string) ([]byte, bool) { + if b, ok := wb.buffers[rel]; ok { + return b.current, b.current != nil + } + content, ok := wb.contentByPath[rel] + return content, ok +} diff --git a/internal/git/generator_write_test.go b/internal/git/generator_write_test.go new file mode 100644 index 000000000..efb44648e --- /dev/null +++ b/internal/git/generator_write_test.go @@ -0,0 +1,253 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +import ( + "os" + "path/filepath" + "strings" + "testing" + + gogit "github.com/go-git/go-git/v6" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + sigsyaml "sigs.k8s.io/yaml" + + v1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + + "github.com/ConfigButler/gitops-reverser/internal/manifestanalyzer" + "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// The write path through a configMapGenerator. The corpus (shape 9) pins one diff or one refusal +// per rule; these are the cases a single-event fixture cannot state. + +const generatorKustomizationYAML = `apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: coffee +resources: + - deployment.yaml +configMapGenerator: + - name: coffee-flags + envs: + - flags.env +generatorOptions: + disableNameSuffixHash: true +` + +const generatorFlagsEnv = "# Turn off during the conference rush.\nENABLE_VOUCHERS=true\nSHOW_BANNER=true\n" + +const generatorDeploymentYAML = `apiVersion: apps/v1 +kind: Deployment +metadata: + name: coffee +spec: + selector: + matchLabels: + app: coffee + template: + metadata: + labels: + app: coffee + spec: + containers: + - name: coffee + image: ghcr.io/example/coffee:1.0.0 +` + +// seedGeneratorWorktree writes a kustomize folder whose ConfigMap a generator renders from an +// env file, beside a Deployment the kustomization lists, and returns the env file's path. +func seedGeneratorWorktree(t *testing.T) (*gogit.Worktree, string) { + t.Helper() + worktree := newWorktreeForTest(t) + root := worktree.Filesystem().Root() + seedFile(t, root, "kustomization.yaml", generatorKustomizationYAML) + seedFile(t, root, "deployment.yaml", generatorDeploymentYAML) + seedFile(t, root, "flags.env", generatorFlagsEnv) + commitCorpusWorktree(t, worktree, "seed a generator folder") + return worktree, filepath.Join(root, "flags.env") +} + +func flagsEvent(operation string, data map[string]interface{}) Event { + event := Event{ + Identifier: types.NewResourceIdentifier("", "v1", "configmaps", "coffee", "coffee-flags"), + Operation: operation, + } + if data != nil { + event.Object = &unstructured.Unstructured{Object: map[string]interface{}{ + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": map[string]interface{}{"name": "coffee-flags", "namespace": "coffee"}, + "data": data, + }} + } + return event +} + +// Two edits of one generated ConfigMap in one batch: the second plans against the bytes the +// first left, so both land and neither is reverted. +func TestPlanFlush_TwoEditsOfOneGeneratedConfigMapInOneBatch(t *testing.T) { + worktree, envPath := seedGeneratorWorktree(t) + + changed, err := flushEventsForTest(t, newContentWriter(types.SensitiveResourcePolicy{}), worktree, corpusMapper(), + flagsEvent("UPDATE", map[string]interface{}{"ENABLE_VOUCHERS": "false", "SHOW_BANNER": "true"}), + flagsEvent("UPDATE", map[string]interface{}{"ENABLE_VOUCHERS": "false", "SHOW_BANNER": "false"}), + ) + require.NoError(t, err) + assert.True(t, changed) + assertFileBytes(t, envPath, + "# Turn off during the conference rush.\nENABLE_VOUCHERS=false\nSHOW_BANNER=false\n", + "both edits land on their own lines") +} + +// A live change to the Deployment beside a generator is written as before, with the generated +// ConfigMap in the render the oracle holds to not moving. +func TestPlanFlush_ANeighbourOfAGeneratorStaysWritable(t *testing.T) { + worktree, envPath := seedGeneratorWorktree(t) + deployment := sharedImageEvent("coffee", "ghcr.io/example/coffee:1.1.0") + deployment.Object.SetNamespace("coffee") + deployment.Identifier.Namespace = "coffee" + deployment.Object.Object["spec"] = map[string]interface{}{ + "selector": map[string]interface{}{"matchLabels": map[string]interface{}{"app": "coffee"}}, + "template": map[string]interface{}{ + "metadata": map[string]interface{}{"labels": map[string]interface{}{"app": "coffee"}}, + "spec": map[string]interface{}{"containers": []interface{}{ + map[string]interface{}{"name": "coffee", "image": "ghcr.io/example/coffee:1.1.0"}, + }}, + }, + } + + changed, err := flushEventsForTest(t, newContentWriter(types.SensitiveResourcePolicy{}), worktree, corpusMapper(), + deployment) + require.NoError(t, err) + assert.True(t, changed) + assertFileBytes( + t, + filepath.Join(worktree.Filesystem().Root(), "deployment.yaml"), + strings.Replace( + generatorDeploymentYAML, + "coffee:1.0.0", + "coffee:1.1.0", + 1, + ), + "the Deployment is edited in place", + ) + assertFileBytes(t, envPath, generatorFlagsEnv, "the generator's input is untouched") +} + +// A deleted generated ConfigMap takes its generator entry and the env file only that entry reads +// with it, whether the event carries the last object or only the identity. With its one entry +// gone, configMapGenerator: goes too. +func TestPlanFlush_ADeletedGeneratedConfigMapRemovesItsEntry(t *testing.T) { + for name, event := range map[string]Event{ + "with the last object": flagsEvent("DELETE", map[string]interface{}{"ENABLE_VOUCHERS": "true"}), + "by identity only": flagsEvent("DELETE", nil), + } { + t.Run(name, func(t *testing.T) { + worktree, envPath := seedGeneratorWorktree(t) + changed, err := flushEventsForTest(t, newContentWriter(types.SensitiveResourcePolicy{}), worktree, + corpusMapper(), event) + require.NoError(t, err) + assert.True(t, changed) + assert.NoFileExists(t, envPath) + assertFileBytes(t, filepath.Join(worktree.Filesystem().Root(), "kustomization.yaml"), + `apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: coffee +resources: + - deployment.yaml +generatorOptions: + disableNameSuffixHash: true +`, "the entry and the emptied configMapGenerator: are gone") + }) + } +} + +// A ConfigMap the target treats as sensitive is never written into a plaintext input file. +func TestPlanFlush_ASensitiveGeneratedConfigMapIsRefused(t *testing.T) { + policy, err := types.ParseSensitiveResourcePolicy("configmaps") + require.NoError(t, err) + worktree, envPath := seedGeneratorWorktree(t) + + _, err = flushEventsForTest(t, newContentWriter(policy), worktree, corpusMapper(), + flagsEvent("UPDATE", map[string]interface{}{"ENABLE_VOUCHERS": "false", "SHOW_BANNER": "true"})) + + var refused *manifestanalyzer.AcceptanceRefusedError + require.ErrorAs(t, err, &refused) + assert.Contains(t, refused.Error(), "a generator input is a plaintext file") + assertFileBytes(t, envPath, generatorFlagsEnv, "no plaintext reaches the input") +} + +// A ${...} token in an input that the live object carries expanded was substituted downstream, by +// Flux postBuild. Writing the expansion back would match the re-render and still corrupt the +// input, so the write is refused as the render-fidelity divergence it is. +func TestPlanFlush_AGeneratedConfigMapWithAnExpandedTokenIsRefused(t *testing.T) { + worktree, envPath := seedGeneratorWorktree(t) + tokenEnv := "HOST=shop.${cluster_domain}\nENABLE_VOUCHERS=true\n" + seedFile(t, worktree.Filesystem().Root(), "flags.env", tokenEnv) + + _, err := flushEventsForTest(t, newContentWriter(types.SensitiveResourcePolicy{}), worktree, corpusMapper(), + flagsEvent("UPDATE", map[string]interface{}{"HOST": "shop.example.com", "ENABLE_VOUCHERS": "false"})) + + var refused *manifestanalyzer.AcceptanceRefusedError + require.ErrorAs(t, err, &refused) + assert.True(t, refused.AllIssuesOfKinds(manifestanalyzer.IssueRenderDoesNotMatchLive)) + assert.Contains(t, refused.Error(), "${cluster_domain}") + assertFileBytes(t, envPath, tokenEnv, "the token stays in the input") +} + +// resyncGenerator folds a complete desired snapshot over the generator folder under one prune +// mode. The Deployment is always in it, so only the generated ConfigMap can be an orphan. +func resyncGenerator( + t *testing.T, + worktree *gogit.Worktree, + mode v1alpha3.PruneMode, + withFlags bool, +) ResyncStats { + t.Helper() + deployment := &unstructured.Unstructured{} + require.NoError(t, sigsyaml.Unmarshal([]byte(generatorDeploymentYAML), &deployment.Object)) + deployment.SetNamespace("coffee") + desired := []manifestanalyzer.DesiredResource{{ + Resource: types.NewResourceIdentifier("apps", "v1", "deployments", "coffee", "coffee"), + Object: deployment, + }} + if withFlags { + flags := flagsEvent("UPDATE", map[string]interface{}{"ENABLE_VOUCHERS": "true", "SHOW_BANNER": "true"}) + desired = append(desired, manifestanalyzer.DesiredResource{Resource: flags.Identifier, Object: flags.Object}) + } + w := &BranchWorker{contentWriter: newContentWriter(types.SensitiveResourcePolicy{}), mapper: corpusMapper()} + stats, _, err := w.applyResyncToWorktree( + t.Context(), worktree, "", ResolvedTargetMetadata{PruneMode: mode}, desired, nil) + require.NoError(t, err) + return stats +} + +// A generated ConfigMap is as much the folder's content as a document is, so a resync that sweeps +// removes the generator entry of one the cluster no longer has, keeps it while the cluster has +// it, and only counts it as retained under a mode that does not sweep. +func TestResync_SweepsAGeneratedConfigMapTheClusterNoLongerHas(t *testing.T) { + t.Run("always, absent: the entry and its env file go", func(t *testing.T) { + worktree, envPath := seedGeneratorWorktree(t) + stats := resyncGenerator(t, worktree, v1alpha3.PruneAlways, false) + assert.Equal(t, 1, stats.Deleted) + assert.NoFileExists(t, envPath) + kustomization, err := os.ReadFile(filepath.Join(worktree.Filesystem().Root(), "kustomization.yaml")) + require.NoError(t, err) + assert.NotContains(t, string(kustomization), "coffee-flags") + }) + t.Run("always, present: kept", func(t *testing.T) { + worktree, envPath := seedGeneratorWorktree(t) + stats := resyncGenerator(t, worktree, v1alpha3.PruneAlways, true) + assert.Zero(t, stats.Deleted) + assertFileBytes(t, envPath, generatorFlagsEnv, "a generated object the cluster has is matched") + }) + t.Run("onEvent, absent: retained and counted", func(t *testing.T) { + worktree, envPath := seedGeneratorWorktree(t) + stats := resyncGenerator(t, worktree, v1alpha3.PruneOnEvent, false) + assert.Zero(t, stats.Deleted) + assert.Equal(t, 1, stats.Retained) + assertFileBytes(t, envPath, generatorFlagsEnv, "onEvent never sweeps") + }) +} diff --git a/internal/git/kustomize_oracle_test.go b/internal/git/kustomize_oracle_test.go index 93493c34f..e46abdb91 100644 --- a/internal/git/kustomize_oracle_test.go +++ b/internal/git/kustomize_oracle_test.go @@ -102,7 +102,7 @@ func seedSharedEntryWorktree(t *testing.T, root string) (string, string, string) // sharedImageEvent is the live Deployment as the cluster holds it: the rendered form, // which is what the source file plus the overlay's entry must reproduce. -func sharedImageEvent(name, image string) Event { //nolint:unparam // image varies by intent, not by call today +func sharedImageEvent(name, image string) Event { return Event{ Object: &unstructured.Unstructured{Object: map[string]interface{}{ "apiVersion": "apps/v1", diff --git a/internal/git/layout_corpus_test.go b/internal/git/layout_corpus_test.go index 873f84d93..3fff48a36 100644 --- a/internal/git/layout_corpus_test.go +++ b/internal/git/layout_corpus_test.go @@ -96,6 +96,9 @@ type corpusScenario struct { // TestGitTargetReadiness_StalledFollowsGitPathAccepted), so the whole file is covered even // though no single test covers all of it. status string + // operation is the event's operation; empty is "CREATE". "DELETE" folds the input in as the + // object's removal. + operation string // emptyRepository seeds NOTHING, for a scenario whose whole subject is a folder that does not // exist yet. It is a flag rather than an empty `repository-empty/` fixture directory because // Git cannot hold an empty directory, and a .gitkeep inside one would show up in the diff the @@ -227,6 +230,99 @@ func layoutCorpus() []corpusScenario { input: "deployment-env-changed.yaml", status: "expected-env-change-status.yaml", }, + { + // A settings file wrapped by a configMapGenerator: the live value replaces the file's + // bytes, so the comment survives as part of the value and the diff is one line. + dir: "shapes/9-kustomize-config-map-generator", + input: "menu-price-changed.yaml", + patch: "expected-menu-price-changed.patch", + }, + { + // An envs: file: the changed key's line is edited in place. + dir: "shapes/9-kustomize-config-map-generator", + input: "flags-value-changed.yaml", + patch: "expected-flags-value-changed.patch", + }, + { + // A new key, and exactly one env file to append it to. + dir: "shapes/9-kustomize-config-map-generator", + input: "flags-key-added.yaml", + patch: "expected-flags-key-added.patch", + }, + { + // A removed key: its line goes, and every other line keeps its bytes. + dir: "shapes/9-kustomize-config-map-generator", + input: "flags-key-removed.yaml", + patch: "expected-flags-key-removed.patch", + }, + { + // A literals: key: its KEY=value item in kustomization.yaml is edited in place. + dir: "shapes/9-kustomize-config-map-generator", + input: "branding-literal-changed.yaml", + patch: "expected-branding-literal-changed.patch", + }, + { + // A new key on an entry with literals: and no env file joins its literals:. + dir: "shapes/9-kustomize-config-map-generator", + input: "branding-key-added.yaml", + patch: "expected-branding-key-added.patch", + }, + { + // A deleted ConfigMap: its entry leaves configMapGenerator:, and the env file only that + // entry reads goes with it, so the next render does not bring the object back. + dir: "shapes/9-kustomize-config-map-generator", + input: "flags-deleted.yaml", + operation: "DELETE", + patch: "expected-flags-deleted.patch", + }, + { + // The refusal halves of shape 9. Each change has no single input to land in, and a + // ConfigMap document beside the generator would collide with it in the build. + dir: "shapes/9-kustomize-config-map-generator", + input: "menu-key-removed.yaml", + status: "expected-menu-key-removed-status.yaml", + }, + { + dir: "shapes/9-kustomize-config-map-generator", + input: "menu-key-added.yaml", + status: "expected-menu-key-added-status.yaml", + }, + { + dir: "shapes/9-kustomize-config-map-generator", + input: "regions-key-added.yaml", + status: "expected-regions-key-added-status.yaml", + }, + { + dir: "shapes/9-kustomize-config-map-generator", + input: "branding-binary-changed.yaml", + status: "expected-branding-binary-changed-status.yaml", + }, + { + dir: "shapes/9-kustomize-config-map-generator", + input: "branding-label-changed.yaml", + status: "expected-branding-label-changed-status.yaml", + }, + { + // The name hash is the one generator refusal the author can clear: the message names + // the setting. + dir: "shapes/9-kustomize-config-map-generator", + config: "gittarget-hashed.yaml", + input: "hashed-menu-changed.yaml", + status: "expected-hashed-status.yaml", + }, + { + // The generator lives in the base the overlay reads: base-owned, as any base document. + dir: "shapes/9-kustomize-config-map-generator", + config: "gittarget-prod.yaml", + input: "shop-menu-changed.yaml", + status: "expected-base-owned-status.yaml", + }, + { + dir: "shapes/9-kustomize-config-map-generator", + config: "gittarget-staging.yaml", + input: "shop-staging-menu-changed.yaml", + status: "expected-behavior-merge-status.yaml", + }, { dir: "specific-examples/homelab-argocd", input: "paperless.yaml", @@ -299,6 +395,9 @@ func runCorpusScenario(t *testing.T, sc corpusScenario) { worktree, seeded := seedCorpusWorktree(t, folder, sc) event := corpusEvent(t, obj, target) + if sc.operation != "" { + event.Operation = sc.operation + } sources, wildcard := readCorpusSourceNamespaces(t, folder, sc.configFile(), target) diff --git a/internal/git/manifestedit/kustomization.go b/internal/git/manifestedit/kustomization.go index 8ff4ee7da..d4b0b7e67 100644 --- a/internal/git/manifestedit/kustomization.go +++ b/internal/git/manifestedit/kustomization.go @@ -3,6 +3,7 @@ package manifestedit import ( + "errors" "fmt" "strings" @@ -472,3 +473,185 @@ func setOverrideScalar(n *yaml.Node, e KustomizationEdit) { n.Style = 0 } } + +// LiteralEdit is one change to a configMapGenerator entry's literals: list. Remove drops the +// KEY=... item; otherwise the item is set to Value, appended when the entry has none for Key. +type LiteralEdit struct { + Key string + Value string + Remove bool +} + +// EditConfigMapGeneratorLiterals applies literal edits to configMapGenerator[entryIndex], which +// must be named entryName, preserving comments and key order. An item is edited in place, so its +// position in the list and every other item keep their bytes. All-or-nothing like its siblings: +// a drifted entry skips the whole call with a diagnostic. +func EditConfigMapGeneratorLiterals( + path string, + content []byte, + entryIndex int, + entryName string, + edits []LiteralEdit, +) (EditResult, []Diagnostic) { + return editGeneratorEntry(path, content, entryIndex, entryName, func(_, entry *yaml.Node) error { + return applyLiteralEdits(entry, edits) + }) +} + +// RemoveConfigMapGeneratorEntry drops configMapGenerator[entryIndex], which must be named +// entryName. An emptied configMapGenerator: drops the key, as an emptied patches: does. +func RemoveConfigMapGeneratorEntry( + path string, + content []byte, + entryIndex int, + entryName string, +) (EditResult, []Diagnostic) { + return editGeneratorEntry(path, content, entryIndex, entryName, func(root, _ *yaml.Node) error { + seq := nodeMapGet(root, configMapGeneratorKey) + seq.Content = append(seq.Content[:entryIndex], seq.Content[entryIndex+1:]...) + if len(seq.Content) == 0 { + nodeMapDelete(root, configMapGeneratorKey) + } + return nil + }) +} + +const configMapGeneratorKey = "configMapGenerator" + +// editGeneratorEntry locates configMapGenerator[entryIndex], verifies its name, runs edit on it, +// and re-encodes the kustomization. +func editGeneratorEntry( + path string, + content []byte, + entryIndex int, + entryName string, + edit func(root, entry *yaml.Node) error, +) (EditResult, []Diagnostic) { + skip := func(format string, args ...interface{}) (EditResult, []Diagnostic) { + return EditResult{Content: content, Mode: EditSkipped}, + []Diagnostic{diag(DiagWarning, Location{Path: path}, format, args...)} + } + docs, idx, root, reason, ok := locateKustomizationDocument(path, content) + if !ok { + return skip("%s", reason) + } + target := docs[idx].body + seq := nodeMapGet(root, configMapGeneratorKey) + if seq == nil || seq.Kind != yaml.SequenceNode || entryIndex < 0 || entryIndex >= len(seq.Content) { + return skip("kustomization %s has no configMapGenerator entry %d", path, entryIndex) + } + entry := seq.Content[entryIndex] + if entry.Kind != yaml.MappingNode { + return skip("kustomization %s: configMapGenerator entry %d is not a mapping", path, entryIndex) + } + if name := nodeMapGet(entry, "name"); name == nil || name.Value != entryName { + return skip("kustomization %s: configMapGenerator entry %d is not named %q", path, entryIndex, entryName) + } + if err := edit(root, entry); err != nil { + return skip("kustomization %s: %v", path, err) + } + encoded, err := encodeNode(root) + if err != nil { + return skip("kustomization %s: re-encode failed: %v", path, err) + } + body := reskinDocument(target, string(encoded)) + if body == target { + return EditResult{Content: content, Mode: EditNoChange}, nil + } + docs[idx].body = body + return EditResult{Content: []byte(joinDocuments(docs)), Mode: EditPatched}, nil +} + +// applyLiteralEdits edits an entry's literals: items, creating the list for a first literal. +func applyLiteralEdits(entry *yaml.Node, edits []LiteralEdit) error { + seq := nodeMapGet(entry, "literals") + if seq == nil { + seq = &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"} + entry.Content = append(entry.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: "literals"}, seq) + } + if seq.Kind != yaml.SequenceNode { + return errors.New("literals is not a sequence") + } + for _, e := range edits { + i := literalIndex(seq, e.Key) + switch { + case e.Remove && i >= 0: + removeSequenceItem(seq, i) + case e.Remove: + case i >= 0: + item := seq.Content[i] + item.Value = e.Key + "=" + LiteralValue(e.Value) + item.Tag = "!!str" + if item.Style != yaml.SingleQuotedStyle && item.Style != yaml.DoubleQuotedStyle { + item.Style = 0 + } + default: + appendSequenceItem(seq, + &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: e.Key + "=" + LiteralValue(e.Value)}) + } + } + if len(seq.Content) == 0 { + nodeMapDelete(entry, "literals") + } + return nil +} + +// appendSequenceItem appends item to seq. A comment below the list belongs to its last item as +// the decoder reads it (a foot comment), so the comment moves to the new last item: otherwise the +// item would land after it, past a blank line. +func appendSequenceItem(seq, item *yaml.Node) { + if n := len(seq.Content); n > 0 { + last := seq.Content[n-1] + item.FootComment, last.FootComment = last.FootComment, "" + } + seq.Content = append(seq.Content, item) +} + +// removeSequenceItem removes seq.Content[i]. Its own head and line comments go with it; a foot +// comment, which describes what follows the item rather than the item itself, moves to the item +// before it so a comment below the list survives the removal of its last item. +func removeSequenceItem(seq *yaml.Node, i int) { + removed := seq.Content[i] + if removed.FootComment != "" && i > 0 { + prev := seq.Content[i-1] + prev.FootComment = joinComments(prev.FootComment, removed.FootComment) + } + seq.Content = append(seq.Content[:i], seq.Content[i+1:]...) +} + +func joinComments(a, b string) string { + if a == "" { + return b + } + return a + "\n" + b +} + +// literalIndex is the position of the KEY=... item for key, or -1. +func literalIndex(seq *yaml.Node, key string) int { + for i, item := range seq.Content { + if item.Kind != yaml.ScalarNode { + continue + } + if k, _, found := strings.Cut(item.Value, "="); found && k == key { + return i + } + } + return -1 +} + +// LiteralValue is the text after "KEY=" that kustomize reads back as value. kustomize removes one +// pair of matching surrounding quotes from a literal (kv.removeQuotes), so a value that is itself +// quoted is wrapped in the other quote to survive that. +func LiteralValue(value string) string { + if len(value) < 2 || value[0] != value[len(value)-1] { + return value + } + switch value[0] { + case '"': + return "'" + value + "'" + case '\'': + return `"` + value + `"` + default: + return value + } +} diff --git a/internal/git/manifestedit/kustomization_test.go b/internal/git/manifestedit/kustomization_test.go index 8c31dab59..1fa7ebae7 100644 --- a/internal/git/manifestedit/kustomization_test.go +++ b/internal/git/manifestedit/kustomization_test.go @@ -5,6 +5,8 @@ package manifestedit import ( "strings" "testing" + + "github.com/stretchr/testify/require" ) const kustomizationFixture = `# pin the app image here @@ -362,3 +364,88 @@ func TestAppendKustomizationResource_RefusalsLeaveContentUntouched(t *testing.T) }) } } + +// A configMapGenerator edit pins the entry by index AND name, so a drifted file is skipped rather +// than mis-edited, exactly as an images: edit is. +func TestConfigMapGeneratorEdits_SkipADriftedEntry(t *testing.T) { + content := []byte("configMapGenerator:\n - name: menu\n literals:\n - A=1\n") + + res, _ := EditConfigMapGeneratorLiterals("kustomization.yaml", content, 0, "flags", + []LiteralEdit{{Key: "A", Value: "2"}}) + require.Equal(t, EditSkipped, res.Mode) + res, _ = RemoveConfigMapGeneratorEntry("kustomization.yaml", content, 1, "menu") + require.Equal(t, EditSkipped, res.Mode) + + res, _ = EditConfigMapGeneratorLiterals("kustomization.yaml", content, 0, "menu", + []LiteralEdit{{Key: "A", Remove: true}, {Key: "Z", Remove: true}}) + require.Equal(t, EditPatched, res.Mode) + require.Equal(t, "configMapGenerator:\n - name: menu\n", string(res.Content), + "an emptied literals: goes, and removing an absent key is a no-op") +} + +// Comments in a literals: list are the author's, and an edit keeps them: an edited item keeps its +// head and line comments, a removed item takes only its own, and a comment below the list stays +// below it whichever item ends up last. +func TestEditConfigMapGeneratorLiterals_KeepsComments(t *testing.T) { + const in = `configMapGenerator: + - name: branding + literals: + # The colour scheme of the shop front. + - THEME=dark # light or dark + # Shown under the logo. + - SLOGAN=Coffee first. + - TIPS=on # off during the rush + # Ask the barista before changing the branding. +` + cases := map[string]struct { + edit LiteralEdit + want string + }{ + "edit an item": {LiteralEdit{Key: "THEME", Value: "light"}, `configMapGenerator: + - name: branding + literals: + # The colour scheme of the shop front. + - THEME=light # light or dark + # Shown under the logo. + - SLOGAN=Coffee first. + - TIPS=on # off during the rush + # Ask the barista before changing the branding. +`}, + "remove an item": {LiteralEdit{Key: "SLOGAN", Remove: true}, `configMapGenerator: + - name: branding + literals: + # The colour scheme of the shop front. + - THEME=dark # light or dark + - TIPS=on # off during the rush + # Ask the barista before changing the branding. +`}, + "remove the last item": {LiteralEdit{Key: "TIPS", Remove: true}, `configMapGenerator: + - name: branding + literals: + # The colour scheme of the shop front. + - THEME=dark # light or dark + # Shown under the logo. + - SLOGAN=Coffee first. + # Ask the barista before changing the branding. +`}, + "append an item": {LiteralEdit{Key: "OPEN", Value: "08:00"}, `configMapGenerator: + - name: branding + literals: + # The colour scheme of the shop front. + - THEME=dark # light or dark + # Shown under the logo. + - SLOGAN=Coffee first. + - TIPS=on # off during the rush + - OPEN=08:00 + # Ask the barista before changing the branding. +`}, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + res, diags := EditConfigMapGeneratorLiterals("kustomization.yaml", []byte(in), 0, "branding", + []LiteralEdit{tc.edit}) + require.Equal(t, EditPatched, res.Mode, "%v", diags) + require.Equal(t, tc.want, string(res.Content)) + }) + } +} diff --git a/internal/git/plan_flush.go b/internal/git/plan_flush.go index f9075141c..57fc242a5 100644 --- a/internal/git/plan_flush.go +++ b/internal/git/plan_flush.go @@ -217,6 +217,10 @@ type writeBatch struct { // (ADDED then MODIFIED, or back and gone again) read the pre-batch render and was wrong about // it. See rendersInBatch. rendered map[renderKey]bool + // removedGenerated records the generated objects this batch already took out of the folder, so + // a second removal of one (a live delete and a sweep, or a replay) is a no-op rather than an + // edit of an entry that is gone. See removeGenerated. + removedGenerated map[*manifestanalyzer.GeneratedObject]bool // touched records, while one event is applied, each file it touched with that file's bytes // before the event. Nil when nothing is recorded. See beginEvent. touched map[string]touchedFile @@ -310,8 +314,10 @@ func newWriteBatch( // this line the live path would leave no trace of why. The analyzer CLI and // scan mode show the same diagnostics offline. logStoreDiagnostics(ctx, store.Diagnostics) - contentByPath := make(map[string][]byte, len(scan.YAMLFiles)) - for _, f := range scan.YAMLFiles { + // The generator inputs are hydrated beside the YAML files: a generated ConfigMap's edit is + // written into one of them, and the oracle renders both. + contentByPath := make(map[string][]byte, len(scan.YAMLFiles)+len(scan.InputFiles)) + for _, f := range scan.RenderFiles() { contentByPath[f.Path] = f.Content } batch := &writeBatch{ @@ -514,7 +520,7 @@ func (wb *writeBatch) applyUpsert(ctx context.Context, event Event) (upsertOutco // is coming back into. func (wb *writeBatch) stagedRender(ctx context.Context) func() { scan := wb.scan - scan.YAMLFiles = wb.files() + scan.YAMLFiles, scan.InputFiles = wb.stagedFiles() staged := manifestanalyzer.BuildStoreFromScan(ctx, scan, wb.mapper, manifestanalyzer.WriterAllowlist(), manifestanalyzer.WithDeclaredNamespace(wb.namespaces.declaredNamespace())) store, docLoc := wb.store, wb.docLoc @@ -522,6 +528,24 @@ func (wb *writeBatch) stagedRender(ctx context.Context) func() { return func() { wb.store, wb.docLoc = store, docLoc } } +// stagedFiles splits the batch's tree into the scan's two views: the YAML files the store models, +// and the generator inputs it only renders. +func (wb *writeBatch) stagedFiles() ([]manifestedit.FileContent, []manifestedit.FileContent) { + inputs := make(map[string]struct{}, len(wb.scan.InputFiles)) + for _, f := range wb.scan.InputFiles { + inputs[f.Path] = struct{}{} + } + var yamlFiles, inputFiles []manifestedit.FileContent + for _, f := range wb.files() { + if _, input := inputs[f.Path]; input { + inputFiles = append(inputFiles, f) + continue + } + yamlFiles = append(yamlFiles, f) + } + return yamlFiles, inputFiles +} + // intendsObject reports whether one of intents is about obj. func intendsObject(intents []manifestanalyzer.WriteIntent, obj *unstructured.Unstructured) bool { for _, in := range intents { @@ -540,6 +564,11 @@ func (wb *writeBatch) upsert(ctx context.Context, event Event) (upsertOutcome, e } dm := wb.store.ByManifestIdentity[id] if dm == nil { + // No document holds it, but a configMapGenerator may render it: then the edit belongs + // in the generator's inputs, and a new document beside it would collide in the build. + if generated := wb.store.GeneratedObjectsFor(event.Object); len(generated) > 0 { + return wb.writeGenerated(ctx, event, generated) + } return wb.createNew(ctx, event) } filePath := wb.docLoc[dm].FilePath @@ -1471,7 +1500,8 @@ func (wb *writeBatch) applyDelete(ctx context.Context, event Event) error { } target, found := wb.resolveDelete(event) if !found { - return nil + // No document holds it, but a configMapGenerator may render it. + return wb.deleteGenerated(ctx, event) } _, err := wb.removeDocument(ctx, event.Identifier, target, documentDeletedLive) return err diff --git a/internal/git/render_scope.go b/internal/git/render_scope.go index c5d58775e..2c9347b4d 100644 --- a/internal/git/render_scope.go +++ b/internal/git/render_scope.go @@ -67,6 +67,7 @@ func scanRenderScope(root, base string) (renderScopeResult, error) { return renderScopeResult{}, err } if len(readFiles) == 0 { + attachGeneratorInputs(&specScan, root, base) return renderScopeResult{scan: specScan, renderBase: base, writeSubdir: ""}, nil } @@ -93,10 +94,20 @@ func scanRenderScope(root, base string) (renderScopeResult, error) { scan.YAMLFiles = append(scan.YAMLFiles, manifestedit.FileContent{Path: key, Content: content}) } sort.Slice(scan.YAMLFiles, func(i, j int) bool { return scan.YAMLFiles[i].Path < scan.YAMLFiles[j].Path }) + attachGeneratorInputs(&scan, root, renderBase) return renderScopeResult{scan: scan, renderBase: renderBase, writeSubdir: writeSubdir}, nil } +// attachGeneratorInputs moves the files a configMapGenerator reads into scan.InputFiles, reading +// a non-YAML one from the worktree under the scan anchor. An out-of-scope input arrives as one of +// the read-scope files, already read; see manifestanalyzer.AttachGeneratorInputs. +func attachGeneratorInputs(scan *manifestanalyzer.FolderScan, root, anchor string) { + manifestanalyzer.AttachGeneratorInputs(scan, func(rel string) ([]byte, bool) { + return readFileBytes(root, cleanSlash(path.Join(anchor, rel))) + }) +} + // resolveReadScope returns the sorted, distinct set of files (slash, worktree-relative) the // subtree's kustomizations read from OUTSIDE spec.path — the exact files kustomize loads, not // whole directories. It follows the resources/patches graph transitively: a referenced file @@ -167,9 +178,11 @@ func reachableTargets(root, base, dir string, content []byte) (reachableSplit, e return reachableSplit{}, nil // unparseable: the acceptance gate refuses it, not us } var out reachableSplit - entries := make([]string, 0, len(resources)+len(patches)) + inputs := manifestanalyzer.KustomizationGeneratorInputs(content) + entries := make([]string, 0, len(resources)+len(patches)+len(inputs)) entries = append(entries, resources...) entries = append(entries, patches...) + entries = append(entries, inputs...) for _, entry := range entries { if manifestanalyzer.IsRemoteBaseEntry(entry) { continue diff --git a/internal/git/resync_flush.go b/internal/git/resync_flush.go index bb553ba68..1ebfc6b07 100644 --- a/internal/git/resync_flush.go +++ b/internal/git/resync_flush.go @@ -530,21 +530,14 @@ func (wb *writeBatch) applyResyncPlan( if action.Kind != manifestanalyzer.PlanDropOrphan { continue } - // The planner names the document by its EFFECTIVE identity; its bytes may carry no - // namespace (one a kustomization supplies), so it is located by the raw identity, exactly - // as the live path's resolveDelete does. - target := deleteTarget{ - filePath: action.Ref.FilePath, - id: rawManifestIDForCurrentBytes(action.Identity, wb.store.ByManifestIdentity[action.Identity]), - } - removed, err := wb.removeDocument(ctx, action.Resource, target, documentDeletedSweep) + removed, err := wb.sweepDrop(ctx, action) if err != nil { return ResyncStats{}, nil, err } if removed { stats.Deleted++ actions.deletes = append(actions.deletes, - resyncDelete{resource: action.Resource, identity: action.Identity}) + resyncDelete{resource: action.Resource, identity: action.Identity, generated: action.Generated}) } } // Skipped stays a plan view (documents present but not editable in place); it is @@ -558,6 +551,22 @@ func (wb *writeBatch) applyResyncPlan( return stats, actions, nil } +// sweepDrop applies one planned drop, of a document or of an object a configMapGenerator renders, +// and reports whether it removed anything. +func (wb *writeBatch) sweepDrop(ctx context.Context, action manifestanalyzer.PlanAction) (bool, error) { + if action.Generated { + return wb.removeGenerated(ctx, action.Resource, objectNamed(action.Identity), documentDeletedSweep) + } + // The planner names the document by its EFFECTIVE identity; its bytes may carry no + // namespace (one a kustomization supplies), so it is located by the raw identity, exactly + // as the live path's resolveDelete does. + target := deleteTarget{ + filePath: action.Ref.FilePath, + id: rawManifestIDForCurrentBytes(action.Identity, wb.store.ByManifestIdentity[action.Identity]), + } + return wb.removeDocument(ctx, action.Resource, target, documentDeletedSweep) +} + // resyncActions is what a resync decided on the tree it was first committed on // (docs/design/applier-echoes.md, F2): the snapshot's objects whose upsert changed that tree, and the // documents its sweep removed, by identity. The snapshot's full membership is used once, to decide; @@ -575,6 +584,8 @@ type resyncActions struct { type resyncDelete struct { resource itypes.ResourceIdentifier identity manifestedit.Identity + // generated marks an object a configMapGenerator rendered: the sweep removed its entry. + generated bool } // empty reports whether the resync decided nothing: it changed nothing in Git, so it does not exist. @@ -615,11 +626,7 @@ func (wb *writeBatch) applyResyncActions( return stats, nil } for _, deleted := range actions.deletes { - target, found := wb.resolveResyncDelete(deleted) - if !found { - continue - } - removed, err := wb.removeDocument(ctx, deleted.resource, target, documentDeletedSweep) + removed, err := wb.replayDelete(ctx, deleted) if err != nil { return ResyncStats{}, err } @@ -630,6 +637,19 @@ func (wb *writeBatch) applyResyncActions( return stats, nil } +// replayDelete removes, from this batch's tree, what a resync's sweep removed, if the tree still +// holds it, and reports whether it did. +func (wb *writeBatch) replayDelete(ctx context.Context, deleted resyncDelete) (bool, error) { + if deleted.generated { + return wb.removeGenerated(ctx, deleted.resource, objectNamed(deleted.identity), documentDeletedSweep) + } + target, found := wb.resolveResyncDelete(deleted) + if !found { + return false, nil + } + return wb.removeDocument(ctx, deleted.resource, target, documentDeletedSweep) +} + // resolveResyncDelete locates, in this batch's tree, the document a resync's sweep removed. func (wb *writeBatch) resolveResyncDelete(deleted resyncDelete) (deleteTarget, bool) { if dm := wb.store.ByManifestIdentity[deleted.identity]; dm != nil { diff --git a/internal/manifestanalyzer/analyzer.go b/internal/manifestanalyzer/analyzer.go index 62f1c1b21..98b22762a 100644 --- a/internal/manifestanalyzer/analyzer.go +++ b/internal/manifestanalyzer/analyzer.go @@ -335,6 +335,10 @@ func collectFiles(fsys fs.FS) FolderScan { sort.Slice(scan.YAMLFiles, func(i, j int) bool { return scan.YAMLFiles[i].Path < scan.YAMLFiles[j].Path }) sort.Strings(scan.NonYAML) sort.Slice(scan.Foreign, func(i, j int) bool { return scan.Foreign[i].Path < scan.Foreign[j].Path }) + AttachGeneratorInputs(&scan, func(p string) ([]byte, bool) { + content, err := fs.ReadFile(fsys, p) + return content, err == nil + }) return scan } diff --git a/internal/manifestanalyzer/generator_write.go b/internal/manifestanalyzer/generator_write.go new file mode 100644 index 000000000..0d51e1a99 --- /dev/null +++ b/internal/manifestanalyzer/generator_write.go @@ -0,0 +1,795 @@ +// SPDX-License-Identifier: Apache-2.0 + +package manifestanalyzer + +import ( + "bytes" + "context" + "encoding/base64" + "errors" + "fmt" + "sort" + "strings" + "unicode" + "unicode/utf8" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + + "github.com/ConfigButler/gitops-reverser/internal/git/manifestedit" + "github.com/ConfigButler/gitops-reverser/internal/types" + "github.com/ConfigButler/gitops-reverser/internal/typeset" +) + +// This file plans the write of a live ConfigMap back into the configMapGenerator entry that +// renders it. The plan is an attribution, and like every attribution it may be wrong: the oracle +// re-renders the folder with the planned bytes and refuses the write unless kustomize then +// produces exactly the live object. +// +// The rules, per data key: +// +// - a files: key whose value changed: the file's whole content becomes the live value; +// - an envs: key whose value changed: its line is edited in place, so comments, order and +// every other line keep their bytes; +// - a literals: key whose value changed: its KEY=value item in kustomization.yaml is edited in +// place, as an images: entry is; +// - a key the live object added: appended to the entry's env file when it has exactly one, and +// otherwise to its literals: when it has some and no env file; +// - a key the live object removed: its env line or literal goes; a files: key, whose value is +// a whole file, has nothing to remove; +// - binaryData, and anything outside data, have no input to land in. +// +// A live delete removes the entry from configMapGenerator: and the input files only it reads; see +// PlanGeneratedDelete. + +// GeneratedObject is one object a configMapGenerator entry renders, as kustomize renders it. +type GeneratedObject struct { + // GeneratedIn is the kustomization whose generator produced it (slash). + GeneratedIn string + // Entry is the configMapGenerator entry that produced it. Nil when the kustomization holds + // no entry of the rendered name, or more than one: the object then has no single source. + Entry *ConfigMapGeneratorEntry + // Rendered is the object as the render produces it. + Rendered *unstructured.Unstructured + // Roots are the render roots that render this entry, in any namespace, sorted. + Roots []string + // SharedInputs are the entry's input files something else also reads: another generator + // entry, a resources: entry or a patch. A delete of the object keeps them. + SharedInputs map[string]struct{} + // Resource is the API-side identity of the rendered object, as a document's ResourceIdentity + // is: set only when the registry resolves its GVK to a followable resource. The resync sweep + // drops only a generated object it can name this way. + Resource *types.ResourceIdentifier +} + +// Identity is the manifest identity of the rendered object. +func (g *GeneratedObject) Identity() manifestedit.Identity { + return manifestedit.Identity{ + APIVersion: g.Rendered.GetAPIVersion(), + Kind: g.Rendered.GetKind(), + Namespace: g.Rendered.GetNamespace(), + Name: g.Rendered.GetName(), + } +} + +// resolveGeneratedResources resolves every generated object's API-side identity, as +// resolveMapping does a document's. A rendered object with no namespace takes the declared one, +// as a namespace-less document does (WithDeclaredNamespace). +func (s *ManifestStore) resolveGeneratedResources( + ctx context.Context, + lookup typeset.Lookup, + declaredNamespace string, +) { + if ctx.Err() != nil || !lookup.Ready() { + return + } + for _, g := range s.generated { + gvk := g.Rendered.GroupVersionKind() + record, known := lookup.ByGVK(gvk) + if !known || !record.Followable() { + continue + } + namespace := "" + if record.Identity.Scope == typeset.ScopeNamespaced { + namespace = g.Rendered.GetNamespace() + if namespace == "" { + namespace = declaredNamespace + } + } + gvr := record.Identity.GVR + ri := types.NewResourceIdentifier(gvr.Group, gvr.Version, gvr.Resource, namespace, g.Rendered.GetName()) + g.Resource = &ri + } +} + +// generatedRender is one generated object one render root produced. +type generatedRender struct { + root string + in string + object *unstructured.Unstructured +} + +// generatedObjects groups the generated renders of a scan into one GeneratedObject per entry and +// namespace, each naming every root that renders the entry. +func generatedObjects(renders []generatedRender, kusts map[string]*kustomizationDoc) []*GeneratedObject { + type entryKey struct{ in, kind, name string } + type objectKey struct { + entry entryKey + namespace string + } + roots := map[entryKey]map[string]struct{}{} + byObject := map[objectKey]*GeneratedObject{} + inputUses := inputUseCounts(kusts) + var order []objectKey + for _, r := range renders { + ek := entryKey{in: r.in, kind: r.object.GetKind(), name: r.object.GetName()} + if roots[ek] == nil { + roots[ek] = map[string]struct{}{} + } + roots[ek][r.root] = struct{}{} + ok := objectKey{entry: ek, namespace: r.object.GetNamespace()} + if _, seen := byObject[ok]; seen { + continue + } + entry := generatorEntryFor(kusts[slashDir(r.in)], r.object) + byObject[ok] = &GeneratedObject{ + GeneratedIn: r.in, + Entry: entry, + Rendered: r.object, + SharedInputs: sharedInputsOf(entry, inputUses), + } + order = append(order, ok) + } + out := make([]*GeneratedObject, 0, len(order)) + for _, ok := range order { + g := byObject[ok] + g.Roots = sortedKeysOfSet(roots[ok.entry]) + out = append(out, g) + } + return out +} + +// inputUseCounts counts, per file, how many generator entries, resources: entries and patches of +// the scan read it. +func inputUseCounts(kusts map[string]*kustomizationDoc) map[string]int { + uses := map[string]int{} + for _, doc := range kusts { + for _, entry := range doc.configMapGenerators { + seen := map[string]struct{}{} + for _, p := range entry.inputPaths() { + if _, dup := seen[p]; !dup { + seen[p] = struct{}{} + uses[p]++ + } + } + } + for _, p := range doc.patches { + uses[p]++ + } + } + for p := range reachedResourceFiles(kusts) { + uses[p]++ + } + return uses +} + +// sharedInputsOf is the entry's inputs that more than one reader reads. +func sharedInputsOf(entry *ConfigMapGeneratorEntry, uses map[string]int) map[string]struct{} { + if entry == nil { + return nil + } + out := map[string]struct{}{} + for _, p := range entry.inputPaths() { + if uses[p] > 1 { + out[p] = struct{}{} + } + } + return out +} + +// generatorEntryFor finds the entry that rendered obj: the one entry of its name, or, when two +// entries share a name, the one whose own namespace is the object's. +func generatorEntryFor(doc *kustomizationDoc, obj *unstructured.Unstructured) *ConfigMapGeneratorEntry { + if doc == nil { + return nil + } + var named []*ConfigMapGeneratorEntry + for i := range doc.configMapGenerators { + if doc.configMapGenerators[i].Name == obj.GetName() { + named = append(named, &doc.configMapGenerators[i]) + } + } + if len(named) == 1 { + return named[0] + } + var match *ConfigMapGeneratorEntry + for _, e := range named { + if e.Namespace == obj.GetNamespace() { + if match != nil { + return nil + } + match = e + } + } + return match +} + +// GeneratedObjectsFor returns every generated object the live object can be: same apiVersion, +// kind and name, and either the same namespace or none, which is the render of a folder whose +// namespace is supplied downstream. More than one result is an object with no single source. +func (s *ManifestStore) GeneratedObjectsFor(live *unstructured.Unstructured) []*GeneratedObject { + if live == nil { + return nil + } + var out []*GeneratedObject + for _, g := range s.generated { + r := g.Rendered + if r.GetAPIVersion() != live.GetAPIVersion() || r.GetKind() != live.GetKind() || + r.GetName() != live.GetName() { + continue + } + if r.GetNamespace() != live.GetNamespace() && r.GetNamespace() != "" { + continue + } + out = append(out, g) + } + return out +} + +// GeneratedWrite is one file a generator write rewrites: an input file, or the kustomization for +// a literal or a removed entry. A nil Content deletes the file. +type GeneratedWrite struct { + Path string + Content []byte +} + +// errGeneratedWrite marks a generated object the writer cannot write back. The message says why; +// the caller names the object and the kustomization. +var errGeneratedWrite = errors.New("no destination for the change") + +// PlanGeneratedWrite plans the write of desired, the live object as the writer projects it, into +// the inputs of the generator that renders gen. writeScope is the write jail (empty: everything +// is writable), and read returns an input file's current bytes. An error is a refusal and names +// the reason; nothing has been written. +func PlanGeneratedWrite( + gen *GeneratedObject, + desired *unstructured.Unstructured, + writeScope string, + read func(path string) ([]byte, bool), +) ([]GeneratedWrite, error) { + entry, err := writableEntry(gen, writeScope) + if err != nil { + return nil, err + } + if fields := changedOutsideData(desired, gen.Rendered); len(fields) > 0 { + return nil, fmt.Errorf("%w: %s changed, and a generated ConfigMap takes it from the kustomization, "+ + "not from an input file", errGeneratedWrite, strings.Join(fields, ", ")) + } + + current, err := readGeneratorSources(entry, read) + if err != nil { + return nil, err + } + liveBinary, _, _ := unstructured.NestedStringMap(desired.Object, "binaryData") + if !equalStringMaps(liveBinary, current.binary) { + return nil, fmt.Errorf("%w: binaryData changed, and the operator writes no binary input file", + errGeneratedWrite) + } + + plan := generatorPlan{entry: entry, writeScope: writeScope, read: read, files: map[string][]byte{}} + liveData, _, _ := unstructured.NestedStringMap(desired.Object, "data") + for _, key := range sortedKeysOfSet(unionStringKeys(liveData, current.data)) { + if err := plan.key(key, liveData, current); err != nil { + return nil, err + } + } + if err := plan.applyLiterals(); err != nil { + return nil, err + } + return plan.writes(read), nil +} + +// writableEntry is the entry behind gen, or the refusal that says why its inputs are not this +// target's to write: no single entry, an entry two render roots share, or one in a read-only base. +func writableEntry(gen *GeneratedObject, writeScope string) (*ConfigMapGeneratorEntry, error) { + entry := gen.Entry + if entry == nil { + return nil, fmt.Errorf("%w: %s holds no single configMapGenerator entry named %q", + errGeneratedWrite, gen.GeneratedIn, gen.Rendered.GetName()) + } + if len(gen.Roots) > 1 { + return nil, fmt.Errorf("%w: the generator is rendered by %d render roots (%s), so its inputs are shared", + errGeneratedWrite, len(gen.Roots), strings.Join(gen.Roots, ", ")) + } + if writeScope != "" && !pathWithin(entry.KustomizationPath, writeScope) { + return nil, fmt.Errorf("%w: the generator lives in %s, which this target reads and never writes", + errGeneratedWrite, entry.KustomizationPath) + } + return entry, nil +} + +// PlanGeneratedDelete plans the removal of the object gen renders: its entry leaves +// configMapGenerator:, and every input file only that entry reads is deleted. An input something +// else also reads stays, and so does one outside the write jail. read returns a file's current +// bytes. An error is a refusal and names the reason. +func PlanGeneratedDelete( + gen *GeneratedObject, + writeScope string, + read func(path string) ([]byte, bool), +) ([]GeneratedWrite, error) { + entry, err := writableEntry(gen, writeScope) + if err != nil { + return nil, err + } + kustomization, ok := read(entry.KustomizationPath) + if !ok { + return nil, fmt.Errorf("%w: cannot read %s", errGeneratedWrite, entry.KustomizationPath) + } + res, diags := manifestedit.RemoveConfigMapGeneratorEntry( + entry.KustomizationPath, kustomization, entry.Index, entry.Name) + if res.Mode != manifestedit.EditPatched { + return nil, fmt.Errorf("%w: the entry cannot be removed from %s%s", + errGeneratedWrite, entry.KustomizationPath, diagnosticSuffix(diags)) + } + writes := []GeneratedWrite{{Path: entry.KustomizationPath, Content: res.Content}} + for _, p := range sortedKeysOfSet(pathsSet(entry.inputPaths())) { + _, shared := gen.SharedInputs[p] + _, present := read(p) + if shared || !present || (writeScope != "" && !pathWithin(p, writeScope)) { + continue + } + writes = append(writes, GeneratedWrite{Path: p}) + } + return writes, nil +} + +func pathsSet(paths []string) map[string]struct{} { + out := make(map[string]struct{}, len(paths)) + for _, p := range paths { + out[p] = struct{}{} + } + return out +} + +// diagnosticSuffix renders an editor's diagnostics after a refusal's sentence. +func diagnosticSuffix(diags []manifestedit.Diagnostic) string { + if len(diags) == 0 { + return "" + } + parts := make([]string, 0, len(diags)) + for _, d := range diags { + parts = append(parts, d.Message) + } + return ": " + strings.Join(parts, "; ") +} + +// generatorPlan accumulates the new content of each input file a write touches. +type generatorPlan struct { + entry *ConfigMapGeneratorEntry + writeScope string + read func(string) ([]byte, bool) + files map[string][]byte + // literals are the literals: edits, applied to the kustomization in one call at the end. + literals []manifestedit.LiteralEdit +} + +// key routes one data key's difference between the live object and the inputs. +func (p *generatorPlan) key(key string, live map[string]string, current generatorSources) error { + value, inLive := live[key] + was, inCurrent := current.data[key] + switch { + case inLive && inCurrent && value == was: + return nil + case !inCurrent: + return p.added(key, value) + } + src, known := current.source[key] + switch { + case !known: + return fmt.Errorf("%w: key %q has no input file", errGeneratedWrite, key) + case src.kind == sourceLiteral && !inLive: + p.literals = append(p.literals, manifestedit.LiteralEdit{Key: key, Remove: true}) + return nil + case src.kind == sourceLiteral: + p.literals = append(p.literals, manifestedit.LiteralEdit{Key: key, Value: value}) + return nil + case !inLive: + return p.removed(key, src) + case src.kind == sourceFile: + if err := p.writable(src.path); err != nil { + return err + } + p.files[src.path] = []byte(value) + return nil + default: + return p.setEnv(src.path, key, value) + } +} + +// added routes a key the live object has and the inputs do not: to the entry's one env file, or, +// for an entry with no env file, to its literals:. +func (p *generatorPlan) added(key, value string) error { + switch { + case len(p.entry.Envs) == 1: + return p.setEnv(p.entry.Envs[0], key, value) + case len(p.entry.Envs) > 1: + return fmt.Errorf("%w: key %q is new, and the entry has %d envs: files, so no single one to add it to", + errGeneratedWrite, key, len(p.entry.Envs)) + case len(p.entry.Literals) > 0: + p.literals = append(p.literals, manifestedit.LiteralEdit{Key: key, Value: value}) + return nil + default: + return fmt.Errorf("%w: key %q is new, and the entry has no envs: file or literals: to add it to", + errGeneratedWrite, key) + } +} + +// applyLiterals folds the literal edits into the kustomization, in one edit of the entry. +func (p *generatorPlan) applyLiterals() error { + if len(p.literals) == 0 { + return nil + } + path := p.entry.KustomizationPath + res, diags := manifestedit.EditConfigMapGeneratorLiterals( + path, p.content(path), p.entry.Index, p.entry.Name, p.literals) + switch res.Mode { + case manifestedit.EditPatched: + p.files[path] = res.Content + return nil + case manifestedit.EditNoChange: + return nil + case manifestedit.EditSkipped, manifestedit.EditDeleted, manifestedit.EditWholeReplace: + } + return fmt.Errorf("%w: the literals: of the entry cannot be edited%s", + errGeneratedWrite, diagnosticSuffix(diags)) +} + +// removed routes a key the inputs have and the live object does not: its env line goes, and a +// files: key, whose value is a whole file, has no line to remove. +func (p *generatorPlan) removed(key string, src generatorSource) error { + if src.kind == sourceFile { + return fmt.Errorf("%w: key %q was removed, and its value is the whole of %s", errGeneratedWrite, key, src.path) + } + if err := p.writable(src.path); err != nil { + return err + } + p.files[src.path] = removeEnvKey(p.content(src.path), key) + return nil +} + +// setEnv sets key on its line of the env file at path, appending a line when there is none. +func (p *generatorPlan) setEnv(path, key, value string) error { + if strings.ContainsAny(value, "\r\n") { + return fmt.Errorf("%w: the value of key %q spans lines, which a line of %s cannot hold", + errGeneratedWrite, key, path) + } + if err := p.writable(path); err != nil { + return err + } + p.files[path] = setEnvValue(p.content(path), key, value) + return nil +} + +// writable refuses an input file outside the write jail. +func (p *generatorPlan) writable(path string) error { + if p.writeScope != "" && !pathWithin(path, p.writeScope) { + return fmt.Errorf("%w: %s is outside spec.path", errGeneratedWrite, path) + } + return nil +} + +// content is an input file's bytes as the plan has left them so far. +func (p *generatorPlan) content(path string) []byte { + if c, ok := p.files[path]; ok { + return c + } + c, _ := p.read(path) + return c +} + +// writes returns the planned files that differ from what read holds, sorted by path. +func (p *generatorPlan) writes(read func(string) ([]byte, bool)) []GeneratedWrite { + paths := make([]string, 0, len(p.files)) + for path := range p.files { + paths = append(paths, path) + } + sort.Strings(paths) + var out []GeneratedWrite + for _, path := range paths { + if before, ok := read(path); ok && bytes.Equal(before, p.files[path]) { + continue + } + out = append(out, GeneratedWrite{Path: path, Content: p.files[path]}) + } + return out +} + +// changedOutsideData names the top-level fields, and the metadata fields, in which desired and +// rendered differ once data, binaryData and the namespace are set aside. The namespace is the +// oracle's to compare: it knows when a target supplies it downstream. +func changedOutsideData(desired, rendered *unstructured.Unstructured) []string { + strip := func(o *unstructured.Unstructured) map[string]interface{} { + c := o.DeepCopy().Object + delete(c, "data") + delete(c, "binaryData") + unstructured.RemoveNestedField(c, "metadata", "namespace") + return c + } + a, b := strip(desired), strip(rendered) + var out []string + for _, key := range sortedKeysOfSet(unionAnyKeys(a, b)) { + if key != "metadata" { + if !sameValue(a[key], b[key]) { + out = append(out, key) + } + continue + } + am, _ := a[key].(map[string]interface{}) + bm, _ := b[key].(map[string]interface{}) + for _, field := range sortedKeysOfSet(unionAnyKeys(am, bm)) { + if !sameValue(am[field], bm[field]) { + out = append(out, "metadata."+field) + } + } + } + return out +} + +func sameValue(a, b interface{}) bool { + return sameObject(&unstructured.Unstructured{Object: map[string]interface{}{"v": a}}, + &unstructured.Unstructured{Object: map[string]interface{}{"v": b}}) +} + +// sourceKind is where a generated key's value comes from. +type sourceKind int + +const ( + sourceFile sourceKind = iota + sourceEnv + sourceLiteral +) + +type generatorSource struct { + kind sourceKind + path string +} + +// generatorSources is the data an entry's inputs produce, read the way kustomize reads them +// (api/kv), and where each key came from. +type generatorSources struct { + data map[string]string + binary map[string]string + source map[string]generatorSource +} + +// readGeneratorSources reads an entry's inputs as they stand. kustomize loads envs:, then +// literals:, then files:, and a key repeated across them fails the build, so the order only +// decides which source a refusal names. +func readGeneratorSources( + entry *ConfigMapGeneratorEntry, + read func(string) ([]byte, bool), +) (generatorSources, error) { + out := generatorSources{ + data: map[string]string{}, + binary: map[string]string{}, + source: map[string]generatorSource{}, + } + for _, env := range entry.Envs { + if err := out.readEnv(env, read); err != nil { + return generatorSources{}, err + } + } + for _, literal := range entry.Literals { + key, value, ok := parseLiteralSource(literal) + if !ok { + return generatorSources{}, fmt.Errorf("%w: literal %q does not parse", errGeneratedWrite, literal) + } + if err := out.add(key, value, generatorSource{kind: sourceLiteral}); err != nil { + return generatorSources{}, err + } + } + for _, f := range entry.Files { + if err := out.readFile(f, read); err != nil { + return generatorSources{}, err + } + } + return out, nil +} + +// add records one key, refusing a key a second input repeats. +func (g generatorSources) add(key, value string, src generatorSource) error { + if _, dup := g.source[key]; dup { + return fmt.Errorf("%w: key %q is set by more than one input of the entry", errGeneratedWrite, key) + } + g.source[key] = src + g.data[key] = value + return nil +} + +// readEnv records every pair of one envs: file. +func (g generatorSources) readEnv(env string, read func(string) ([]byte, bool)) error { + content, ok := read(env) + if !ok { + return fmt.Errorf("%w: cannot read %s", errGeneratedWrite, env) + } + for _, kv := range envPairs(content) { + if err := g.add(kv.key, kv.value, generatorSource{kind: sourceEnv, path: env}); err != nil { + return err + } + } + return nil +} + +// readFile records one files: source. kustomize puts a file that is not UTF-8 into binaryData, +// base64-encoded. +func (g generatorSources) readFile(f GeneratorFileSource, read func(string) ([]byte, bool)) error { + content, ok := read(f.Path) + if !ok { + return fmt.Errorf("%w: cannot read %s", errGeneratedWrite, f.Path) + } + if err := g.add(f.Key, string(content), generatorSource{kind: sourceFile, path: f.Path}); err != nil { + return err + } + if !utf8.Valid(content) { + delete(g.data, f.Key) + g.binary[f.Key] = base64.StdEncoding.EncodeToString(content) + } + return nil +} + +// parseLiteralSource splits a literals: entry the way kustomize does (kv.parseLiteralSource): +// at the first "=", with one pair of matching surrounding quotes removed from the value. +func parseLiteralSource(source string) (string, string, bool) { + key, value, found := strings.Cut(source, "=") + if !found || key == "" { + return "", "", false + } + if len(value) >= 2 && value[0] == value[len(value)-1] && (value[0] == '"' || value[0] == '\'') { + value = value[1 : len(value)-1] + } + return key, value, true +} + +// utf8BOM is the byte-order mark kustomize strips from an env file's first line. +const utf8BOM = "\xEF\xBB\xBF" + +// envLine is one line of an env file, as kustomize reads it (kv.keyValuesFromLine). +type envLine struct { + // raw is the line with its terminator. + raw []byte + // prefix is what precedes the key: a byte-order mark and leading whitespace. + prefix []byte + // key is empty for a blank line or a comment. + key, value string + // terminator is "\n", "\r\n", or empty on a last line without one. + terminator string +} + +type envPair struct{ key, value string } + +// envLines splits an env file into lines and parses each as kustomize does: leading whitespace +// is trimmed, a line starting with "#" is a comment, the key is everything before the first "=" +// and the value everything after it, verbatim. A line with no "=" is a key with an empty value. +func envLines(content []byte) []envLine { + var out []envLine + for i := 0; len(content) > 0; i++ { + end := bytes.IndexByte(content, '\n') + raw := content + if end >= 0 { + raw = content[:end+1] + } + content = content[len(raw):] + + line := envLine{raw: raw} + body := raw + switch { + case bytes.HasSuffix(body, []byte("\r\n")): + body, line.terminator = body[:len(body)-2], "\r\n" + case bytes.HasSuffix(body, []byte("\n")): + body, line.terminator = body[:len(body)-1], "\n" + } + start := 0 + if i == 0 && bytes.HasPrefix(body, []byte(utf8BOM)) { + start = len(utf8BOM) + } + trimmed := bytes.TrimLeftFunc(body[start:], unicode.IsSpace) + line.prefix = body[:len(body)-len(trimmed)] + if len(trimmed) > 0 && trimmed[0] != '#' { + key, value, _ := strings.Cut(string(trimmed), "=") + line.key, line.value = key, value + } + out = append(out, line) + } + return out +} + +// envPairs is the key/value pairs of an env file, in order. +func envPairs(content []byte) []envPair { + var out []envPair + for _, l := range envLines(content) { + if l.key != "" { + out = append(out, envPair{key: l.key, value: l.value}) + } + } + return out +} + +// setEnvValue sets key to value on the first line that defines it, keeping the line's indent and +// terminator, or appends "key=value" when no line does. +func setEnvValue(content []byte, key, value string) []byte { + var out []byte + set := false + for _, l := range envLines(content) { + if !set && l.key == key { + out = append(out, l.prefix...) + out = append(out, key+"="+value+l.terminator...) + set = true + continue + } + out = append(out, l.raw...) + } + if set { + return out + } + if len(out) > 0 && out[len(out)-1] != '\n' { + out = append(out, '\n') + } + return append(out, key+"="+value+"\n"...) +} + +// removeEnvKey drops every line that defines key. +func removeEnvKey(content []byte, key string) []byte { + out := []byte{} + for _, l := range envLines(content) { + if l.key == key { + continue + } + out = append(out, l.raw...) + } + return out +} + +func equalStringMaps(a, b map[string]string) bool { + if len(a) != len(b) { + return false + } + for k, v := range a { + if w, ok := b[k]; !ok || w != v { + return false + } + } + return true +} + +func unionStringKeys(a, b map[string]string) map[string]struct{} { + out := make(map[string]struct{}, len(a)+len(b)) + for k := range a { + out[k] = struct{}{} + } + for k := range b { + out[k] = struct{}{} + } + return out +} + +func unionAnyKeys(a, b map[string]interface{}) map[string]struct{} { + out := make(map[string]struct{}, len(a)+len(b)) + for k := range a { + out[k] = struct{}{} + } + for k := range b { + out[k] = struct{}{} + } + return out +} + +// pathSet is the set of slash paths of files. +func pathSet(files []manifestedit.FileContent) map[string]struct{} { + if len(files) == 0 { + return nil + } + out := make(map[string]struct{}, len(files)) + for _, f := range files { + out[filepathToSlash(f.Path)] = struct{}{} + } + return out +} diff --git a/internal/manifestanalyzer/generators.go b/internal/manifestanalyzer/generators.go new file mode 100644 index 000000000..625d397be --- /dev/null +++ b/internal/manifestanalyzer/generators.go @@ -0,0 +1,269 @@ +// SPDX-License-Identifier: Apache-2.0 + +package manifestanalyzer + +import ( + "path" + "sort" + "strings" + + kustypes "sigs.k8s.io/kustomize/api/types" + + "github.com/ConfigButler/gitops-reverser/internal/git/manifestedit" +) + +// This file models the one generator the operator writes back through: a configMapGenerator +// whose name hash is disabled. Its input files are the configuration people maintain, so a live +// ConfigMap edit lands in the file (files:) or on the line (envs:) the value came from, and the +// render oracle proves the write by building the folder again. +// +// A generator with the name hash is refused, and that is not about finding the source. The +// re-render names the object differently, so the object somebody edited is replaced by another +// one on the next apply, and "the render reproduces the edited object" is false by construction. + +// featureGeneratorNameHash refuses a configMapGenerator entry whose name carries the content +// hash. The author can disable the hash; see unsupportedKustomizeDetail for the action. +const featureGeneratorNameHash = "configMapGenerator-name-hash" + +// featureGeneratorBehavior refuses a configMapGenerator entry with behavior: merge or replace. +// Such an entry rewrites a ConfigMap a base generates, so the object is shared with the base and +// no single file owns it. +const featureGeneratorBehavior = "configMapGenerator-behavior" + +// ConfigMapGeneratorEntry is one configMapGenerator entry of a kustomization, with every input +// path resolved against the scan root. +type ConfigMapGeneratorEntry struct { + // KustomizationPath is the kustomization file declaring the entry (slash). + KustomizationPath string + // Index is the entry's position in configMapGenerator:. + Index int + // Name and Namespace are the entry's own fields. Namespace is usually empty: a namespace: + // transformer supplies it. + Name, Namespace string + // Files are the files: sources, in listed order. + Files []GeneratorFileSource + // Envs are the envs: files, resolved, in listed order. + Envs []string + // Literals are the literals: entries, raw ("KEY=value"). + Literals []string +} + +// GeneratorFileSource is one files: source: the data key and the file whose whole content is +// its value. +type GeneratorFileSource struct { + Key string + Path string +} + +// inputPaths lists every file the entry reads. +func (e ConfigMapGeneratorEntry) inputPaths() []string { + out := make([]string, 0, len(e.Files)+len(e.Envs)) + for _, f := range e.Files { + out = append(out, f.Path) + } + return append(out, e.Envs...) +} + +// configMapGenerators converts kustomize's parsed configMapGenerator entries into the writer's +// model and names every feature that refuses the folder. An entry whose sources do not parse is +// left out of the model; kustomize fails the build on it, and that refusal names it. +func configMapGenerators(k kustypes.Kustomization, kustomizationPath string) ([]ConfigMapGeneratorEntry, []string) { + if len(k.ConfigMapGenerator) == 0 { + return nil, nil + } + dir := slashDir(kustomizationPath) + globalHashOff := k.GeneratorOptions != nil && k.GeneratorOptions.DisableNameSuffixHash + features := map[string]struct{}{} + var out []ConfigMapGeneratorEntry + for i, g := range k.ConfigMapGenerator { + switch g.Behavior { + case "", "create": + // kustomize ORs the two (types.MergeGlobalOptionsIntoLocal): either one disables it. + if !globalHashOff && (g.Options == nil || !g.Options.DisableNameSuffixHash) { + features[featureGeneratorNameHash] = struct{}{} + } + default: + // Refused whatever its hash setting, so the hash is not named beside it: disabling + // it would not make the entry adoptable. + features[featureGeneratorBehavior] = struct{}{} + } + entry, ok := generatorEntry(g, kustomizationPath, dir, i) + if ok { + out = append(out, entry) + } + } + return out, sortedKeysOfSet(features) +} + +// generatorEntry resolves one entry's sources. ok is false when a files: source does not parse +// the way kustomize parses it. +func generatorEntry( + g kustypes.ConfigMapArgs, + kustomizationPath, dir string, + index int, +) (ConfigMapGeneratorEntry, bool) { + entry := ConfigMapGeneratorEntry{ + KustomizationPath: kustomizationPath, + Index: index, + Name: g.Name, + Namespace: g.Namespace, + Literals: append([]string(nil), g.LiteralSources...), + } + for _, source := range g.FileSources { + key, file, ok := parseFileSource(source) + if !ok { + return ConfigMapGeneratorEntry{}, false + } + entry.Files = append(entry.Files, GeneratorFileSource{Key: key, Path: cleanJoin(dir, file)}) + } + for _, env := range g.EnvSources { + entry.Envs = append(entry.Envs, cleanJoin(dir, env)) + } + return entry, true +} + +// parseFileSource splits a files: source the way kustomize does (generators.ParseFileSource, +// which is internal): "path" keys the value by the file's base name, "key=path" names the key, +// and anything else is an error. +func parseFileSource(source string) (string, string, bool) { + switch strings.Count(source, "=") { + case 0: + return path.Base(source), source, true + case 1: + key, file, _ := strings.Cut(source, "=") + return key, file, key != "" && file != "" + default: + return "", "", false + } +} + +// KustomizationGeneratorInputs returns the local files a kustomization's configMapGenerator +// entries read, relative to its own directory. The live writer follows them alongside +// KustomizationBuildRefs, so a base outside spec.path is rendered with its generator inputs. +func KustomizationGeneratorInputs(content []byte) []string { + var k kustypes.Kustomization + if err := k.Unmarshal(content); err != nil { + return nil + } + k.FixKustomization() + var out []string + for _, g := range k.ConfigMapGenerator { + for _, source := range g.FileSources { + if _, file, ok := parseFileSource(source); ok { + out = append(out, file) + } + } + out = append(out, g.EnvSources...) + } + return out +} + +// AttachGeneratorInputs moves every file a configMapGenerator reads into scan.InputFiles: out of +// YAMLFiles, so a settings file is never indexed as a manifest or refused as non-KRM YAML, and +// out of Foreign, so a .env or .properties file is not refused as foreign content. read returns a +// scanned path's bytes; it is called only for files the walk did not already read. +// +// The inputs are named context, kept in the scan so the render can read them and the writer can +// edit them, and never modelled as documents. A file that a kustomization also lists under +// resources: stays a resource: kustomize renders it as an object, so it must be mirrored. +func AttachGeneratorInputs(scan *FolderScan, read func(path string) ([]byte, bool)) { + kusts := parseKustomizations(scan.YAMLFiles) + inputs := generatorInputSet(kusts) + if len(inputs) == 0 { + return + } + resources := reachedResourceFiles(kusts) + a := inputAttacher{scan: scan, read: read, have: map[string]struct{}{}} + a.isInput = func(p string) bool { + p = filepathToSlash(p) + _, input := inputs[p] + _, resource := resources[p] + return input && !resource + } + for _, f := range scan.InputFiles { + a.have[filepathToSlash(f.Path)] = struct{}{} + } + a.fromYAML() + a.fromForeign() + for _, p := range scan.NonYAML { + if a.isInput(p) { + a.readInput(p) + } + } + sort.Slice(scan.InputFiles, func(i, j int) bool { return scan.InputFiles[i].Path < scan.InputFiles[j].Path }) +} + +// generatorInputSet is every file a configMapGenerator entry of the scan reads. +func generatorInputSet(kusts map[string]*kustomizationDoc) map[string]struct{} { + inputs := map[string]struct{}{} + for _, doc := range kusts { + for _, entry := range doc.configMapGenerators { + for _, p := range entry.inputPaths() { + if p != "" { + inputs[p] = struct{}{} + } + } + } + } + return inputs +} + +// inputAttacher moves generator inputs into scan.InputFiles, each at most once. +type inputAttacher struct { + scan *FolderScan + read func(string) ([]byte, bool) + isInput func(string) bool + have map[string]struct{} +} + +// fromYAML moves the YAML files that are inputs, already read by the walk. +func (a *inputAttacher) fromYAML() { + kept := a.scan.YAMLFiles[:0:0] + for _, f := range a.scan.YAMLFiles { + if !a.isInput(f.Path) { + kept = append(kept, f) + continue + } + if _, dup := a.have[filepathToSlash(f.Path)]; !dup { + a.scan.InputFiles = append(a.scan.InputFiles, f) + a.have[filepathToSlash(f.Path)] = struct{}{} + } + } + a.scan.YAMLFiles = kept +} + +// fromForeign reads the foreign files that are inputs and stops refusing them. +func (a *inputAttacher) fromForeign() { + kept := a.scan.Foreign[:0:0] + for _, f := range a.scan.Foreign { + if f.Kind == ForeignFile && a.isInput(f.Path) && a.readInput(f.Path) { + continue + } + kept = append(kept, f) + } + a.scan.Foreign = kept +} + +// readInput reads one input the walk did not, reporting whether it is now attached. +func (a *inputAttacher) readInput(p string) bool { + if _, dup := a.have[filepathToSlash(p)]; dup { + return true + } + content, ok := a.read(p) + if !ok { + return false + } + a.scan.InputFiles = append(a.scan.InputFiles, manifestedit.FileContent{Path: p, Content: content}) + a.have[filepathToSlash(p)] = struct{}{} + return true +} + +// RenderFiles is every file a render reads: the YAML files and the generator inputs. +func (s FolderScan) RenderFiles() []manifestedit.FileContent { + if len(s.InputFiles) == 0 { + return s.YAMLFiles + } + out := make([]manifestedit.FileContent, 0, len(s.YAMLFiles)+len(s.InputFiles)) + out = append(out, s.YAMLFiles...) + return append(out, s.InputFiles...) +} diff --git a/internal/manifestanalyzer/generators_test.go b/internal/manifestanalyzer/generators_test.go new file mode 100644 index 000000000..0aca5ef88 --- /dev/null +++ b/internal/manifestanalyzer/generators_test.go @@ -0,0 +1,446 @@ +// SPDX-License-Identifier: Apache-2.0 + +package manifestanalyzer + +import ( + "context" + "fmt" + "strings" + "testing" + "testing/fstest" + + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + + "github.com/ConfigButler/gitops-reverser/internal/git/manifestedit" + "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// coffeeKustomization is the issue's own example: a settings file and an env file, each wrapped +// by a configMapGenerator with the name hash disabled. +const coffeeKustomization = `apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: coffee +resources: + - deployment.yaml +configMapGenerator: + - name: coffee-menu + files: + - menu.yaml + - name: coffee-flags + envs: + - flags.env +generatorOptions: + disableNameSuffixHash: true +` + +const coffeeDeployment = `apiVersion: apps/v1 +kind: Deployment +metadata: + name: coffee +spec: + template: + spec: + containers: + - name: coffee + image: ghcr.io/example/coffee:1.0.0 +` + +const coffeeMenu = "# Prices in cents.\nproducts:\n - sku: espresso\n priceCents: 275\n" + +func coffeeFS() fstest.MapFS { + return fstest.MapFS{ + "kustomization.yaml": {Data: []byte(coffeeKustomization)}, + "deployment.yaml": {Data: []byte(coffeeDeployment)}, + "menu.yaml": {Data: []byte(coffeeMenu)}, + "flags.env": {Data: []byte("# Turn off during the rush.\nENABLE_VOUCHERS=true\nSHOW_BANNER=true\n")}, + } +} + +// The inputs of a supported generator are named context: the folder is accepted, the .env file is +// not foreign content, and the settings file is not refused as non-KRM YAML. +func TestAccept_ConfigMapGeneratorWithTheHashDisabled(t *testing.T) { + store := buildStoreFS(context.Background(), coffeeFS(), nil, WriterAllowlist()) + acc := AcceptStructureOnly(store) + require.True(t, acc.Accepted, "issues: %+v", acc.Issues) + require.Empty(t, store.Foreign) + require.Equal(t, map[string]struct{}{"flags.env": {}, "menu.yaml": {}}, store.GeneratorInputs) + require.NotContains(t, store.FilesByPath, "menu.yaml", "a generator input is never a managed file") +} + +func TestConfigMapGenerators_Features(t *testing.T) { + cases := map[string]struct { + kustomization string + want []string + }{ + "hash disabled globally": { + kustomization: "configMapGenerator:\n - name: a\n literals: [k=v]\n" + + "generatorOptions:\n disableNameSuffixHash: true\n", + }, + "hash disabled on the entry": { + kustomization: "configMapGenerator:\n - name: a\n literals: [k=v]\n" + + " options:\n disableNameSuffixHash: true\n", + }, + "hash left on": { + kustomization: "configMapGenerator:\n - name: a\n literals: [k=v]\n", + want: []string{featureGeneratorNameHash}, + }, + "behavior merge is refused, and the hash is not named beside it": { + kustomization: "configMapGenerator:\n - name: a\n behavior: merge\n literals: [k=v]\n", + want: []string{featureGeneratorBehavior}, + }, + "behavior create is the default": { + kustomization: "configMapGenerator:\n - name: a\n behavior: create\n literals: [k=v]\n" + + "generatorOptions:\n disableNameSuffixHash: true\n", + }, + "secretGenerator stays refused": { + kustomization: "secretGenerator:\n - name: a\n literals: [k=v]\n" + + "generatorOptions:\n disableNameSuffixHash: true\n", + want: []string{"secretGenerator"}, + }, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + _, features := parseKustomization([]byte(tc.kustomization), "kustomization.yaml", nil) + if len(tc.want) == 0 { + require.Empty(t, features) + return + } + require.Equal(t, tc.want, features) + }) + } +} + +// The hash refusal is the one generator refusal its author can clear, so the message says how. +func TestAccept_ConfigMapGeneratorWithTheHashNamesTheAction(t *testing.T) { + fsys := coffeeFS() + fsys["kustomization.yaml"] = &fstest.MapFile{Data: []byte( + strings.Replace(coffeeKustomization, "generatorOptions:\n disableNameSuffixHash: true\n", "", 1))} + acc := AcceptStructureOnly(buildStoreFS(context.Background(), fsys, nil, WriterAllowlist())) + require.False(t, acc.Accepted) + require.Len(t, acc.Issues, 1) + issue := acc.Issues[0] + require.Equal(t, IssueUnsupportedKustomize, issue.Kind) + require.True(t, issue.Solvable) + require.Equal(t, ActorRepositoryAuthor, issue.Actor) + require.Contains(t, issue.Message, "set disableNameSuffixHash: true") +} + +// A file both listed under resources: and read by a generator stays a resource: kustomize renders +// it as an object, so it must be mirrored, never hidden as an input. +func TestAttachGeneratorInputs_AResourceStaysAResource(t *testing.T) { + scan := FolderScan{YAMLFiles: genFiles{ + { + "kustomization.yaml", + "resources: [cm.yaml]\nconfigMapGenerator:\n - name: copy\n files: [cm.yaml, notes.yaml]\n", + }, + {"cm.yaml", "apiVersion: v1\nkind: ConfigMap\nmetadata: {name: cm}\n"}, + {"notes.yaml", "just: notes\n"}, + }.files(), NonYAML: []string{"flags.env"}, Foreign: []ForeignEntry{{Path: "flags.env", Kind: ForeignFile}}} + AttachGeneratorInputs(&scan, func(string) ([]byte, bool) { return nil, false }) + + require.ElementsMatch(t, []string{"cm.yaml", "kustomization.yaml"}, pathsOf(scan.YAMLFiles)) + require.Equal(t, []string{"notes.yaml"}, pathsOf(scan.InputFiles)) + require.Len(t, scan.Foreign, 1, "a foreign file no generator names stays foreign") +} + +// TestReadGeneratorSources_AgreesWithKustomize asks kustomize rather than our reading of it: the +// env file below holds every line shape kv.keyValuesFromLine treats specially, and the data we +// derive must be the data the real build generates. +func TestReadGeneratorSources_AgreesWithKustomize(t *testing.T) { + env := "\xEF\xBB\xBFFIRST=after a byte-order mark\r\n" + + "# a comment\n" + + " INDENTED=kept \n" + + "\n" + + "EQUALS=a=b=c\n" + + "QUOTED=\"not unquoted\"\n" + + "BARE\n" + + "LAST=no newline" + files := genFiles{ + {"kustomization.yaml", "configMapGenerator:\n - name: probe\n envs: [probe.env]\n" + + " literals: [\"LIT='quoted'\"]\n files: [key=probe.txt]\n" + + "generatorOptions:\n disableNameSuffixHash: true\n"}, + {"probe.env", env}, + {"probe.txt", "whole\nfile\n"}, + }.files() + + rendered, err := renderRoot(files, ".") + require.NoError(t, err) + require.Len(t, rendered, 1) + want, _, _ := unstructured.NestedStringMap(rendered[0].Object.Object, "data") + + doc, features := parseKustomization(files[0].Content, "kustomization.yaml", nil) + require.Empty(t, features) + got, err := readGeneratorSources(&doc.configMapGenerators[0], readerOf(files)) + require.NoError(t, err) + require.Equal(t, want, got.data) +} + +func TestSetEnvValue(t *testing.T) { + cases := map[string]struct { + in, key, value, want string + }{ + "edits the line in place": {"# c\nA=1\nB=2\n", "A", "9", "# c\nA=9\nB=2\n"}, + "keeps the indent": {" A=1\n", "A", "9", " A=9\n"}, + "keeps a CRLF terminator": {"A=1\r\nB=2\r\n", "A", "9", "A=9\r\nB=2\r\n"}, + "keeps the byte-order mark": {"\xEF\xBB\xBFA=1\n", "A", "9", "\xEF\xBB\xBFA=9\n"}, + "gives a bare key a value": {"A\n", "A", "9", "A=9\n"}, + "appends a new key": {"A=1\n", "B", "2", "A=1\nB=2\n"}, + "appends after a last line": {"A=1", "B", "2", "A=1\nB=2\n"}, + "appends to an empty file": {"", "B", "2", "B=2\n"}, + "a commented key is not a key": {"# A=1\n", "A", "2", "# A=1\nA=2\n"}, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + require.Equal(t, tc.want, string(setEnvValue([]byte(tc.in), tc.key, tc.value))) + }) + } +} + +func TestRemoveEnvKey(t *testing.T) { + require.Equal(t, "# c\nB=2\n", string(removeEnvKey([]byte("# c\nA=1\nB=2\n"), "A"))) + require.Equal(t, "A=1\n", string(removeEnvKey([]byte("A=1\nB=2"), "B"))) + require.Equal(t, "A=1\n", string(removeEnvKey([]byte("A=1\n"), "Z")), "an absent key is a no-op") +} + +// generatedStore builds the writer's store over fsys and returns the generated object the live +// object resolves to. +func generatedFor(t *testing.T, fsys fstest.MapFS, live *unstructured.Unstructured) []*GeneratedObject { + t.Helper() + store := buildStoreFS(context.Background(), fsys, nil, WriterAllowlist()) + return store.GeneratedObjectsFor(live) +} + +func liveConfigMap(namespace, name string, data map[string]interface{}) *unstructured.Unstructured { + obj := &unstructured.Unstructured{Object: map[string]interface{}{ + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": map[string]interface{}{"name": name, "namespace": namespace}, + }} + if data != nil { + obj.Object["data"] = data + } + return obj +} + +func TestPlanGeneratedWrite_RoutesEachKey(t *testing.T) { + fsys := coffeeFS() + read := func(p string) ([]byte, bool) { + f, ok := fsys[p] + if !ok { + return nil, false + } + return f.Data, true + } + live := liveConfigMap("coffee", "coffee-flags", map[string]interface{}{ + "ENABLE_VOUCHERS": "false", + "SHOW_TIPS": "true", + }) + generated := generatedFor(t, fsys, live) + require.Len(t, generated, 1) + + writes, err := PlanGeneratedWrite(generated[0], live, "", read) + require.NoError(t, err) + require.Equal(t, []GeneratedWrite{{ + Path: "flags.env", + Content: []byte("# Turn off during the rush.\nENABLE_VOUCHERS=false\nSHOW_TIPS=true\n"), + }}, writes) + + // In sync: nothing to write. + inSync := liveConfigMap("coffee", "coffee-menu", map[string]interface{}{"menu.yaml": coffeeMenu}) + generated = generatedFor(t, fsys, inSync) + require.Len(t, generated, 1) + writes, err = PlanGeneratedWrite(generated[0], inSync, "", read) + require.NoError(t, err) + require.Empty(t, writes) +} + +func TestPlanGeneratedWrite_Refusals(t *testing.T) { + sharedBase := fstest.MapFS{ + "base/kustomization.yaml": {Data: []byte("configMapGenerator:\n - name: menu\n envs: [menu.env]\n" + + "generatorOptions:\n disableNameSuffixHash: true\n")}, + "base/menu.env": {Data: []byte("A=1\n")}, + "overlays/a/kustomization.yaml": {Data: []byte("resources: [../../base]\n")}, + "overlays/b/kustomization.yaml": {Data: []byte("resources: [../../base]\n")}, + "overlays/b/unrelated-config.yaml": {Data: []byte("apiVersion: v1\nkind: ConfigMap\nmetadata: {name: x}\n")}, + } + readFrom := func(fsys fstest.MapFS) func(string) ([]byte, bool) { + return func(p string) ([]byte, bool) { + f, ok := fsys[p] + if !ok { + return nil, false + } + return f.Data, true + } + } + + cases := map[string]struct { + fsys fstest.MapFS + live *unstructured.Unstructured + writeScope string + want string + }{ + "a base two overlays render shares its inputs": { + fsys: sharedBase, + live: liveConfigMap("", "menu", map[string]interface{}{"A": "2"}), + want: "rendered by 2 render roots (overlays/a, overlays/b)", + }, + "a value with a newline has no env line to hold it": { + fsys: coffeeFS(), + live: liveConfigMap("coffee", "coffee-flags", map[string]interface{}{ + "ENABLE_VOUCHERS": "true\nfalse", "SHOW_BANNER": "true", + }), + want: `the value of key "ENABLE_VOUCHERS" spans lines`, + }, + "a generator outside the write jail is base-owned": { + fsys: coffeeFS(), + live: liveConfigMap("coffee", "coffee-flags", map[string]interface{}{"ENABLE_VOUCHERS": "false"}), + writeScope: "overlays/prod", + want: "the generator lives in kustomization.yaml, which this target reads and never writes", + }, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + generated := generatedFor(t, tc.fsys, tc.live) + require.NotEmpty(t, generated) + _, err := PlanGeneratedWrite(generated[0], tc.live, tc.writeScope, readFrom(tc.fsys)) + require.ErrorIs(t, err, errGeneratedWrite) + require.ErrorContains(t, err, tc.want) + }) + } +} + +// Two entries of one name, told apart only by their own namespace: the live object's namespace +// picks the entry, and an object neither entry's namespace names has no single source. +func TestGeneratedObjectsFor_EntriesSharingAName(t *testing.T) { + fsys := fstest.MapFS{ + "kustomization.yaml": {Data: []byte("configMapGenerator:\n" + + " - {name: menu, namespace: eu, literals: [A=1]}\n" + + " - {name: menu, namespace: us, literals: [A=2]}\n" + + "generatorOptions:\n disableNameSuffixHash: true\n")}, + } + generated := generatedFor(t, fsys, liveConfigMap("us", "menu", nil)) + require.Len(t, generated, 1) + require.NotNil(t, generated[0].Entry) + require.Equal(t, "us", generated[0].Entry.Namespace) + + require.Empty(t, generatedFor(t, fsys, liveConfigMap("ap", "menu", nil))) +} + +type genFile struct{ path, content string } + +type genFiles []genFile + +func (g genFiles) files() []manifestedit.FileContent { + out := make([]manifestedit.FileContent, 0, len(g)) + for _, f := range g { + out = append(out, manifestedit.FileContent{Path: f.path, Content: []byte(f.content)}) + } + return out +} + +func pathsOf(files []manifestedit.FileContent) []string { + out := make([]string, 0, len(files)) + for _, f := range files { + out = append(out, f.Path) + } + return out +} + +func readerOf(files []manifestedit.FileContent) func(string) ([]byte, bool) { + return func(p string) ([]byte, bool) { + for _, f := range files { + if f.Path == p { + return f.Content, true + } + } + return nil, false + } +} + +// A delete keeps an input another entry also reads, and an input outside the write jail; a +// generator in a read-only base is refused, as an edit to it is. +func TestPlanGeneratedDelete(t *testing.T) { + fsys := fstest.MapFS{ + "kustomization.yaml": {Data: []byte("configMapGenerator:\n" + + " - {name: a, envs: [shared.env, own.env]}\n" + + " - {name: b, envs: [shared.env]}\n" + + "generatorOptions:\n disableNameSuffixHash: true\n")}, + "shared.env": {Data: []byte("S=1\n")}, + "own.env": {Data: []byte("O=1\n")}, + } + read := func(p string) ([]byte, bool) { + f, ok := fsys[p] + if !ok { + return nil, false + } + return f.Data, true + } + generated := generatedFor(t, fsys, liveConfigMap("", "a", nil)) + require.Len(t, generated, 1) + + writes, err := PlanGeneratedDelete(generated[0], "", read) + require.NoError(t, err) + require.Equal(t, []GeneratedWrite{ + {Path: "kustomization.yaml", Content: []byte("configMapGenerator:\n" + + " - {name: b, envs: [shared.env]}\n" + + "generatorOptions:\n disableNameSuffixHash: true\n")}, + {Path: "own.env"}, + }, writes, "shared.env is read by entry b too, so it stays") + + _, err = PlanGeneratedDelete(generated[0], "overlays/prod", read) + require.ErrorContains(t, err, "which this target reads and never writes") +} + +// TestLiteralValue_AgreesWithKustomize writes literals the way the planner does and asks a real +// build what they come back as: a value that is itself quoted must survive kustomize removing one +// pair of surrounding quotes. +func TestLiteralValue_AgreesWithKustomize(t *testing.T) { + values := []string{`plain`, `"double"`, `'single'`, `"unbalanced`, `a=b`, `'`, ``, "two\nlines"} + edits := make([]manifestedit.LiteralEdit, 0, len(values)) + want := map[string]string{} + for i, v := range values { + key := fmt.Sprintf("K%d", i) + edits = append(edits, manifestedit.LiteralEdit{Key: key, Value: v}) + want[key] = v + } + kustomization := []byte("configMapGenerator:\n - name: probe\n literals: [SEED=x]\n" + + "generatorOptions:\n disableNameSuffixHash: true\n") + res, diags := manifestedit.EditConfigMapGeneratorLiterals("kustomization.yaml", kustomization, 0, "probe", edits) + require.Equal(t, manifestedit.EditPatched, res.Mode, "%v", diags) + + rendered, err := renderRoot([]manifestedit.FileContent{{Path: "kustomization.yaml", Content: res.Content}}, ".") + require.NoError(t, err) + got, _, _ := unstructured.NestedStringMap(rendered[0].Object.Object, "data") + want["SEED"] = "x" + require.Equal(t, want, got) +} + +// The sweep treats a generated object as it treats a document: dropped only when the registry +// names it, it is in the plan's scope, and the policy drops orphans. +func TestBuildPlan_SweepsAGeneratedObject(t *testing.T) { + fsys := coffeeFS() + drops := func(store *ManifestStore, inScope func(types.ResourceIdentifier) bool) []PlanAction { + plan := BuildScopedPlan(store, nil, nil, Policy{Sweep: SweepDropOrphans}, inScope) + var out []PlanAction + for _, a := range plan.Actions { + if a.Generated { + out = append(out, a) + } + } + return out + } + followable := buildStoreFS(context.Background(), fsys, snapMapper(), WriterAllowlist()) + + got := drops(followable, allInScope) + require.Len(t, got, 2, "both generated ConfigMaps are absent from the empty snapshot") + require.Equal(t, PlanDropOrphan, got[0].Kind) + require.Equal(t, "kustomization.yaml", got[0].Ref.FilePath) + require.Equal(t, "coffee", got[0].Resource.Namespace) + + deploymentsOnly := func(id types.ResourceIdentifier) bool { return id.Resource == "deployments" } + require.Empty(t, drops(followable, deploymentsOnly), "a per-type plan sweeps only its own type") + + structureOnly := buildStoreFS(context.Background(), fsys, nil, WriterAllowlist()) + require.Empty(t, drops(structureOnly, allInScope), "a scan with no registry names nothing to drop") +} diff --git a/internal/manifestanalyzer/gittargetignore.go b/internal/manifestanalyzer/gittargetignore.go index fcc2c5189..e521e8b5c 100644 --- a/internal/manifestanalyzer/gittargetignore.go +++ b/internal/manifestanalyzer/gittargetignore.go @@ -363,7 +363,10 @@ func foreignMessage(f ForeignEntry) string { // every folder walker produces, so the analyzer scan and the live writer feed the store // and the acceptance gate identically. type FolderScan struct { - YAMLFiles []manifestedit.FileContent + YAMLFiles []manifestedit.FileContent + // InputFiles are the files a configMapGenerator reads, YAML or not: render inputs the writer + // may edit, never modelled as documents. See AttachGeneratorInputs. + InputFiles []manifestedit.FileContent NonYAML []string Foreign []ForeignEntry Ignore *IgnoreMatcher diff --git a/internal/manifestanalyzer/kustomization_parse.go b/internal/manifestanalyzer/kustomization_parse.go index 6cf744bca..c399ac260 100644 --- a/internal/manifestanalyzer/kustomization_parse.go +++ b/internal/manifestanalyzer/kustomization_parse.go @@ -35,12 +35,14 @@ func supportedKustomizationFields() map[string]struct{} { "TypeMeta": {}, // apiVersion / kind "MetaData": {}, // the kustomization's own metadata; renders nothing - // The modelled subset: the contextual-namespace walk plus the two - // edit-through channels. - "Namespace": {}, - "Resources": {}, - "Images": {}, - "Replicas": {}, + // The modelled subset: the contextual-namespace walk plus the edit-through channels. + // A configMapGenerator is modelled only with the name hash disabled and no behavior: + // merge or replace; configMapGenerators names the rest. + "Namespace": {}, + "Resources": {}, + "Images": {}, + "Replicas": {}, + "ConfigMapGenerator": {}, // TOLERATED, NOT AUTHORED: kustomize applies it, we mirror what it renders, nothing is // routed INTO it. Only safe because the projection leaves every field the BUILD supplies @@ -59,7 +61,7 @@ func supportedKustomizationFields() map[string]struct{} { "CommonAnnotations": {}, "BuildMetadata": {}, - "GeneratorOptions": {}, // inert on its own; every generator is refused below + "GeneratorOptions": {}, // read with configMapGenerator; secretGenerator is refused "SortOptions": {}, // output ordering only } } @@ -146,6 +148,11 @@ func parseKustomization(content []byte, path string, tree map[string][]byte) (*k if doc.replicas, ok = replicaOverrides(k.Replicas, path); !ok { features[featureMalformedReplicas] = struct{}{} } + var generatorFeatures []string + doc.configMapGenerators, generatorFeatures = configMapGenerators(k, path) + for _, f := range generatorFeatures { + features[f] = struct{}{} + } doc.patches = patchPaths(k.Patches, slashDir(path), tree) for _, refusal := range patchRefusals(k.Patches, slashDir(path), tree) { features[refusal] = struct{}{} diff --git a/internal/manifestanalyzer/kustomize_render.go b/internal/manifestanalyzer/kustomize_render.go index f3e0cc34b..1912a0f29 100644 --- a/internal/manifestanalyzer/kustomize_render.go +++ b/internal/manifestanalyzer/kustomize_render.go @@ -97,8 +97,11 @@ type renderedObject struct { // Object is the rendered result: what the GitOps controller will apply. Object *unstructured.Unstructured // OriginPath is the source file that produced it, relative to the scan root. - // Empty for a generated resource (which the acceptance gate refuses anyway). + // Empty for a generated resource. OriginPath string + // GeneratedIn is the kustomization whose configMapGenerator produced it, relative to the + // scan root. Empty for every object with an OriginPath, and for one another generator made. + GeneratedIn string // TransformedBy lists the kustomizations whose transformers touched it, in // build order (innermost base first) — the override chain, from kustomize. TransformedBy []transformation @@ -327,6 +330,7 @@ func collectRendered(resMap resmap.ResMap, rootDir string) ([]renderedObject, er ro := renderedObject{ Object: obj, OriginPath: originOf(obj, rootDir), + GeneratedIn: configMapGeneratedIn(obj, rootDir), TransformedBy: transformationsOf(obj, rootDir), } unstructured.RemoveNestedField(obj.Object, "metadata", "annotations", originAnnotation) @@ -355,6 +359,44 @@ func originOf(obj *unstructured.Unstructured, rootDir string) string { return path.Clean(path.Join(rootDir, origin.Path)) } +// configMapGeneratedIn reads the kustomization a ConfigMapGenerator ran in, normalised to +// scan-root-relative. kustomize records a generated object's origin as the kustomization and the +// builtin that produced it, not as a file: +// +// config.kubernetes.io/origin: | +// configuredIn: ../base/kustomization.yaml +// configuredBy: {apiVersion: builtin, kind: ConfigMapGenerator} +func configMapGeneratedIn(obj *unstructured.Unstructured, rootDir string) string { + raw := obj.GetAnnotations()[originAnnotation] + if raw == "" { + return "" + } + var origin struct { + ConfiguredIn string `json:"configuredIn"` + ConfiguredBy struct { + Kind string `json:"kind"` + } `json:"configuredBy"` + } + if err := yaml.Unmarshal([]byte(raw), &origin); err != nil || + origin.ConfiguredIn == "" || origin.ConfiguredBy.Kind != configMapGeneratorKind { + return "" + } + return path.Clean(path.Join(rootDir, origin.ConfiguredIn)) +} + +// configMapGeneratorKind is the builtin behind configMapGenerator:, as kustomize names it in a +// generated object's origin. +const configMapGeneratorKind = "ConfigMapGenerator" + +// sourceKey is the file the object is attributed to: its origin file, or the kustomization whose +// generator produced it. A kustomization is never a resource's origin, so the two cannot collide. +func (ro renderedObject) sourceKey() string { + if ro.OriginPath != "" { + return ro.OriginPath + } + return ro.GeneratedIn +} + // transformationsOf reads the ordered transformer chain kustomize applied, so the // writer knows which kustomizations govern this object and in what order. func transformationsOf(obj *unstructured.Unstructured, rootDir string) []transformation { diff --git a/internal/manifestanalyzer/kustomize_render_hostile_test.go b/internal/manifestanalyzer/kustomize_render_hostile_test.go index ddea42219..71ade8554 100644 --- a/internal/manifestanalyzer/kustomize_render_hostile_test.go +++ b/internal/manifestanalyzer/kustomize_render_hostile_test.go @@ -125,7 +125,7 @@ func TestRenderChains_CycleIsBuiltAndRefused(t *testing.T) { kusts := parseKustomizations(files) require.Empty(t, renderRoots(kusts), "a cycle has no render root; that is the whole problem") - chains, _, failed := renderChains(files, kusts) + chains, _, failed, _ := renderChains(files, kusts) require.Empty(t, chains) require.Len(t, failed, 1, "the cycle must produce exactly one refusal, not one per member") diff --git a/internal/manifestanalyzer/kustomize_render_test.go b/internal/manifestanalyzer/kustomize_render_test.go index 502f45a03..186f53c10 100644 --- a/internal/manifestanalyzer/kustomize_render_test.go +++ b/internal/manifestanalyzer/kustomize_render_test.go @@ -51,7 +51,7 @@ func TestProjection_InSyncCorpusFolderIsANoOp(t *testing.T) { skipped++ t.Skipf("not renderable, and refused by the acceptance gate: %v", err) } - chains, _, _ := renderChains(root.files, parseKustomizations(root.files)) + chains, _, _, _ := renderChains(root.files, parseKustomizations(root.files)) for _, ro := range rendered { if ro.OriginPath == "" { diff --git a/internal/manifestanalyzer/override_chain.go b/internal/manifestanalyzer/override_chain.go index 09151259c..bf9ec341d 100644 --- a/internal/manifestanalyzer/override_chain.go +++ b/internal/manifestanalyzer/override_chain.go @@ -136,6 +136,8 @@ func markReachable(kusts map[string]*kustomizationDoc, dir string, covered map[s // moves every object it covers, so what a folder's files declare and what it renders to // are different answers whenever a layout transform is involved. // +// It also returns every object a configMapGenerator produced, per root, for generatedObjects. +// // It also returns, per kustomization path, the roots that FAILED to build. Those // must refuse the folder, and it is important to see why a silent skip would be // unsafe rather than merely unhelpful: a root that does not build yields no chain, @@ -148,10 +150,11 @@ func markReachable(kusts map[string]*kustomizationDoc, dir string, covered map[s func renderChains( files []manifestedit.FileContent, kusts map[string]*kustomizationDoc, -) (map[chainKey]*overrideAssignment, map[string]RenderedTypes, map[string]string) { +) (map[chainKey]*overrideAssignment, map[string]RenderedTypes, map[string]string, []generatedRender) { out := map[chainKey]*overrideAssignment{} inventory := map[string]RenderedTypes{} failed := map[string]string{} + var generated []generatedRender // One dye plan for the whole scan, so the nonces are stable across roots and a base // reached by two overlays is dyed identically in both. Building it costs no render. @@ -175,7 +178,12 @@ func renderChains( for _, ro := range rendered { if ro.OriginPath == "" { - continue // a generated resource: it has no source document to edit + // A generated resource has no source document; a configMapGenerator's is + // recorded so the writer can route an edit into the generator's inputs. + if ro.GeneratedIn != "" { + generated = append(generated, generatedRender{root: rootDir, in: ro.GeneratedIn, object: ro.Object}) + } + continue } key := chainKey{ originPath: ro.OriginPath, @@ -185,7 +193,7 @@ func renderChains( record(out, key, chainOf(ro, kusts), attribution[key]) } } - return out, inventory, failed + return out, inventory, failed, generated } // inventoryOf reduces one root's rendered objects to what a tool must know before it diff --git a/internal/manifestanalyzer/overrides_projection_test.go b/internal/manifestanalyzer/overrides_projection_test.go index d0f9e4664..0f590e6d5 100644 --- a/internal/manifestanalyzer/overrides_projection_test.go +++ b/internal/manifestanalyzer/overrides_projection_test.go @@ -35,7 +35,7 @@ func splitFixture( t.Helper() desired := &unstructured.Unstructured{Object: live} - assignments, _, failed := renderChains(files, parseKustomizations(files)) + assignments, _, failed, _ := renderChains(files, parseKustomizations(files)) require.Empty(t, failed, "the fixture must be a folder kustomize can build") assignment := assignments[chainKey{ diff --git a/internal/manifestanalyzer/overrides_test.go b/internal/manifestanalyzer/overrides_test.go index ec5278ad3..98818a1e1 100644 --- a/internal/manifestanalyzer/overrides_test.go +++ b/internal/manifestanalyzer/overrides_test.go @@ -276,7 +276,7 @@ spec: {Path: "kustomization.yaml", Content: []byte("resources:\n - deployment.yaml\n")}, } - assignments, _, failed := renderChains(files, parseKustomizations(files)) + assignments, _, failed, _ := renderChains(files, parseKustomizations(files)) require.Empty(t, failed) a := assignments[chainKey{originPath: "deployment.yaml", kind: "Deployment", name: "web"}] diff --git a/internal/manifestanalyzer/plan.go b/internal/manifestanalyzer/plan.go index d83cba966..cc1679d1c 100644 --- a/internal/manifestanalyzer/plan.go +++ b/internal/manifestanalyzer/plan.go @@ -103,6 +103,9 @@ type PlanAction struct { // Desired is the clean object Git should contain, set for create/patch/replace // and nil for removals and skips. Desired *unstructured.Unstructured + // Generated marks a PlanDropOrphan of an object a configMapGenerator renders rather than + // of a document: Ref names its kustomization, and the drop removes the generator entry. + Generated bool // Reason is a human-readable explanation, carried so renderers and status need // no recomputation. Reason string @@ -237,7 +240,9 @@ func BuildScopedPlan( collided: collidedIdentities(store), matched: map[*DocumentModel]bool{}, inScope: inScope, - sweep: policy.Sweep, + + matchedGenerated: map[*GeneratedObject]bool{}, + sweep: policy.Sweep, } // Desired side: create / patch / replace / skip for every cluster object, and @@ -263,6 +268,11 @@ func BuildScopedPlan( b.planGitOnly(dm) } } + // The render's other members: what a configMapGenerator renders is as much the folder's + // content as a document is, so an object the cluster no longer has is swept the same way. + for _, g := range store.generated { + b.planGeneratedOnly(g) + } sortActions(b.actions) return Plan{ @@ -286,8 +296,10 @@ type planBuilder struct { collided map[manifestedit.Identity]bool matched map[*DocumentModel]bool - actions []PlanAction - diags []manifestedit.Diagnostic + // matchedGenerated is the generated objects a desired object matched, the sweep's other half. + matchedGenerated map[*GeneratedObject]bool + actions []PlanAction + diags []manifestedit.Diagnostic // inScope gates the Git-only sweep to a subset of resolved resource identities. For the // whole-folder BuildPlan it is allInScope (always true); for a per-type reconcile/sweep // it matches one type's (group, resource), so out-of-scope documents are never dropped. @@ -323,6 +335,15 @@ func (b *planBuilder) planDesired(dr DesiredResource) { if b.nonClaiming[id] { return } + // A configMapGenerator renders it: there is nothing to create, and its write goes into the + // generator's inputs, which only the writer plans (PlanGeneratedWrite). It is matched, so + // the sweep keeps its entry. + if generated := b.store.GeneratedObjectsFor(obj); len(generated) > 0 { + for _, g := range generated { + b.matchedGenerated[g] = true + } + return + } b.actions = append(b.actions, PlanAction{ Kind: PlanCreate, Identity: id, @@ -384,6 +405,11 @@ func (b *planBuilder) protectFromSweep(dr DesiredResource) { "desired snapshot entry for %s has no object; protected from sweep (malformed entry, not a delete)", dr.Resource.Key()), } + for _, g := range b.store.generated { + if g.Resource != nil && *g.Resource == dr.Resource { + b.matchedGenerated[g] = true + } + } if dm := b.store.ByResourceIdentity[dr.Resource]; dm != nil { b.matched[dm] = true ref := b.docLoc[dm] @@ -430,14 +456,7 @@ func (b *planBuilder) planGitOnly(dm *DocumentModel) { // The target's prune policy keeps inferred deletions off. The drop is not planned // at all — not planned and then filtered — so it cannot reach the plan's action // list, its ordering, or the commit. Counting it is the only trace it leaves. - b.retained++ - if b.retainedByType == nil { - b.retainedByType = map[schema.GroupVersionResource]int{} - } - id := resourceOf(dm) - b.retainedByType[schema.GroupVersionResource{ - Group: id.Group, Version: id.Version, Resource: id.Resource, - }]++ + b.retain(resourceOf(dm)) return } b.actions = append(b.actions, PlanAction{ @@ -446,6 +465,41 @@ func (b *planBuilder) planGitOnly(dm *DocumentModel) { }) } +// retain counts one drop the prune policy suppressed. +func (b *planBuilder) retain(id types.ResourceIdentifier) { + b.retained++ + if b.retainedByType == nil { + b.retainedByType = map[schema.GroupVersionResource]int{} + } + b.retainedByType[schema.GroupVersionResource{ + Group: id.Group, Version: id.Version, Resource: id.Resource, + }]++ +} + +// planGeneratedOnly classifies one generated object that no desired object matched, as +// planGitOnly does a document: only one the registry names as a followable resource, in scope, +// is dropped, and only under a prune policy that drops orphans. +func (b *planBuilder) planGeneratedOnly(g *GeneratedObject) { + if b.matchedGenerated[g] || g.Resource == nil { + return + } + if b.inScope != nil && !b.inScope(*g.Resource) { + return + } + if !b.sweep.DropsOrphans() { + b.retain(*g.Resource) + return + } + b.actions = append(b.actions, PlanAction{ + Kind: PlanDropOrphan, + Ref: RecordRef{FilePath: g.GeneratedIn}, + Identity: g.Identity(), + Resource: *g.Resource, + Generated: true, + Reason: "generated resource absent from the cluster: its configMapGenerator entry is dropped", + }) +} + // actionFromDecision maps a manifestedit decision intent to a plan action kind. The // boolean is false for ActionNoChange, which produces no plan action: an in-sync // document is not a change. ActionDelete cannot arise here (desired is non-nil), so diff --git a/internal/manifestanalyzer/render_verify.go b/internal/manifestanalyzer/render_verify.go index f61061c16..9ed74612f 100644 --- a/internal/manifestanalyzer/render_verify.go +++ b/internal/manifestanalyzer/render_verify.go @@ -313,11 +313,12 @@ func sameObject(a, b *unstructured.Unstructured) bool { // renderedByKey indexes a render by the document each object came from. The key carries the // ORIGIN FILE, not just kind/name, so two same-named objects rendered from different source -// files stay distinct. +// files stay distinct. A generated object is keyed by the kustomization that generated it, which +// is also the SourcePath its write intent names. func renderedByKey(objects []renderedObject) map[chainKey]renderedObject { out := make(map[chainKey]renderedObject, len(objects)) for _, o := range objects { - out[chainKey{originPath: o.OriginPath, kind: o.Object.GetKind(), name: o.Object.GetName()}] = o + out[chainKey{originPath: o.sourceKey(), kind: o.Object.GetKind(), name: o.Object.GetName()}] = o } return out } diff --git a/internal/manifestanalyzer/solvable.go b/internal/manifestanalyzer/solvable.go index 2c56642b2..61656a9e5 100644 --- a/internal/manifestanalyzer/solvable.go +++ b/internal/manifestanalyzer/solvable.go @@ -87,6 +87,9 @@ func kustomizeFeatureClassification() map[string]Classification { featureMalformedReplicas: {Solvable: true, Actor: ActorRepositoryAuthor}, featurePatchOutsideTree: {Solvable: true, Actor: ActorRepositoryAuthor}, featureRenderFailed: {Solvable: true, Actor: ActorRepositoryAuthor}, + // A choice the author made, and one setting away from adoptable: with the hash off, + // the object keeps its name and the generator becomes writable. + featureGeneratorNameHash: {Solvable: true, Actor: ActorRepositoryAuthor}, // Read-only context this release does not fetch, and merge semantics it does not // model. Nobody can make the folder adoptable while it declares them. @@ -105,20 +108,20 @@ func kustomizeFeatureClassification() map[string]Classification { // Build-time synthesis, post-read rewriting, and plugin code: no source document // to route an edit back into. - featurePatchInline: {Solvable: false}, - featurePatchJSON6902: {Solvable: false}, - "components": {Solvable: false}, - "configMapGenerator": {Solvable: false}, - "secretGenerator": {Solvable: false}, - "generators": {Solvable: false}, - "transformers": {Solvable: false}, - "replacements": {Solvable: false}, - "vars": {Solvable: false}, - "validators": {Solvable: false}, - "namePrefix": {Solvable: false}, - "nameSuffix": {Solvable: false}, - "patchesStrategicMerge": {Solvable: false}, - "patchesJson6902": {Solvable: false}, + featurePatchInline: {Solvable: false}, + featurePatchJSON6902: {Solvable: false}, + "components": {Solvable: false}, + featureGeneratorBehavior: {Solvable: false}, + "secretGenerator": {Solvable: false}, + "generators": {Solvable: false}, + "transformers": {Solvable: false}, + "replacements": {Solvable: false}, + "vars": {Solvable: false}, + "validators": {Solvable: false}, + "namePrefix": {Solvable: false}, + "nameSuffix": {Solvable: false}, + "patchesStrategicMerge": {Solvable: false}, + "patchesJson6902": {Solvable: false}, } } @@ -164,13 +167,21 @@ func unsupportedKustomizeDetail(class Classification, features []string, decodeE return "kustomization uses an unsupported feature the operator cannot map back to editable source" } stem := "kustomization uses unsupported feature(s): " - if class.Solvable { - // Every solvable construct is a fault in the file or a path reaching out of the - // tree, so "fixed in the repository" holds for all five without naming a person - // the classification may not have named. + switch { + case class.Solvable && onlySettingFeatures(features): + // A deliberate setting rather than a mistake, so the file has no "fault"; the action + // the detail carries is the change. + stem = "kustomization uses a setting that can be changed in the repository: " + case class.Solvable: + // Every other solvable construct is a fault in the file or a path reaching out of the + // tree, so "fixed in the repository" holds for them all without naming a person the + // classification may not have named. stem = "kustomization has a fault that can be fixed in the repository: " } detail := stem + strings.Join(features, ", ") + if action := kustomizeFeatureActions(features); action != "" { + detail += " (" + action + ")" + } if decodeErr != "" { // "unparseable" on its own says nothing a user can act on. kustomize's decoder // knows exactly what is wrong — that a resources: is a string, or that the file @@ -180,6 +191,33 @@ func unsupportedKustomizeDetail(class Classification, features []string, decodeE return detail } +// onlySettingFeatures reports whether every feature is a setting the author chose, rather than a +// fault in the file. +func onlySettingFeatures(features []string) bool { + for _, f := range features { + if f != featureGeneratorNameHash { + return false + } + } + return len(features) > 0 +} + +// kustomizeFeatureActions names what the author can do about the features that have one action, +// joined in feature order. Most features have none: naming one for a construct that cannot be +// adopted would send someone off to change a file for nothing. +func kustomizeFeatureActions(features []string) string { + actions := map[string]string{ + featureGeneratorNameHash: "set disableNameSuffixHash: true in generatorOptions or on the entry", + } + var out []string + for _, f := range features { + if a, ok := actions[f]; ok { + out = append(out, a) + } + } + return strings.Join(out, "; ") +} + // retainedUnsupportedFeatures names the constructs behind one unsupported retention: the // parsed feature set, plus featureRenderFailed when kustomize could not build the root at // all. It returns nothing for a supported retention, so the field is empty exactly when diff --git a/internal/manifestanalyzer/solvable_test.go b/internal/manifestanalyzer/solvable_test.go index 3cead0f39..2c308e118 100644 --- a/internal/manifestanalyzer/solvable_test.go +++ b/internal/manifestanalyzer/solvable_test.go @@ -313,7 +313,7 @@ func TestAcceptanceEmissionPathsAreClassified(t *testing.T) { "unsupported kustomize, the support boundary": { fsys: fstest.MapFS{ "kustomization.yaml": {Data: []byte("apiVersion: kustomize.config.k8s.io/v1beta1\n" + - "kind: Kustomization\nconfigMapGenerator:\n - name: settings\n literals:\n - a=b\n")}, + "kind: Kustomization\nsecretGenerator:\n - name: settings\n literals:\n - a=b\n")}, }, policy: AcceptancePolicy{Allowlist: DefaultAllowlist()}, want: []wantIssue{{ @@ -435,7 +435,7 @@ func TestClassifyKustomizeFeatures_SolvableOnlyWhenEveryConstructIs(t *testing.T "unrecognised only": {[]string{"someFutureField"}, unsolvable}, "the author's file": {[]string{featureUnparseable}, solvable}, "context we do not read": {[]string{featureRemoteBase}, unsolvable}, - "the boundary": {[]string{"configMapGenerator"}, unsolvable}, + "the boundary": {[]string{"secretGenerator"}, unsolvable}, "solvable and not": {[]string{featureUnparseable, featureRemoteBase}, unsolvable}, "two unsolvable": {[]string{featureRemoteBase, "vars"}, unsolvable}, // An unrecognised construct is a construct we cannot vouch for, so the answer is the @@ -497,7 +497,7 @@ func TestRefusedStructuralIsClassified(t *testing.T) { want Classification }{ "a generator is not solvable": { - kustomization: kustHeader + "configMapGenerator:\n - name: settings\n literals:\n - a=b\n", + kustomization: kustHeader + "secretGenerator:\n - name: settings\n literals:\n - a=b\n", want: Classification{Solvable: false}, }, "helm inflation is not solvable": { @@ -560,8 +560,8 @@ func TestUnsupportedKustomizeDetailAgreesWithClassification(t *testing.T) { }, "an unsupported construct keeps the stem it always had": { class: Classification{Solvable: false}, - features: []string{"configMapGenerator", "namePrefix"}, - wantContains: "kustomization uses unsupported feature(s): configMapGenerator, namePrefix", + features: []string{"secretGenerator", "namePrefix"}, + wantContains: "kustomization uses unsupported feature(s): secretGenerator, namePrefix", }, "no features at all is the conservative sentence": { class: Classification{}, diff --git a/internal/manifestanalyzer/source_form_test.go b/internal/manifestanalyzer/source_form_test.go index 400553679..a98e91e92 100644 --- a/internal/manifestanalyzer/source_form_test.go +++ b/internal/manifestanalyzer/source_form_test.go @@ -199,7 +199,7 @@ spec: var gitRaw map[string]interface{} require.NoError(t, yaml.Unmarshal(contentOf(t, files, sourceDoc), &gitRaw)) - chains, _, _ := renderChains(files, parseKustomizations(files)) + chains, _, _, _ := renderChains(files, parseKustomizations(files)) rendered := chains[chainKey{originPath: sourceDoc, kind: "Deployment", name: "web"}].rendered _, _, err := SplitDesiredForOverrides(gitRaw, &unstructured.Unstructured{Object: live}, rendered, "") diff --git a/internal/manifestanalyzer/store.go b/internal/manifestanalyzer/store.go index 817886941..ecd813d44 100644 --- a/internal/manifestanalyzer/store.go +++ b/internal/manifestanalyzer/store.go @@ -91,6 +91,14 @@ type ManifestStore struct { // See docs/design/support-boundary/values-file-projection.md §2 (Move 1). ValueFileRefs map[string]struct{} + // GeneratorInputs is the set of files a configMapGenerator reads (FolderScan.InputFiles): + // named context the render reads and the writer may edit, never a document, never foreign. + GeneratorInputs map[string]struct{} + + // generated is every object a configMapGenerator renders, one per generator entry and + // namespace. See GeneratedObjectsFor. + generated []*GeneratedObject + // RenderedInventory records what each render root RENDERS TO, keyed by the root's // directory: which type lands in which namespace, from the objects kustomize actually // produced. It is what a tool must settle before it provisions from a folder, and it is @@ -425,7 +433,8 @@ func buildStore( kusts := parseKustomizations(yamlFiles) resourceFiles := resourceFilePaths(yamlFiles) nsAssignments := kustomizeNamespaceAssignments(kusts, resourceFiles) - ovAssignments, renderedInventory, renderFailures := renderChains(yamlFiles, kusts) + // The render reads the generator inputs too; the store models only the YAML files. + ovAssignments, renderedInventory, renderFailures, generated := renderChains(scan.RenderFiles(), kusts) store := &ManifestStore{ FilesByPath: map[string]*FileModel{}, @@ -452,8 +461,12 @@ func buildStore( reachedByMultipleRoots: renderRootFanIn(kusts), renderInputs: reachedResourceFiles(kusts), renderedDocuments: renderedDocumentsOf(ovAssignments), + GeneratorInputs: pathSet(scan.InputFiles), + generated: generatedObjects(generated, kusts), } + store.resolveGeneratedResources(ctx, lookup, settings.declaredNamespace) + hasNamedRecord := store.materializeRecords(ctx, inv.Records, materializeInputs{ lookup: lookup, allowlist: allowlist, @@ -838,6 +851,10 @@ type kustomizationDoc struct { // the scan root. They are BUILD INPUTS, not resources: retained outside the managed model, so // no live object is ever mirrored over one and no sweep ever deletes one. patches []string + // configMapGenerators are its configMapGenerator entries, with input paths resolved against + // the scan root. Entries that refuse the folder are still listed: their inputs are named + // context either way. + configMapGenerators []ConfigMapGeneratorEntry // features names every unmodelled feature, sorted — the refusal, in the user's own words. features []string unsupported bool // uses generators/components/remote bases/name(pre|suf)fix/an unauthorable patch/... diff --git a/internal/sanitize/types.go b/internal/sanitize/types.go index ea7ef8a2a..9a85939c4 100644 --- a/internal/sanitize/types.go +++ b/internal/sanitize/types.go @@ -108,11 +108,18 @@ func isOperationalLabel(key string) bool { // travel with it into every other. Sibling `kcp.io/` keys are not assumed to be // bookkeeping too, so this is not a prefix strip either. // +// `config.kubernetes.io/origin` and `alpha.config.kubernetes.io/transformations` are kustomize's +// build provenance, written by a build whose buildMetadata asks for them: the file and the +// transformers that produced the object. Flux asks for originAnnotations on its own whenever a +// kustomization.yaml has no resources: (fluxcd/pkg/kustomize, kustomize_generator.go), so a +// generator-only folder's objects carry one live. They describe the build, never the source. +// // Exercised end-to-end against a real Argo CD in // test/e2e/argocd_bi_directional_e2e_test.go. func isOperationalAnnotation(key string) bool { switch key { - case "argocd.argoproj.io/tracking-id", "argocd.argoproj.io/installation-id", "kcp.io/cluster": + case "argocd.argoproj.io/tracking-id", "argocd.argoproj.io/installation-id", "kcp.io/cluster", + "config.kubernetes.io/origin", "alpha.config.kubernetes.io/transformations": return true } diff --git a/internal/sanitize/types_test.go b/internal/sanitize/types_test.go index 64b0344b6..c89f8e990 100644 --- a/internal/sanitize/types_test.go +++ b/internal/sanitize/types_test.go @@ -291,6 +291,19 @@ func TestCleanAnnotations(t *testing.T) { "user-annotation": "kept", }, }, + { + // kustomize's build provenance, which Flux requests for a kustomization.yaml with no + // resources:, so a generator-only folder's live objects carry it. + name: "remove kustomize build provenance", + input: map[string]string{ + "config.kubernetes.io/origin": "configuredIn: kustomization.yaml\n", + "alpha.config.kubernetes.io/transformations": "- configuredIn: kustomization.yaml\n", + "config.kubernetes.io/local-config": "kept", + }, + expected: map[string]string{ + "config.kubernetes.io/local-config": "kept", + }, + }, { // The exact-key set exists precisely so these survive. A blanket // `argocd.argoproj.io/` prefix strip would silently drop a user's sync diff --git a/pkg/manifestanalyzer/folder_test.go b/pkg/manifestanalyzer/folder_test.go index 260bb06cb..f86c5621a 100644 --- a/pkg/manifestanalyzer/folder_test.go +++ b/pkg/manifestanalyzer/folder_test.go @@ -266,3 +266,45 @@ func TestFolderReport_JSONRoundTrip(t *testing.T) { require.NoError(t, json.Unmarshal(buf.Bytes(), &decoded)) require.Equal(t, report, decoded) } + +// The two configMapGenerator shapes of the generator write-through, a settings file and an env +// file, are accepted with the name hash disabled, and refused with it on, naming the setting. +func TestScanFolder_ConfigMapGenerator(t *testing.T) { + t.Parallel() + + const hashOff = "generatorOptions:\n disableNameSuffixHash: true\n" + const kustomization = `apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: coffee +configMapGenerator: + - name: coffee-menu + files: + - menu.yaml + - name: coffee-flags + envs: + - flags.env +` + write := func(t *testing.T, kust string) string { + t.Helper() + root := t.TempDir() + files := map[string]string{ + "kustomization.yaml": kust, + "menu.yaml": "# Prices in cents.\nproducts:\n - sku: espresso\n priceCents: 275\n", + "flags.env": "# Turn off during the rush.\nENABLE_VOUCHERS=true\n", + } + for name, body := range files { + require.NoError(t, os.WriteFile(filepath.Join(root, name), []byte(body), 0o600)) + } + return root + } + + report, err := manifestanalyzer.ScanFolder(context.Background(), write(t, kustomization+hashOff)) + require.NoError(t, err) + require.True(t, report.Status.Accepted, "issues: %+v", report.Status.Issues) + + report, err = manifestanalyzer.ScanFolder(context.Background(), write(t, kustomization)) + require.NoError(t, err) + require.False(t, report.Status.Accepted) + require.Len(t, report.Status.Issues, 1) + require.Contains(t, report.Status.Issues[0].Message, "set disableNameSuffixHash: true") +} diff --git a/test/e2e/config_map_generator_e2e_test.go b/test/e2e/config_map_generator_e2e_test.go new file mode 100644 index 000000000..c2e1245d5 --- /dev/null +++ b/test/e2e/config_map_generator_e2e_test.go @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: Apache-2.0 + +package e2e + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +const ( + configMapGeneratorFixtureRoot = "test/e2e/fixtures/config-map-generator" + configMapGeneratorGitPath = "apps/coffee" + configMapGeneratorTimeout = 2 * time.Minute +) + +// A ConfigMap a configMapGenerator renders, edited through the API, is written back into the +// generator's input file as a one-line diff, and Flux, building the folder from that commit, applies +// the same object again. Deleting one through the API removes its generator entry and input file, +// and Flux's next apply leaves it deleted. Flux applies the folder before the GitTarget exists, so the ConfigMap the +// spec edits is the one Flux made, and Flux's next apply is what proves the write converges. +var _ = Describe("Manager ConfigMap Generator", Label("manager", "config-map-generator", "flux"), Ordered, func() { + const ( + providerName = "config-map-generator-provider" + targetName = "config-map-generator-target" + ruleName = "config-map-generator-rule" + ) + var ( + testNs, fluxName, fluxSecretName string + repo *RepoArtifacts + ) + + BeforeAll(func() { + testNs = testNamespaceFor("manager-config-map-generator") + // Flux labels every object it applies with the Kustomization's name, and a label value + // holds at most 63 bytes, so the name cannot carry the namespace. + fluxName = fmt.Sprintf("cmg-%d", GinkgoRandomSeed()) + fluxSecretName = fluxName + "-git" + _, _ = kubectlRun("create", "namespace", testNs) + + By("setting up the Gitea repo, its credentials and the GitProvider") + repo = SetupRepo(resolveE2EContext(), testNs, fmt.Sprintf("e2e-config-map-generator-%d", GinkgoRandomSeed())) + _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) + createReadyGitProvider(providerName, testNs, repo.GitSecretHTTP, repo.RepoURLHTTP) + }) + + AfterAll(func() { + cleanupWatchRule(ruleName, testNs) + cleanupGitTarget(targetName, testNs) + cleanupNamespacedResource(testNs, "gitprovider", providerName) + cleanupNamespacedResource("flux-system", "kustomization", fluxName) + cleanupNamespacedResource("flux-system", "gitrepository", fluxName) + cleanupNamespacedResource("flux-system", "secret", fluxSecretName) + cleanupNamespace(testNs) + }) + + It("writes a generated ConfigMap's edit to its env line and its delete to the entry, as Flux applies", func() { + By("seeding a folder whose ConfigMaps a configMapGenerator builds from menu.yaml and flags.env") + fixture := renderInPlaceFixtureFolder(configMapGeneratorFixtureRoot, testNs) + DeferCleanup(func() { _ = os.RemoveAll(fixture) }) + seedRenderedFolderIntoRepo(repo, testNs, fixture, configMapGeneratorGitPath) + + By("letting Flux build the folder and apply the generated ConfigMaps") + applyFluxApplier(testNs, repo, fluxName, fluxSecretName, "main", "./"+configMapGeneratorGitPath) + Eventually(func(g Gomega) { + g.Expect(generatedFlag(g, testNs, "ENABLE_VOUCHERS")).To(Equal("true")) + }, configMapGeneratorTimeout, 2*time.Second).Should(Succeed()) + uid := generatedFlagsUID(testNs) + + By("pointing a GitTarget at the folder: the generator is adopted, not refused") + createValidatedGitTarget(targetName, testNs, providerName, configMapGeneratorGitPath) + applyGeneratedConfigMapWatchRule(ruleName, testNs, targetName) + waitForGitTargetGitPathAccepted(targetName, testNs) + waitForStreamsRunning(targetName, testNs) + Eventually(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + }, configMapGeneratorTimeout, 2*time.Second).Should(Succeed()) + seeded := gitHead(repo.CheckoutDir) + + By("adopting the in-sync folder without a commit") + Consistently(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + g.Expect(gitHead(repo.CheckoutDir)).To(Equal(seeded), "an in-sync generator has nothing to write") + }, 15*time.Second, 3*time.Second).Should(Succeed()) + + By("turning the vouchers off through the API") + _, err := kubectlRunInNamespace(testNs, "patch", "configmap", "coffee-flags", "--type=merge", + "-p", `{"data":{"ENABLE_VOUCHERS":"false"}}`) + Expect(err).NotTo(HaveOccurred(), "failed to patch the generated ConfigMap") + + By("committing a one-line diff to flags.env, and no other file of the folder") + envPath := configMapGeneratorGitPath + "/flags.env" + var written string + Eventually(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + written = gitHead(repo.CheckoutDir) + g.Expect(written).NotTo(Equal(seeded)) + }, configMapGeneratorTimeout, 2*time.Second).Should(Succeed()) + changed, err := gitRun(repo.CheckoutDir, "show", "--format=", "--name-only", written) + Expect(err).NotTo(HaveOccurred()) + Expect(withoutBootstrapFiles(strings.Fields(changed))).To(Equal([]string{envPath}), + "the commit touches only the env file, beside the operator's first-write bootstrap files") + diff, err := gitRun(repo.CheckoutDir, "show", "--format=", "--unified=0", written, "--", envPath) + Expect(err).NotTo(HaveOccurred()) + Expect(diffLines(diff)).To(Equal([]string{"-ENABLE_VOUCHERS=true", "+ENABLE_VOUCHERS=false"})) + content, err := os.ReadFile(filepath.Join(repo.CheckoutDir, envPath)) + Expect(err).NotTo(HaveOccurred()) + Expect(string(content)).To(Equal( + "# Turn off during the conference rush.\nENABLE_VOUCHERS=false\nSHOW_BANNER=true\n"), + "the comment and the untouched line keep their bytes") + + By("letting Flux apply that commit: it renders the same object, under the same UID") + reconcileFluxKustomizationWithSource(fluxName) + Eventually(func(g Gomega) { + g.Expect(fluxLastAppliedRevision(g, fluxName)).To(ContainSubstring(written)) + }, configMapGeneratorTimeout, 2*time.Second).Should(Succeed()) + Expect(generatedFlag(Default, testNs, "ENABLE_VOUCHERS")).To(Equal("false")) + Expect(generatedFlagsUID(testNs)).To(Equal(uid), "the hash is disabled, so the object keeps its name") + + By("converging: Flux's apply is not mirrored back as another commit") + Consistently(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + g.Expect(gitHead(repo.CheckoutDir)).To(Equal(written)) + }, 20*time.Second, 4*time.Second).Should(Succeed()) + + By("deleting coffee-menu through the API: its entry and menu.yaml leave the folder") + _, err = kubectlRunInNamespace(testNs, "delete", "configmap", "coffee-menu") + Expect(err).NotTo(HaveOccurred(), "failed to delete the generated ConfigMap") + menuPath := filepath.Join(repo.CheckoutDir, configMapGeneratorGitPath, "menu.yaml") + var removed string + Eventually(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + removed = gitHead(repo.CheckoutDir) + g.Expect(menuPath).NotTo(BeAnExistingFile()) + }, configMapGeneratorTimeout, 2*time.Second).Should(Succeed()) + kustomizationPath := filepath.Join(repo.CheckoutDir, configMapGeneratorGitPath, "kustomization.yaml") + kustomization, err := os.ReadFile(kustomizationPath) + Expect(err).NotTo(HaveOccurred()) + Expect(string(kustomization)).NotTo(ContainSubstring("coffee-menu")) + Expect(string(kustomization)).To(ContainSubstring("coffee-flags"), "the other entry stays") + + By("letting Flux apply the removal: the next render does not bring the ConfigMap back") + reconcileFluxKustomizationWithSource(fluxName) + Eventually(func(g Gomega) { + g.Expect(fluxLastAppliedRevision(g, fluxName)).To(ContainSubstring(removed)) + }, configMapGeneratorTimeout, 2*time.Second).Should(Succeed()) + _, err = kubectlRunInNamespace(testNs, "get", "configmap", "coffee-menu") + Expect(err).To(HaveOccurred(), "coffee-menu must stay deleted") + Expect(generatedFlag(Default, testNs, "ENABLE_VOUCHERS")).To(Equal("false")) + }) +}) + +// applyGeneratedConfigMapWatchRule watches the ConfigMaps the fixture's generator labels, and no +// other ConfigMap of the namespace. +func applyGeneratedConfigMapWatchRule(name, namespace, target string) { + GinkgoHelper() + manifest := fmt.Sprintf(`apiVersion: configbutler.ai/v1alpha3 +kind: WatchRule +metadata: + name: %s + namespace: %s +spec: + gitTargetRef: + name: %s + rules: + - resources: ["configmaps"] + objectSelector: + matchLabels: + app.kubernetes.io/part-of: coffee-generator-e2e +`, name, namespace, target) + _, err := kubectlRunWithStdin(namespace, manifest, "apply", "-f", "-") + Expect(err).NotTo(HaveOccurred(), "failed to apply the WatchRule") + verifyResourceStatus("watchrule", name, namespace, "True", "Succeeded", "") +} + +// generatedFlag is one key of the live coffee-flags ConfigMap. +func generatedFlag(g Gomega, namespace, key string) string { + out, err := kubectlRunInNamespace(namespace, "get", "configmap", "coffee-flags", + "-o", "jsonpath={.data."+key+"}") + g.Expect(err).NotTo(HaveOccurred()) + return strings.TrimSpace(out) +} + +// generatedFlagsUID is the UID of the live coffee-flags ConfigMap. +func generatedFlagsUID(namespace string) string { + GinkgoHelper() + out, err := kubectlRunInNamespace(namespace, "get", "configmap", "coffee-flags", "-o", "jsonpath={.metadata.uid}") + Expect(err).NotTo(HaveOccurred()) + return strings.TrimSpace(out) +} + +// gitHead is the commit checked out in dir. +func gitHead(dir string) string { + GinkgoHelper() + out, err := gitRun(dir, "rev-parse", "HEAD") + Expect(err).NotTo(HaveOccurred()) + return strings.TrimSpace(out) +} + +// withoutBootstrapFiles drops the files the operator adds to a folder with its first commit there +// (internal/git/bootstrapped-repo-template), which the folder under test was seeded without. +func withoutBootstrapFiles(paths []string) []string { + var out []string + for _, p := range paths { + switch filepath.Base(p) { + case "README.md", ".gittargetignore", ".sops.yaml": + continue + } + out = append(out, p) + } + return out +} + +// diffLines is the removed and added lines of a unified diff, without its headers. +func diffLines(diff string) []string { + var out []string + for _, line := range strings.Split(diff, "\n") { + if strings.HasPrefix(line, "---") || strings.HasPrefix(line, "+++") { + continue + } + if strings.HasPrefix(line, "-") || strings.HasPrefix(line, "+") { + out = append(out, line) + } + } + return out +} + +// reconcileFluxKustomizationWithSource asks Flux to fetch the source and apply the Kustomization. +func reconcileFluxKustomizationWithSource(name string) { + GinkgoHelper() + args := []string{"reconcile", "kustomization", name, "-n", "flux-system", "--with-source", "--timeout", "90s"} + if context := strings.TrimSpace(kubectlContext()); context != "" { + args = append([]string{"--context", context}, args...) + } + output, err := exec.Command("flux", args...).CombinedOutput() + Expect(err).NotTo(HaveOccurred(), "flux %s: %s", strings.Join(args, " "), strings.TrimSpace(string(output))) +} diff --git a/test/e2e/fixtures/config-map-generator/flags.env b/test/e2e/fixtures/config-map-generator/flags.env new file mode 100644 index 000000000..31b88c7b6 --- /dev/null +++ b/test/e2e/fixtures/config-map-generator/flags.env @@ -0,0 +1,3 @@ +# Turn off during the conference rush. +ENABLE_VOUCHERS=true +SHOW_BANNER=true diff --git a/test/e2e/fixtures/config-map-generator/kustomization.yaml b/test/e2e/fixtures/config-map-generator/kustomization.yaml new file mode 100644 index 000000000..1e8a50215 --- /dev/null +++ b/test/e2e/fixtures/config-map-generator/kustomization.yaml @@ -0,0 +1,16 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: __E2E_NAMESPACE__ +configMapGenerator: + - name: coffee-menu + files: + - menu.yaml + - name: coffee-flags + envs: + - flags.env +generatorOptions: + disableNameSuffixHash: true + # The WatchRule selects on this label, so the namespace's other ConfigMaps + # (kube-root-ca.crt) are not mirrored into the folder. + labels: + app.kubernetes.io/part-of: coffee-generator-e2e diff --git a/test/e2e/fixtures/config-map-generator/menu.yaml b/test/e2e/fixtures/config-map-generator/menu.yaml new file mode 100644 index 000000000..c807e86b6 --- /dev/null +++ b/test/e2e/fixtures/config-map-generator/menu.yaml @@ -0,0 +1,6 @@ +# Prices in cents. Keep espresso the cheapest item. +products: + - sku: espresso + priceCents: 275 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/gitops-layouts/support-today.md b/test/fixtures/gitops-layouts/support-today.md index aa1bc42e2..e3b6111eb 100644 --- a/test/fixtures/gitops-layouts/support-today.md +++ b/test/fixtures/gitops-layouts/support-today.md @@ -35,12 +35,12 @@ Reading rules: | 2-rendered/helm-chart | 0 | All reported candidates accepted | 1 | 0 | plain=1 | - | None | | 2-rendered/helm-environment-values | 0 | All reported candidates accepted | 1 | 0 | plain=1 | - | None | | 2-rendered/kustomize-overlay-minimal | 0 | All reported candidates accepted | 2 | 0 | kustomize-overlay=2 | - | None | -| 2-rendered/kustomize-overlays | 0 | Partial | 1 | 3 | kustomize-single=1, refused-structural=3 | configMapGenerator, namePrefix, nameSuffix, remote-base, secretGenerator | refused-structural [not solvable]: kustomization uses unsupported feature(s): remote-base
refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator, nameSuffix, secretGenerator
refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator, namePrefix | +| 2-rendered/kustomize-overlays | 0 | Partial | 1 | 3 | kustomize-single=1, refused-structural=3 | configMapGenerator-name-hash, namePrefix, nameSuffix, remote-base, secretGenerator | refused-structural [not solvable]: kustomization uses unsupported feature(s): remote-base
refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator-name-hash, nameSuffix, secretGenerator (set disableNameSuffixHash: true in generatorOptions or on the entry)
refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator-name-hash, namePrefix (set disableNameSuffixHash: true in generatorOptions or on the entry) | | 2-rendered/rendered-manifests | 0 | Partial | 3 | 2 | plain=3, refused-structural=2 | namePrefix | refused-structural [not solvable]: kustomization uses unsupported feature(s): namePrefix
refused-structural [not solvable]: kustomization uses unsupported feature(s): namePrefix | | 3-expanded/argocd-applicationset-directories | 0 | All reported candidates accepted | 5 | 0 | plain=5 | - | None | | 3-expanded/argocd-applicationset-files | 0 | No reported candidates accepted | 0 | 1 | plain=1 | - | non-krm-yaml [solvable]: chart/Chart.yaml: YAML is not a Kubernetes manifest
foreign-file [solvable]: chart/templates/_helpers.tpl: foreign file chart/templates/_helpers.tpl is not a managed manifest; remove it or name it in .gittargetignore
non-krm-yaml [solvable]: chart/templates/deployment.yaml: YAML is not a Kubernetes manifest
non-krm-yaml [solvable]: chart/templates/service.yaml: YAML is not a Kubernetes manifest
+5 more | | 3-expanded/argocd-multicluster-matrix | 0 | All reported candidates accepted | 4 | 0 | plain=4 | - | None | -| 3-expanded/flux-helmrelease | 0 | Partial | 3 | 1 | kustomize-single=3, refused-structural=1 | configMapGenerator | refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator | +| 3-expanded/flux-helmrelease | 0 | All reported candidates accepted | 4 | 0 | kustomize-single=4 | - | None | | 3-expanded/flux-resourceset-inline | 0 | All reported candidates accepted | 1 | 0 | plain=1 | - | None | | 3-expanded/flux-resourceset-pull-requests | 0 | All reported candidates accepted | 1 | 0 | plain=1 | - | None | | 4-machine-written/flux-image-automation | 0 | All reported candidates accepted | 3 | 0 | kustomize-single=3 | - | None | @@ -152,7 +152,7 @@ Read graph: ## 2-rendered/kustomize-overlays Reported rc `0`. Accepted `1`, refused `3`. -Unsupported constructs: `configMapGenerator, namePrefix, nameSuffix, remote-base, secretGenerator`. +Unsupported constructs: `configMapGenerator-name-hash, namePrefix, nameSuffix, remote-base, secretGenerator`. Read graph: @@ -163,8 +163,8 @@ Read graph: |---|---|---|---|---|---| | `apps/backend/base` | `kustomize-single` | true | `-` | 2/2/0 | none | | `apps/backend/overlays/production` | `refused-structural` | false | `backend-production` | 0/0/0 | refused-structural [not solvable]: kustomization uses unsupported feature(s): remote-base | -| `apps/frontend/overlays/production` | `refused-structural` | false | `frontend-production` | 2/0/3 | refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator, nameSuffix, secretGenerator | -| `apps/frontend/overlays/staging` | `refused-structural` | false | `frontend-staging` | 2/0/1 | refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator, namePrefix | +| `apps/frontend/overlays/production` | `refused-structural` | false | `frontend-production` | 2/0/2 | refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator-name-hash, nameSuffix, secretGenerator (set disableNameSuffixHash: true in generatorOptions or on the entry) | +| `apps/frontend/overlays/staging` | `refused-structural` | false | `frontend-staging` | 2/0/0 | refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator-name-hash, namePrefix (set disableNameSuffixHash: true in generatorOptions or on the entry) | ## 2-rendered/rendered-manifests @@ -220,12 +220,12 @@ Unsupported constructs: `none`. ## 3-expanded/flux-helmrelease -Reported rc `0`. Accepted `3`, refused `1`. -Unsupported constructs: `configMapGenerator`. +Reported rc `0`. Accepted `4`, refused `0`. +Unsupported constructs: `none`. | Candidate | Layout | Accepted today | Namespace | rendered/editable/non-KRM | Refusal reasons | |---|---|---|---|---|---| -| `apps/frontend` | `refused-structural` | false | `-` | 1/1/1 | refused-structural [not solvable]: kustomization uses unsupported feature(s): configMapGenerator | +| `apps/frontend` | `kustomize-single` | true | `-` | 1/1/0 | none | | `clusters/production` | `kustomize-single` | true | `flux-system` | 7/7/0 | none | | `infrastructure/controllers/ingress-nginx` | `kustomize-single` | true | `flux-system` | 3/3/0 | none | | `infrastructure/sources` | `kustomize-single` | true | `flux-system` | 3/3/0 | none | diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/README.md b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/README.md new file mode 100644 index 000000000..2c76b6fb0 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/README.md @@ -0,0 +1,55 @@ +# 9: A ConfigMap a `configMapGenerator` renders + +> **This scenario exercises the write path, not placement.** No document holds the object: a +> `configMapGenerator` builds it from files beside the kustomization, and those files are where a +> live edit lands. + +Generators are how kustomize users keep configuration as files: a `menu.yaml` or a `flags.env` +next to the manifests, wrapped by a generator. With the name hash disabled the generated object +keeps its name, so the edit can be written into the input and the re-render oracle can check it. + +## Starting repository + +```text +apps/coffee/ # config/gittarget.yaml + kustomization.yaml # four generator entries, disableNameSuffixHash: true + deployment.yaml + menu.yaml # coffee-menu: files: [menu.yaml] + flags.env # coffee-flags: envs: [flags.env] + regions/eu.env, us.env # coffee-regions: two envs: files + branding/ # coffee-branding: files: (one text, one binary) + literals: +apps/hashed/ # config/gittarget-hashed.yaml: the name hash left on +apps/shop/ + base/ # a generator in the base + overlays/prod/ # config/gittarget-prod.yaml: reads the base + overlays/staging/ # config/gittarget-staging.yaml: behavior: merge over the base +``` + +## What a write does + +| Live change | Result | +|---|---| +| `menu.yaml` key changed | the file's bytes become the live value: [`expected-menu-price-changed.patch`](expected-menu-price-changed.patch) | +| `envs:` key changed | that line is edited in place: [`expected-flags-value-changed.patch`](expected-flags-value-changed.patch) | +| key added, one `envs:` file | `KEY=value` is appended: [`expected-flags-key-added.patch`](expected-flags-key-added.patch) | +| `envs:` key removed | the line is removed: [`expected-flags-key-removed.patch`](expected-flags-key-removed.patch) | +| `literals:` key changed | the item is edited in place: [`expected-branding-literal-changed.patch`](expected-branding-literal-changed.patch) | +| key added, `literals:` and no `envs:` file | appended to `literals:`: [`expected-branding-key-added.patch`](expected-branding-key-added.patch) | +| ConfigMap deleted | the entry and `flags.env` go: [`expected-flags-deleted.patch`](expected-flags-deleted.patch) | +| `files:` key removed | refused: [`expected-menu-key-removed-status.yaml`](expected-menu-key-removed-status.yaml) | +| key added, only `files:` sources | refused: [`expected-menu-key-added-status.yaml`](expected-menu-key-added-status.yaml) | +| key added, two `envs:` files | refused: [`expected-regions-key-added-status.yaml`](expected-regions-key-added-status.yaml) | +| `binaryData` changed | refused: [`expected-branding-binary-changed-status.yaml`](expected-branding-binary-changed-status.yaml) | +| label from the entry's `options:` changed | refused: [`expected-branding-label-changed-status.yaml`](expected-branding-label-changed-status.yaml) | +| name hash not disabled | folder refused, naming the setting: [`expected-hashed-status.yaml`](expected-hashed-status.yaml) | +| generator in a read-only base | refused: [`expected-base-owned-status.yaml`](expected-base-owned-status.yaml) | +| `behavior: merge` over a base's generator | folder refused: [`expected-behavior-merge-status.yaml`](expected-behavior-merge-status.yaml) | + +A changed key is edited on its own line or item, so comments, order and every untouched line keep +their bytes. A delete keeps an input file another entry, a resource or a patch also reads. Every +write goes through the re-render oracle: the folder is built again with the planned bytes, and the +commit happens only when the generator then renders the live object. + +A refusal has the same cause each time: the change has no single input file to land in. Writing a +ConfigMap document beside the generator instead would collide with the generated object in the +build, so nothing is written. diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-hashed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-hashed.yaml new file mode 100644 index 000000000..4e1426413 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-hashed.yaml @@ -0,0 +1,25 @@ +apiVersion: configbutler.ai/v1alpha3 +kind: GitBranch +metadata: + name: app-repository-main + namespace: coffee +spec: + gitProviderRef: + name: app-repository + branch: main + allowedGitTargets: + - consumerNamespaces: {from: Same} +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitTarget +metadata: + name: hashed-menu + namespace: coffee +spec: + gitBranchRef: + name: app-repository-main + kubeProviderRef: + name: local + namespace: gitops-reverser + path: apps/hashed + # The same generator with the name hash left on, which is refused. diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-prod.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-prod.yaml new file mode 100644 index 000000000..370371203 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-prod.yaml @@ -0,0 +1,25 @@ +apiVersion: configbutler.ai/v1alpha3 +kind: GitBranch +metadata: + name: app-repository-main + namespace: shop-prod +spec: + gitProviderRef: + name: app-repository + branch: main + allowedGitTargets: + - consumerNamespaces: {from: Same} +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitTarget +metadata: + name: shop + namespace: shop-prod +spec: + gitBranchRef: + name: app-repository-main + kubeProviderRef: + name: local + namespace: gitops-reverser + path: apps/shop/overlays/prod + # The generator lives in ../../base, which this target reads and never writes. diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-staging.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-staging.yaml new file mode 100644 index 000000000..8c50b3841 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget-staging.yaml @@ -0,0 +1,25 @@ +apiVersion: configbutler.ai/v1alpha3 +kind: GitBranch +metadata: + name: app-repository-main + namespace: shop-staging +spec: + gitProviderRef: + name: app-repository + branch: main + allowedGitTargets: + - consumerNamespaces: {from: Same} +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitTarget +metadata: + name: shop + namespace: shop-staging +spec: + gitBranchRef: + name: app-repository-main + kubeProviderRef: + name: local + namespace: gitops-reverser + path: apps/shop/overlays/staging + # The overlay merges into the base's generator with behavior: merge. diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget.yaml new file mode 100644 index 000000000..829782814 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/config/gittarget.yaml @@ -0,0 +1,26 @@ +apiVersion: configbutler.ai/v1alpha3 +kind: GitBranch +metadata: + name: app-repository-main + namespace: coffee +spec: + gitProviderRef: + name: app-repository + branch: main + allowedGitTargets: + - consumerNamespaces: {from: Same} +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitTarget +metadata: + name: coffee + namespace: coffee +spec: + gitBranchRef: + name: app-repository-main + kubeProviderRef: + name: local + namespace: gitops-reverser + path: apps/coffee + # Nothing is declared: a configMapGenerator with the name hash disabled is + # adopted as it is, and its input files are where a ConfigMap edit lands. diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-base-owned-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-base-owned-status.yaml new file mode 100644 index 000000000..b7b467b0c --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-base-owned-status.yaml @@ -0,0 +1,13 @@ +# The generator lives in ../../base, which the overlay reads and never writes: base-owned, +# the same rule as for a base document. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: WriteBoundaryRefused + message: >- + ConfigMap/shop-menu generated in base/kustomization.yaml: no destination for the change: + the generator lives in base/kustomization.yaml, which this target reads and never writes diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-behavior-merge-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-behavior-merge-status.yaml new file mode 100644 index 000000000..5af81ea57 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-behavior-merge-status.yaml @@ -0,0 +1,14 @@ +# behavior: merge rewrites the ConfigMap the base generates, so the object is shared with +# the base and no one file owns it. Refused whatever the hash setting. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: UnsupportedContent + message: >- + overlays/staging/kustomization.yaml: kustomization uses unsupported feature(s): + configMapGenerator-behavior; the operator will not write into this folder (a + kustomize-build-failed diagnostic on this path carries the build error) diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-binary-changed-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-binary-changed-status.yaml new file mode 100644 index 000000000..f7a0c1046 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-binary-changed-status.yaml @@ -0,0 +1,13 @@ +# The logo is not UTF-8, so kustomize puts it in binaryData. The operator writes no binary +# input file. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: WriteBoundaryRefused + message: >- + ConfigMap/coffee-branding generated in kustomization.yaml: no destination for the + change: binaryData changed, and the operator writes no binary input file diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-key-added.patch b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-key-added.patch new file mode 100644 index 000000000..63993348d --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-key-added.patch @@ -0,0 +1,11 @@ +diff --git a/apps/coffee/kustomization.yaml b/apps/coffee/kustomization.yaml +--- a/apps/coffee/kustomization.yaml ++++ b/apps/coffee/kustomization.yaml +@@ -24,6 +24,7 @@ - tagline=branding/tagline.txt + - logo=branding/logo.bin + literals: + - THEME=dark ++ - SLOGAN=Coffee first. + options: + labels: + team: baristas diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-label-changed-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-label-changed-status.yaml new file mode 100644 index 000000000..d2e756563 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-label-changed-status.yaml @@ -0,0 +1,14 @@ +# The team label comes from the entry's options:. A generated ConfigMap's metadata is the +# kustomization's, and no input file holds it. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: WriteBoundaryRefused + message: >- + ConfigMap/coffee-branding generated in kustomization.yaml: no destination for the + change: metadata.labels changed, and a generated ConfigMap takes it from the + kustomization, not from an input file diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-literal-changed.patch b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-literal-changed.patch new file mode 100644 index 000000000..1b9cc46f5 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-branding-literal-changed.patch @@ -0,0 +1,12 @@ +diff --git a/apps/coffee/kustomization.yaml b/apps/coffee/kustomization.yaml +--- a/apps/coffee/kustomization.yaml ++++ b/apps/coffee/kustomization.yaml +@@ -23,7 +23,7 @@ files: + - tagline=branding/tagline.txt + - logo=branding/logo.bin + literals: +- - THEME=dark ++ - THEME=light + options: + labels: + team: baristas diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-deleted.patch b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-deleted.patch new file mode 100644 index 000000000..e6bb754ef --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-deleted.patch @@ -0,0 +1,22 @@ +diff --git a/apps/coffee/flags.env b/apps/coffee/flags.env +deleted file mode 100644 +--- a/apps/coffee/flags.env ++++ /dev/null +@@ -1,3 +0,0 @@ +-# Turn off during the conference rush. +-ENABLE_VOUCHERS=true +-SHOW_BANNER=true +diff --git a/apps/coffee/kustomization.yaml b/apps/coffee/kustomization.yaml +--- a/apps/coffee/kustomization.yaml ++++ b/apps/coffee/kustomization.yaml +@@ -8,10 +8,6 @@ # The settings file people care about: the whole file is one data key. + - name: coffee-menu + files: + - menu.yaml +- # One env file: every line is a data key. +- - name: coffee-flags +- envs: +- - flags.env + # Two env files: a new key has no single file to go to. + - name: coffee-regions + envs: diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-key-added.patch b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-key-added.patch new file mode 100644 index 000000000..1e1adc485 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-key-added.patch @@ -0,0 +1,8 @@ +diff --git a/apps/coffee/flags.env b/apps/coffee/flags.env +--- a/apps/coffee/flags.env ++++ b/apps/coffee/flags.env +@@ -1,3 +1,4 @@ + # Turn off during the conference rush. + ENABLE_VOUCHERS=true + SHOW_BANNER=true ++SHOW_TIPS=true diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-key-removed.patch b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-key-removed.patch new file mode 100644 index 000000000..3b1926f68 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-key-removed.patch @@ -0,0 +1,7 @@ +diff --git a/apps/coffee/flags.env b/apps/coffee/flags.env +--- a/apps/coffee/flags.env ++++ b/apps/coffee/flags.env +@@ -1,3 +1,2 @@ + # Turn off during the conference rush. + ENABLE_VOUCHERS=true +-SHOW_BANNER=true diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-value-changed.patch b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-value-changed.patch new file mode 100644 index 000000000..09c3b0745 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-flags-value-changed.patch @@ -0,0 +1,8 @@ +diff --git a/apps/coffee/flags.env b/apps/coffee/flags.env +--- a/apps/coffee/flags.env ++++ b/apps/coffee/flags.env +@@ -1,3 +1,3 @@ + # Turn off during the conference rush. +-ENABLE_VOUCHERS=true ++ENABLE_VOUCHERS=false + SHOW_BANNER=true diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-hashed-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-hashed-status.yaml new file mode 100644 index 000000000..17e29550d --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-hashed-status.yaml @@ -0,0 +1,16 @@ +# The name hash couples the name to the content: the object someone edits is replaced by a +# differently named one on the next render. The folder is refused, and the message names the +# setting that makes it adoptable. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: UnsupportedContent + message: >- + kustomization.yaml: kustomization uses a setting that can be changed in the repository: + configMapGenerator-name-hash (set disableNameSuffixHash: true in generatorOptions or on + the entry); the operator will not write into this folder (a kustomize-build-failed + diagnostic on this path carries the build error) diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-key-added-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-key-added-status.yaml new file mode 100644 index 000000000..7aa2e96a8 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-key-added-status.yaml @@ -0,0 +1,13 @@ +# A new key, and the entry reads only files:. A files: source holds exactly one key, the whole +# file, so there is no input a second key could go to. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: WriteBoundaryRefused + message: >- + ConfigMap/coffee-menu generated in kustomization.yaml: no destination for the change: key + "currency" is new, and the entry has no envs: file or literals: to add it to diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-key-removed-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-key-removed-status.yaml new file mode 100644 index 000000000..7c2c85e14 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-key-removed-status.yaml @@ -0,0 +1,13 @@ +# The menu.yaml key's value is the whole of menu.yaml, so removing the key has no line to +# remove. Deleting the file and its files: entry is a change the operator leaves to a person. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: WriteBoundaryRefused + message: >- + ConfigMap/coffee-menu generated in kustomization.yaml: no destination for the change: + key "menu.yaml" was removed, and its value is the whole of menu.yaml diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-price-changed.patch b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-price-changed.patch new file mode 100644 index 000000000..cdf2f5aa5 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-menu-price-changed.patch @@ -0,0 +1,11 @@ +diff --git a/apps/coffee/menu.yaml b/apps/coffee/menu.yaml +--- a/apps/coffee/menu.yaml ++++ b/apps/coffee/menu.yaml +@@ -1,6 +1,6 @@ + # Prices in cents. Keep espresso the cheapest item. + products: + - sku: espresso +- priceCents: 275 ++ priceCents: 300 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-regions-key-added-status.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-regions-key-added-status.yaml new file mode 100644 index 000000000..edb2d4ae9 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/expected-regions-key-added-status.yaml @@ -0,0 +1,13 @@ +# A new key, and two envs: files it could go to. Picking one would decide which region the +# key belongs to, which nothing in the live object says. +# +# No patch: the flush is refused before a byte moves. GitPathAccepted's message is the writer's +# own and is asserted by the corpus. +status: + conditions: + - type: GitPathAccepted + status: "False" + reason: WriteBoundaryRefused + message: >- + ConfigMap/coffee-regions generated in kustomization.yaml: no destination for the change: + key "CLOSING" is new, and the entry has 2 envs: files, so no single one to add it to diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-binary-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-binary-changed.yaml new file mode 100644 index 000000000..07bba7421 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-binary-changed.yaml @@ -0,0 +1,16 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# A new logo, which lands in binaryData. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-branding + namespace: coffee + labels: + team: baristas +data: + tagline: | + Fresh every morning. + THEME: dark +binaryData: + logo: R0lGODlh diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-key-added.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-key-added.yaml new file mode 100644 index 000000000..c89c1d4db --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-key-added.yaml @@ -0,0 +1,17 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# A new key, and only files: and literals: to put it in. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-branding + namespace: coffee + labels: + team: baristas +data: + tagline: | + Fresh every morning. + THEME: dark + SLOGAN: Coffee first. +binaryData: + logo: iVBORw0KGgo= diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-label-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-label-changed.yaml new file mode 100644 index 000000000..c9d7e9c1f --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-label-changed.yaml @@ -0,0 +1,16 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# The team label comes from the entry's options:, not from an input file. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-branding + namespace: coffee + labels: + team: roasters +data: + tagline: | + Fresh every morning. + THEME: dark +binaryData: + logo: iVBORw0KGgo= diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-literal-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-literal-changed.yaml new file mode 100644 index 000000000..2bee4d684 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/branding-literal-changed.yaml @@ -0,0 +1,16 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# THEME comes from a literals: entry in kustomization.yaml. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-branding + namespace: coffee + labels: + team: baristas +data: + tagline: | + Fresh every morning. + THEME: light +binaryData: + logo: iVBORw0KGgo= diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-deleted.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-deleted.yaml new file mode 100644 index 000000000..7bd1ebf9a --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-deleted.yaml @@ -0,0 +1,9 @@ +# Abridged live object, as last seen: kubectl delete configmap coffee-flags -n coffee. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-flags + namespace: coffee +data: + ENABLE_VOUCHERS: "true" + SHOW_BANNER: "true" diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-key-added.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-key-added.yaml new file mode 100644 index 000000000..cdd48125c --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-key-added.yaml @@ -0,0 +1,12 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# A new key, SHOW_TIPS. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-flags + namespace: coffee +data: + ENABLE_VOUCHERS: "true" + SHOW_BANNER: "true" + SHOW_TIPS: "true" diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-key-removed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-key-removed.yaml new file mode 100644 index 000000000..0c024e60a --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-key-removed.yaml @@ -0,0 +1,10 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# SHOW_BANNER is gone. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-flags + namespace: coffee +data: + ENABLE_VOUCHERS: "true" diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-value-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-value-changed.yaml new file mode 100644 index 000000000..ec8bfa910 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/flags-value-changed.yaml @@ -0,0 +1,11 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# kubectl patch configmap coffee-flags -p '{"data":{"ENABLE_VOUCHERS":"false"}}' +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-flags + namespace: coffee +data: + ENABLE_VOUCHERS: "false" + SHOW_BANNER: "true" diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/hashed-menu-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/hashed-menu-changed.yaml new file mode 100644 index 000000000..53c5408f6 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/hashed-menu-changed.yaml @@ -0,0 +1,11 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# Any edit: the folder itself is refused. +apiVersion: v1 +kind: ConfigMap +metadata: + name: hashed-menu-7b8f9g2k4c + namespace: coffee +data: + menu.yaml: | + products: [] diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-key-added.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-key-added.yaml new file mode 100644 index 000000000..03cbf974d --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-key-added.yaml @@ -0,0 +1,17 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# A new key on an entry that reads only files:. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-menu + namespace: coffee +data: + menu.yaml: | + # Prices in cents. Keep espresso the cheapest item. + products: + - sku: espresso + priceCents: 275 + - sku: flat-white + priceCents: 395 + currency: EUR diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-key-removed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-key-removed.yaml new file mode 100644 index 000000000..ce96edb04 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-key-removed.yaml @@ -0,0 +1,9 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# The menu.yaml key is gone, and with it the whole file's content. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-menu + namespace: coffee +data: {} diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-price-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-price-changed.yaml new file mode 100644 index 000000000..d59de9ce8 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/menu-price-changed.yaml @@ -0,0 +1,16 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# The espresso is now 300 cents. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-menu + namespace: coffee +data: + menu.yaml: | + # Prices in cents. Keep espresso the cheapest item. + products: + - sku: espresso + priceCents: 300 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/regions-key-added.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/regions-key-added.yaml new file mode 100644 index 000000000..e6783df62 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/regions-key-added.yaml @@ -0,0 +1,13 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# A new key, and two env files it could go to. +apiVersion: v1 +kind: ConfigMap +metadata: + name: coffee-regions + namespace: coffee +data: + OPENING: "08:00" + OPENING_US: "07:00" + TIPPING: "true" + CLOSING: "18:00" diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/shop-menu-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/shop-menu-changed.yaml new file mode 100644 index 000000000..a5aebab49 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/shop-menu-changed.yaml @@ -0,0 +1,16 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# The espresso is now 300 cents in production. +apiVersion: v1 +kind: ConfigMap +metadata: + name: shop-menu + namespace: shop-prod +data: + menu.yaml: | + # Prices in cents. Keep espresso the cheapest item. + products: + - sku: espresso + priceCents: 300 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/shop-staging-menu-changed.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/shop-staging-menu-changed.yaml new file mode 100644 index 000000000..02e9381a4 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/input/shop-staging-menu-changed.yaml @@ -0,0 +1,16 @@ +# Abridged live object: the captured original also carries uid, resourceVersion, +# creationTimestamp and managedFields, none of which reach Git. +# The espresso is now 300 cents in staging. +apiVersion: v1 +kind: ConfigMap +metadata: + name: shop-menu + namespace: shop-staging +data: + menu.yaml: | + # Prices in cents. Keep espresso the cheapest item. + products: + - sku: espresso + priceCents: 300 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/branding/logo.bin b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/branding/logo.bin new file mode 100644 index 000000000..0dd1608e4 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/branding/logo.bin @@ -0,0 +1,2 @@ +‰PNG + diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/branding/tagline.txt b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/branding/tagline.txt new file mode 100644 index 000000000..2dcb51f21 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/branding/tagline.txt @@ -0,0 +1 @@ +Fresh every morning. diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/deployment.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/deployment.yaml new file mode 100644 index 000000000..2509a4c33 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/deployment.yaml @@ -0,0 +1,20 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: coffee +spec: + replicas: 1 + selector: + matchLabels: + app: coffee + template: + metadata: + labels: + app: coffee + spec: + containers: + - name: coffee + image: ghcr.io/example/coffee:1.0.0 + envFrom: + - configMapRef: + name: coffee-flags diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/flags.env b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/flags.env new file mode 100644 index 000000000..31b88c7b6 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/flags.env @@ -0,0 +1,3 @@ +# Turn off during the conference rush. +ENABLE_VOUCHERS=true +SHOW_BANNER=true diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/kustomization.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/kustomization.yaml new file mode 100644 index 000000000..604159876 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/kustomization.yaml @@ -0,0 +1,31 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: coffee +resources: + - deployment.yaml +configMapGenerator: + # The settings file people care about: the whole file is one data key. + - name: coffee-menu + files: + - menu.yaml + # One env file: every line is a data key. + - name: coffee-flags + envs: + - flags.env + # Two env files: a new key has no single file to go to. + - name: coffee-regions + envs: + - regions/eu.env + - regions/us.env + # A text file, a binary file and a literal: a new key joins the literals. + - name: coffee-branding + files: + - tagline=branding/tagline.txt + - logo=branding/logo.bin + literals: + - THEME=dark + options: + labels: + team: baristas +generatorOptions: + disableNameSuffixHash: true diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/menu.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/menu.yaml new file mode 100644 index 000000000..c807e86b6 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/menu.yaml @@ -0,0 +1,6 @@ +# Prices in cents. Keep espresso the cheapest item. +products: + - sku: espresso + priceCents: 275 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/regions/eu.env b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/regions/eu.env new file mode 100644 index 000000000..f3a23c06f --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/regions/eu.env @@ -0,0 +1 @@ +OPENING=08:00 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/regions/us.env b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/regions/us.env new file mode 100644 index 000000000..15c20c3cc --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/coffee/regions/us.env @@ -0,0 +1,2 @@ +OPENING_US=07:00 +TIPPING=true diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/hashed/kustomization.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/hashed/kustomization.yaml new file mode 100644 index 000000000..be27432da --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/hashed/kustomization.yaml @@ -0,0 +1,9 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: coffee +# No disableNameSuffixHash: the generated name carries a content hash, so the +# object someone edits is replaced by a differently named one on the next render. +configMapGenerator: + - name: hashed-menu + files: + - menu.yaml diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/hashed/menu.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/hashed/menu.yaml new file mode 100644 index 000000000..c807e86b6 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/hashed/menu.yaml @@ -0,0 +1,6 @@ +# Prices in cents. Keep espresso the cheapest item. +products: + - sku: espresso + priceCents: 275 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/base/kustomization.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/base/kustomization.yaml new file mode 100644 index 000000000..9fae2f6c5 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/base/kustomization.yaml @@ -0,0 +1,8 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +configMapGenerator: + - name: shop-menu + files: + - menu.yaml +generatorOptions: + disableNameSuffixHash: true diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/base/menu.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/base/menu.yaml new file mode 100644 index 000000000..c807e86b6 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/base/menu.yaml @@ -0,0 +1,6 @@ +# Prices in cents. Keep espresso the cheapest item. +products: + - sku: espresso + priceCents: 275 + - sku: flat-white + priceCents: 395 diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/overlays/prod/kustomization.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/overlays/prod/kustomization.yaml new file mode 100644 index 000000000..b15f4d846 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/overlays/prod/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: shop-prod +resources: + - ../../base diff --git a/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/overlays/staging/kustomization.yaml b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/overlays/staging/kustomization.yaml new file mode 100644 index 000000000..a6d947ea9 --- /dev/null +++ b/test/fixtures/layout-corpus/shapes/9-kustomize-config-map-generator/repository/apps/shop/overlays/staging/kustomization.yaml @@ -0,0 +1,11 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: shop-staging +resources: + - ../../base +# Rewrites the ConfigMap the base generates: the object is shared with the base. +configMapGenerator: + - name: shop-menu + behavior: merge + literals: + - STAGE=staging diff --git a/test/fixtures/layout-corpus/shapes/README.md b/test/fixtures/layout-corpus/shapes/README.md index f59212b51..c92b6725c 100644 --- a/test/fixtures/layout-corpus/shapes/README.md +++ b/test/fixtures/layout-corpus/shapes/README.md @@ -30,6 +30,10 @@ a `ConfigMap` named `checkout-config` in namespace `shop`. Each folder holds `co | [6](6-kustomize-base-and-overlays/README.md) | Base + three environment overlays | — | unset | none | | [7](7-kustomize-layered/README.md) | Layered kustomize | — | unset | none | | [8](8-base-owned-field-edit/README.md) | Editing a field the **base** owns | — | unset | none — **write path, not placement** | +| [9](9-kustomize-config-map-generator/README.md) | A ConfigMap a `configMapGenerator` renders | — | unset | none — **write path, not placement** | + +Scenario 9 is not a folder shape either: it is the write path for an object no document holds, +because a generator builds it from the files beside its kustomization. Scenario 8 is not an eighth folder shape: it is shape 6's repository with the question changed from *where does a new file go* to *what happens when the changed field belongs to the base*. It sits in