Repository navigation
feat(cluster): admit and apply project Kubernetes manifests - #1479
Conversation
First step toward oblien#1039: the engine for deploying user-supplied Kubernetes manifests into a k3s cluster project, with no UI or deploy wiring yet. - parseProjectManifests: multi-document YAML, size and object-count limits, errors that name the failing document. - admitProjectManifests: an allowlist of namespaced kinds (ConfigMap, Secret, ClusterIP Service, Deployment, Job, CronJob). Refuses cluster-scoped kinds, other namespaces, ownerReferences, finalizers, reserved labels, NodePort, host networking/PID/IPC, hostPath, PVCs, privileged containers, hostPort and non-default service accounts. Pins objects to the project namespace, stamps ownership labels and a content hash, and disables token automount. - applyProjectManifests: create, skip unchanged, replace changed objects it owns (keeping a Service's cluster IP), and prune manifest objects that are no longer declared. Openship's own app objects are never replaced. - Project deletion accepts owned ConfigMap, Job and CronJob objects. Refs oblien#1039
Hydralerne
left a comment
There was a problem hiding this comment.
This is a useful foundation for #1039, and the PR clearly states that deployment/UI wiring is deferred. It should remain open, but the admission and reconciliation boundary needs two fixes before merge:
- Reserved ownership keys and finalizers are only checked on the outer object. I ran
admitProjectManifestswith a Deployment whosespec.template.metadata.labelscontainsopenship.io/deployment: existing-releaseandapp.kubernetes.io/managed-by: openship, with the matching reserved selector and a custom pod finalizer. It is accepted unchanged. Existing runtime Services and pod lookups selectopenship.io/deployment, so these pods can enter an existing managed workload's selector; the finalizer can also prevent cleanup. Validate pod metadata/selectors, and both Job and pod template metadata inside CronJobs. Enforce the ownership boundary at every controller-generated object, not only the parent. - The fake API accepts updates Kubernetes rejects.
applyProjectManifestsblindly PUTs every changed owned object. Job pod templates and Deployment selectors are immutable, so the advertised update path can fail after earlier objects were already changed. Define explicit per-kind behavior for immutable changes and test it against a real API server. At minimum, preflight unsupported changes before making mutations; do not silently delete/recreate workloads to get around immutability.
The first finding is reproduced against this branch's actual admission function. No user endpoint exposes it yet, so this is a pre-merge blocker, not evidence of a live tenant compromise. Please add negative admission cases and a real k3s apply/update/prune test covering these boundaries.
Hydralerne
left a comment
There was a problem hiding this comment.
Applied the ownership and reconciliation fixes directly to this branch and verified this exact commit. Manifest changes now preflight all API writes and pruning before mutation; updates and deletes carry concurrency preconditions, and reserved OpenShip ownership metadata is rejected throughout workload templates. Changed Jobs require a new name. This remains a manifest foundation, not a cross-resource transaction or a deployed upload flow.
Validation: 71 unit/runtime checks plus 3 real K3s tests passed locally, including immutable-update failure before earlier resources change and owned-only pruning. All 21 CI checks pass, including the new Kubernetes manifests job. Temporary test infrastructure was removed.
Add the admission and reconciliation foundation for project-scoped Kubernetes manifests (refs #1039). This PR does not expose a manifest upload UI, HTTP route, Helm support or deployment-pipeline integration.
The allowlist supports ConfigMaps, Secrets, ClusterIP Services, Deployments, Jobs and CronJobs inside the project's namespace. Admission rejects reserved ownership labels/annotations/selectors, owner references and finalizers at both outer objects and generated pod/job templates. It also retains the existing restrictions on host access, privileged containers, non-default service accounts, persistent claims and externally exposed Services. Project deletion recognizes the additional owned resource kinds. Manifest Deployments stay out of the native release/orphan inventory, so arbitrary manifest names cannot be mistaken for native release references or block namespace cleanup. Inventory still verifies project and runtime ownership before excluding them; namespace cleanup retains its full ownership and persistent-storage checks.
Reconciliation reads ownership first, then asks the real API server to dry-run the complete requested change before committing any writes. Existing Services retain assigned IP defaults. Jobs are explicitly create-only: a changed definition requires a new Job name. Other immutable or invalid changes fail server validation without deleting/recreating a workload. Updates retain resource versions; pruning requires the inspected UID and resource version and runs only after successful creates/updates. DELETE dry-runs include
dryRunin the DeleteOptions body, as required when sending a body.This is not a transaction across Kubernetes resources. Concurrent edits or transport failures during commit can still interrupt the operation; stale writes fail rather than overwriting changes, and no pruning follows a failed create/update.
Validation:
The three real K3s checks passed in CI on the previous head. They were not repeated locally during this update because the Docker daemon is unavailable; the new head reruns the same CI suite.