Add configurable pod and container securityContext, including readOnlyRootFilesystem support

[mouchar]

Motivation

The chart cannot currently be deployed on a cluster that enforces a restrictive pod security policy (Pod Security Admission, OPA Gatekeeper, Kyverno):

• A pod-level securityContext is exposed for only zookeeper, bookkeeper, broker and oxia.server.
• There is no container-level securityContext anywhere, so allowPrivilegeEscalation, capabilities, seccompProfile and readOnlyRootFilesystem are not settable at any value.
• The autorecovery StatefulSet renders no securityContext and has no probes.
• readOnlyRootFilesystem cannot work unaided, because the Pulsar images rewrite /pulsar/conf on startup and log to /pulsar/logs.

Modifications

Two commits, reviewable independently.

1. securityContext cascade. Adds global podSecurityContext and containerSecurityContext, each merged with a matching per-component override that wins per key:

podSecurityContext        <- <component>.securityContext
containerSecurityContext  <- <component>.containerSecurityContext

Applied to all 18 pod templates, including initContainers and the init/cleanup Jobs. The merge uses mergeOverwrite rather than merge, because merge treats zero values in its destination as absent and would silently discard a per-component fsGroup: 0 or allowPrivilegeEscalation: false.

2. readOnlyRootFilesystem support. When the effective container securityContext sets it, the chart mounts an emptyDir over /pulsar/conf, /pulsar/logs and /tmp and prepends a copy-pulsar-conf initContainer that seeds the conf volume from the image, for the components that run a Pulsar image. writableRootfsVolumes: false opts out in favour of your own extraVolumes/extraVolumeMounts.

This could not be done from values: values-supplied initContainers are appended after the built-in ones so a seeding container cannot run first, and only autorecovery exposes initContainersExtraVolumeMounts, so broker/wait-bookkeeper-ready — which runs apply-config-from-env.py — would fail.

Also adds examples/values-restricted-psp.yaml as a worked example, including the per-component overrides needed to move the four components shipping fsGroup: 0 off GID 0.

Rendered output with default values is unchanged except for three deliberate items:

• The autorecovery StatefulSet gains liveness/readiness probes. It had none and no knob to add them. The daemon does not start BookKeeper's HTTP service, so they target the Prometheus stats endpoint on autorecovery.ports.http — the endpoint the PodMonitor already scrapes.
• The zookeeper and broker sts-cleanup upgrade-hook Jobs gain pod template labels via pulsar.template.labels. They were the only pod templates without them, so .Values.labels never reached their pods.
• Three Jobs now render the fsGroup: 0 of the component they belong to. Those pods mount only ConfigMaps, Secrets and the service account token, so this has no functional effect.

Verifying this change

• Make sure that the change passes the CI checks.

helm lint clean. kubeconform against k8s 1.25 / 1.31 / 1.36 for the default values, every .ci/clusters/* config and .ci/templates-all-values.yaml, with the new feature both off and on.

Rendered output diffed against master across all of the above: identical apart from the three items listed, and byte-identical for commit 2 when readOnlyRootFilesystem is unset.

Runtime-tested on EKS 1.34 with the AWS EBS CSI driver (gp3/ext4, fsGroupPolicy: ReadWriteOnceWithFSType), upgrading in place from published 4.7.0:

• Moving fsGroup from 0 to 10000 triggers the recursive relabel and preserves data: a 5000-message unacked backlog was consumed intact afterwards, with identical storageSize and no pod restarts.
• With readOnlyRootFilesystem: true, all pods reach Ready, copy-pulsar-conf and the built-in init containers complete, apply-config-from-env.py writes to the seeded conf volume, and produce/consume keeps working with no read-only filesystem errors.

Also exercised on a 4-node kind cluster (k8s 1.36.1) with the default values, with the restricted-PSP example applied as an upgrade, and with Oxia as the metadata store.

[lhotari]

Thanks for this — it's careful work and the parts that are right are very right. I verified your central compatibility claim and it holds exactly: with default values the rendered output differs from master in precisely the three items you list, 43 diff lines total. With oxia and pulsar_manager enabled it's byte-identical. kubeconform 1.31 strict passes for both the default render and values-restricted-psp.yaml. The mergeOverwrite + deepCopy choice is correct, and the reasoning in the helper comment about merge discarding zero values is right.

Commit 1 (the securityContext cascade) looks close to mergeable on its own. The issues are in commit 2's edge cases and in the docs.

Blocking

Four issues, all inline: a --reuse-values render failure, and three cases where readOnlyRootFilesystem produces a pod that can't start. The first one is the most urgent since it breaks upgrades for everyone who uses that flag, regardless of whether they touch the new feature.

The fsGroup / GID 0 framing — please reconsider

This is the part I'd most like changed, because I think the example teaches the wrong lesson, and it's the reason fsGroup: 0 is the chart default in the first place.

fsGroup: 0 is not a privilege. Group 0 inside a container is an ordinary group; root power comes from UID 0 and capabilities. It does grant DAC access to group-0-accessible files — which is exactly the mechanism that makes the Pulsar image work under an arbitrary UID — but it is not "running as root".

Concretely: the Kubernetes restricted Pod Security Standard places no constraint on fsGroup, fsGroupChangePolicy or supplementalGroups at all. Its controls are runAsNonRoot, runAsUser != 0, allowPrivilegeEscalation, capabilities, seccompProfile, and the host/volume rules. The PSS-restricted policy sets that Gatekeeper and Kyverno ship mirror PSA and likewise don't restrict fsGroup — both engines can via separately installed policy, but nothing does out of the box.

So the four fsGroup: 10000 overrides in examples/values-restricted-psp.yaml aren't needed for the file's stated purpose, and they're the single most operationally dangerous lines in it — a recursive chown of every bookie ledger volume, which your own warning comment at the top of the file describes.

The background, from Red Hat's UID guide: "the user in the Container always has GID=0, which is the root group", and Red Hat's recommendation is that writable files "should be owned by the root group and be read/writable by GID=0". That's why the chart defaults to fsGroup: 0.

Worth knowing though — I went and checked, and on a stock OpenShift project neither value is admissible. From OpenShift's shipped manifest (openshift/cluster-kube-apiserver-operator, bindata/bootkube/scc-manifests/…_00_scc-restricted-v2.yaml), restricted-v2 is:

runAsUser:          {type: MustRunAsRange}
fsGroup:            {type: MustRunAs}
supplementalGroups: {type: RunAsAny}
allowPrivilegeEscalation: false
requiredDropCapabilities: [ALL]
seccompProfiles: [runtime/default]

and the MustRunAs implementation (openshift/apiserver-library-go, pkg/securitycontextconstraints/group/mustrunas.go) validates any pod-supplied group against the namespace's preallocated range, rejecting with "%d is not an allowed group". On a typical range (~1000620000/10000): fsGroup: 10000 rejected, fsGroup: 0 rejected, runAsUser: 10000 rejected. Omitting them lets admission inject correct values. (fsGroup: 0 works on OpenShift today under anyuid or a custom SCC, which use fsGroup: RunAsAny.)

Nice result worth calling out: your containerSecurityContext block matches restricted-v2 field for field. That half of the example is exactly right.

Suggested changes:

• Keep the component fsGroup: 0 defaults as they are.
• Keep the example's containerSecurityContext block verbatim.
• Drop runAsUser/runAsGroup/fsGroup/supplementalGroups from the example's podSecurityContext, keeping runAsNonRoot: true, and delete the four per-component fsGroup: 10000 blocks.
• Retitle the README's "Moving off GID 0" — it reads as a hardening step, which it isn't — and note that PSA restricted doesn't constrain fsGroup.
• If you want an OpenShift example, it needs to be a separate file that clears the defaults (zookeeper: {securityContext: null} etc.), because simply removing the overrides restores fsGroup: 0, which restricted-v2 also rejects.

Non-blocking

• User-supplied <component>.initContainers are raw toYaml passthrough and never merged with containerSecurityContext — same for oxia.coordinator.extraContainers (oxia-coordinator-deployment.yaml:93) and dekaf…extraContainers (dekaf-deployment.yaml:105). The README says the settings apply to "every container and initContainer the chart renders". Either narrow the wording or merge them.
• autorecovery.probe.liveness.enabled: true adds a default-on livenessProbe to a component that previously had none, targeting an endpoint that comes from the image's BookKeeper stats defaults rather than anything the chart sets. A user who changes statsProviderClass/prometheusStatsHttpPort via autorecovery.configData turns a working deployment into a restart loop, where that config was previously inert. Defaulting liveness false and readiness true would be the safer compatibility choice.
• "No functional effect" for the three Jobs inheriting fsGroup: 0 is true for volume ownership but not for admission — fsGroup is an SCC input on OpenShift. Narrow, but the claim as written is incomplete.

_{Reviewed with Codex gpt-5.6-sol and Claude Opus 5; every finding reproduced locally by rendering the chart, validating with kubeconform -strict, and checking OpenShift's shipped SCC manifests.}

[lhotari]

This breaks helm upgrade --reuse-values for every existing release, whether or not they use the new feature.

.Values.autorecovery.probe.liveness.enabled is dereferenced unconditionally here (same for .readiness and .startup below), but autorecovery.probe is a brand-new values key. --reuse-values reuses the previous release's computed values without coalescing in the new chart's defaults, so the map is simply absent:

$ helm template test charts/pulsar --set 'autorecovery.probe=null' \
    -s templates/autorecovery-statefulset.yaml
Error: template: .../autorecovery-statefulset.yaml:180:22: executing ... at
<.Values.autorecovery.probe.liveness.enabled>: nil pointer evaluating interface {}.liveness

Rendering aborts before a single manifest reaches the API server.

writableRootfsVolumes has the same exposure from the other direction: under --reuse-values it's absent, and pulsar.rootfs.enabled tests it by truthiness, so "absent" is indistinguishable from an explicit false. Someone upgrading with --reuse-values --set containerSecurityContext.readOnlyRootFilesystem=true would get read-only roots with no seed container and no mounts.

Guarding with dig or hasKey/default throughout would fix both, e.g.:

{{- if (dig "probe" "liveness" "enabled" false .Values.autorecovery) }}

[lhotari]

The entire initContainers: block — including copy-pulsar-conf two lines below — is inside this waitMetadataTimeout > 0 guard, but the rootfs volumes and the main container's mounts are not. So with waitMetadataTimeout: 0 the bookie gets an empty emptyDir mounted over /pulsar/conf and nothing to seed it:

$ helm template test charts/pulsar \
    --set containerSecurityContext.readOnlyRootFilesystem=true \
    --set bookkeeper.waitMetadataTimeout=0

renders the bookie with no initContainers at all, while the main container still mounts /pulsar/conf, /pulsar/logs and /tmp and all three emptyDir volumes are present. The empty conf volume shadows the image's copy, so apply-config-from-env.py conf/bookkeeper.conf has no file to edit and the bookie never starts.

The seed container needs to render whenever pulsar.rootfs.enabled is true, independently of waitMetadataTimeout — which probably means hoisting initContainers: out of this guard and emitting it when either condition holds.

[lhotari]

This Job breaks under a global readOnlyRootFilesystem: true. Rendered with --set containerSecurityContext.readOnlyRootFilesystem=true --set components.pulsar_manager=true, the container gets readOnlyRootFilesystem: true, image apachepulsar/pulsar-all, and no volumeMounts at all — while the script below does cd /tmp and then curl -sS -D headers.txt, later grepping headers.txt for the token and JSESSIONID. curl can't create the file, so manager initialization never completes.

This also means the README's "Components that do not run a Pulsar image — oxia, dekaf, pulsar_manager — are not given these volumes" is inaccurate: this Job does run a Pulsar image, and it does write to /tmp.

Either give it the rootfs volumes like the other Pulsar-image workloads, or exclude it from the container security context.

[lhotari]

These four overrides are the part I'd most like to see removed — see the main review comment for the full reasoning.

Short version: PSA restricted (and the PSS-restricted policy sets that Gatekeeper and Kyverno ship) place no constraint on fsGroup at all, so nothing in this file's stated target requires moving off GID 0. Meanwhile these four lines are the most operationally dangerous thing in the example — they trigger a recursive chown of every bookie ledger volume, exactly as your warning at the top of the file describes.

And fsGroup: 0 is deliberate: on OpenShift the container user always has GID 0, so group-0 ownership is what makes volumes writable under an arbitrary assigned UID. Moving off it is not a hardening step.

Suggest deleting these four blocks and the runAsUser/runAsGroup/fsGroup/supplementalGroups lines from podSecurityContext above, keeping runAsNonRoot: true and the whole containerSecurityContext block — those are the controls restricted actually enforces.

[lhotari]

This isn't right, and it's repeated in four places in values.yaml too.

fsGroup is applied as a supplementary group to every container process, so runAsGroup doesn't need to match it. The Kubernetes reference example deliberately uses runAsUser: 1000, runAsGroup: 3000, fsGroup: 2000, and its own id output is:

uid=1000 gid=3000 groups=2000,3000,4000

with the group-2000 volume writable.

Worth fixing because the advice actively backfires on OpenShift: following it makes users replace their primary GID 0 — the thing that gives them access to the root-group-owned files baked into the Pulsar image — for no benefit. That undercuts the very model the chart's fsGroup: 0 default relies on.

[lhotari]

Same correction as the README note — runAsGroup does not need to equal fsGroup, since fsGroup is added as a supplementary group. This wording is repeated at lines 860 and 979 as well.

The first half of the sentence is good though: "Override fsGroup here if a cluster policy forbids GID 0" is appropriately conditional, and reads much better than the README's "Moving off GID 0" heading. Might be worth making the README match this framing rather than the other way round.

[mouchar]

Hello Lari, thank you for your time spent on reviewing my PR. All four blocking issues are
fixed, and you're right about the fsGroup framing — the confusion was mine, explained below.

Fixed

• --reuse-values. Every autorecovery.probe lookup now goes through dig with a
  default, and pulsar.rootfs.enabled uses hasKey so an absent writableRootfsVolumes
  means "chart default: enabled" rather than being indistinguishable from an explicit
  false. Verified against a real release record rather than --set …=null: install
  published 4.7.0, then helm upgrade --reuse-values --dry-run — nil pointer before, clean
  render after, and the --set …readOnlyRootFilesystem=true variant now emits the seed
  container and volumes instead of nothing.
• bookkeeper waitMetadataTimeout. initContainers: is emitted whenever anything needs
  it — waitMetadataTimeout, the rootfs feature, cacerts, or user-supplied
  bookkeeper.initContainers — with verify-clusterid still gated on the timeout.
• pulsar-manager cluster-initialize. Now gets the rootfs volumes. The README claim is
  corrected: the pulsar_manager StatefulSet runs apachepulsar/pulsar-manager and stays
  excluded, but its cluster-initialize Job runs a Pulsar image and is included.
• runAsGroup / fsGroup. Removed from the README and all three places in
  values.yaml, replaced with the opposite.
• autorecovery probes. Both liveness and readiness now default to false. Readiness
  targets the same /metrics endpoint, so it has the same failure mode — worse, in a way,
  since a permanently unready pod stalls the StatefulSet rollout rather than just
  restarting. With both off this is purely an added knob, and the default-values delta
  against master drops from the 43 lines you measured to 21 — only the sts-cleanup pod
  labels and the three Jobs rendering their component's fsGroup: 0 remain.
• Values-supplied containers. README now says "every container the chart itself
  renders" and names <component>.initContainers, oxia.coordinator.extraContainers and
  dekaf.deployment.extraContainers as verbatim passthrough. I narrowed the wording rather
  than merging, on the grounds that silently injecting into user-provided YAML is the more
  surprising behaviour — happy to change that if you disagree.
• "No functional effect" for the Jobs inheriting fsGroup: 0 — reworded, since it is
  also an SCC input on OpenShift.

One pre-existing bug, found while fixing the second item

The initContainers: guard is identical on master, so this is latent today and unrelated
to this PR:

helm template t charts/pulsar --set bookkeeper.waitMetadataTimeout=0 \
  --set bookkeeper.initContainers[0].name=x --set bookkeeper.initContainers[0].image=busybox
Error: YAML parse error ... did not find expected key

The user's init containers are emitted without their parent key. My change fixes it as a
side effect. Say the word if you'd rather have it as a separate PR.

The "restricted" confusion — my fault

We were talking about different things. You read it as the restricted Pod Security
Standard; I was using "restricted" in the everyday sense, and naming the file
values-restricted-psp.yaml while the README said "Pod Security Admission, OPA Gatekeeper,
Kyverno" made that reading the obvious one. You're right that PSS restricted places no
constraint on fsGroup — I confirmed it on a namespace labelled
pod-security.kubernetes.io/enforce: restricted, where fsGroup: 0 is admitted and runs
as uid=10000 gid=0(root).

What I was actually targeting is a Gatekeeper constraint built on the PSP-derived
K8sPSPAllowedUsers template with

fsGroup: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]}

which rejects fsGroup: 0 at admission. This is not hypothetical or bespoke — regulated
environments, financial-sector ones in particular, deploy exactly this and workloads are
denied rather than merely flagged. So the capability is needed, but calling it "restricted"
was wrong.

The example is therefore split rather than deleted:

• examples/values-psa-restricted.yaml (new) — PSS restricted: runAsNonRoot: true
  plus your containerSecurityContext block verbatim, and the per-component fsGroup: 0
  defaults left completely alone.
• examples/values-restricted-group-ranges.yaml (renamed) — the group-range case,
  opening with an explicit note that this is not what PSS restricted requires and
  pointing at the other file as the safer default.

The README section is retitled "Overriding fsGroup, if a policy constrains group IDs" and
now states that fsGroup: 0 is not a privilege, that it is what lets the images run under
an arbitrary assigned UID, and that restricted does not constrain it.

I have not added an OpenShift example. It needs to clear the chart defaults rather than
override them, and I'd rather test that on a real cluster before shipping it than guess —
happy to follow up separately.

Re-verification

helm lint clean; 129 clean kubeconform runs across k8s 1.25/1.31/1.36 over the default
values, every .ci/clusters/* config and .ci/templates-all-values.yaml, feature off and
on; both example files render and pass kubeconform -strict on all three versions.