Skip to content

fix: grant the bundle RBAC to the OLM v1 installer ServiceAccount - #3923

Merged
openshift-merge-bot[bot] merged 1 commit into
redhat-developer:mainfrom
zdrapela:fix/olm-v1-installer-rbac
Oct 24, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
redhat-developer:mainfrom
zdrapela:fix/olm-v1-installer-rbac

Conversation

@zdrapela

@zdrapela zdrapela commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Description

OLM v1 installs a bundle with a dedicated installer ServiceAccount, and the API server refuses to create RBAC that grants permissions the installer does not hold itself. The installer ClusterRole rendered by prepare-restricted-environment.sh was not updated when the operator gained access to pods, apiservers/ingresses and the DevHubPluginCatalog CRD, so the OLM v1 path fails with:

failed to create resource: clusterroles.rbac.authorization.k8s.io "rhdh-operator.v2.2-..." is forbidden: ... attempting to grant RBAC permissions not currently held: {pods}, {apiservers}, {ingresses}, {devhubplugincatalogs}, {devhubplugincatalogs/status}

The Helm release then fails halfway: the CRD, the Deployment and the ClusterRoleBinding exist, so the operator pod runs without any permissions and reports forbidden for every resource it watches.

This adds the missing rules and make check-installer-rbac, which compares the installer ClusterRole with the bundle CSV and the ClusterRoles shipped in the bundle, so the two cannot drift again. The check fails on the current main and passes with the fix.

Which issue(s) does this PR fix or relate to

Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4056

Found while fixing the RHDH disconnected OCP nightly jobs: redhat-developer/rhdh#5709

PR acceptance criteria

  • Tests
  • Documentation

How to test changes / Special notes to the reviewer

make check-installer-rbac

On main it reports the 14 missing permissions; with this PR it passes.

@zdrapela
zdrapela requested a review from a team as a code owner October 24, 2026 10:18
@openshift-ci
openshift-ci Bot requested review from gazarenkov and rm3l October 24, 2026 10:18
Comment thread hack/check-installer-rbac.sh Fixed
Comment thread hack/check-installer-rbac.sh Fixed
Comment thread hack/check-installer-rbac.sh Fixed
Comment thread hack/check-installer-rbac.sh Fixed
Comment thread hack/check-installer-rbac.sh Fixed
Comment thread hack/check-installer-rbac.sh Fixed
@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Grant bundle RBAC to the OLM v1 installer ServiceAccount

🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes


AI Description

• Grant the OLM v1 installer the missing bundle permissions so Kubernetes accepts its RBAC
 resources.
• Compare installer permissions against both bundle CSVs and shipped ClusterRoles to prevent drift.
• Run the check in CI when relevant pull-request files change.
Diagram

graph TD
  CI["Pull request CI"] --> Check{"RBAC comparison"}
  CSV["Bundle CSVs"] --> Check
  Roles["Bundle ClusterRoles"] --> Check
  Installer["Installer ClusterRole"] --> Check
  Installer --> SA["Installer ServiceAccount"] --> API["Kubernetes API"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Generate installer RBAC from bundle manifests
  • ➕ Could eliminate manual permission drift rather than detect it.
  • ➖ Would change the preparation workflow and still need to account for installer-specific lifecycle and resource-management permissions.

Recommendation: Keep the targeted permission fix and comparison check for this PR: they address the failed installation without changing how the installer role is rendered. Generating the role from bundle manifests is worth considering separately if manual drift remains frequent.

Files changed (4) +171 / -4

Bug fix (1) +7 / -4
prepare-restricted-environment.shGrant missing permissions to the installer role +7/-4

Grant missing permissions to the installer role

• Extends the rendered ClusterRole with pod, OpenShift API server and ingress, and DevHubPluginCatalog permissions required by the bundle. This lets the OLM v1 installer create the bundle’s RBAC resources.

.rhdh/scripts/prepare-restricted-environment.sh

Other (3) +164 / -0
installer-rbac-check.yamlCheck installer RBAC on relevant pull requests +43/-0

Check installer RBAC on relevant pull requests

• Adds a pull-request workflow for main and release branches. It installs yq and runs the Make target when selected RBAC, bundle CSV, script, or workflow files change.

.github/workflows/installer-rbac-check.yaml

MakefileExpose the installer RBAC check +4/-0

Expose the installer RBAC check

• Adds a phony check-installer-rbac target for local and CI use.

Makefile

check-installer-rbac.shDetect installer and bundle permission drift +117/-0

Detect installer and bundle permission drift

• Extracts the installer ClusterRole from the preparation script and compares its permissions with both bundle CSVs and shipped ClusterRoles. Reports missing group, resource, and verb combinations and fails the check.

hack/check-installer-rbac.sh

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (1)



View medium (3)
🟠 **Medium**
1. Cleanup failure can hide RBAC results 📘 Rule violation
Description
The new EXIT trap runs rm -rf "${tmp_dir}" without neutralizing a non-zero cleanup status. If
cleanup fails while set -e is active, the trap can change the script's final status after the RBAC
check has completed, causing a misleading failure or masking the check result.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The EXIT trap runs `rm -rf "${tmp_dir}"` while `set -e` is enabled, so a cleanup failure can alter the final result of the installer RBAC check.

## Fix Focus Areas
- hack/check-installer-rbac.sh[49-50]

## Recommended Fix
Make the trap cleanup failure-safe by appending `|| true` to the `rm -rf` command or using a narrowly scoped cleanup function that disables errexit for the cleanup operation.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


2. Bundle role changes can bypass the check 🐞 Bug
Description
The workflow’s changed-file filter includes bundle CSVs but omits the bundle ClusterRole manifests
that check-installer-rbac.sh reads. A pull request that changes only one of those roles skips the
check, even if its new permissions exceed those of the installer role.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Bundle ClusterRole changes can skip the installer RBAC check because the workflow does not watch those manifests.
## Fix Focus Areas
- .github/workflows/installer-rbac-check.yaml[25-31]
## Recommended Fix
Add the bundle ClusterRole manifest paths to the changed-file filter. Include any source RBAC paths whose changes should also trigger this check.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


3. Named-resource limits can go undetected 🐞 Bug
Description
rules_to_triples and the CSV extraction discard each rule’s resourceNames before comparing
permissions. If an installer rule becomes limited to named objects while the bundle requires
unrestricted access to the same resource and verb, the check passes although that grant is not
covered.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The check drops resourceNames, so a restricted installer grant can appear to cover an unrestricted bundle grant.
## Fix Focus Areas
- hack/check-installer-rbac.sh[69-74]
- hack/check-installer-rbac.sh[80-103]
## Recommended Fix
Preserve and compare resourceNames when evaluating rule coverage, including the distinction between an absent restriction and a named-resource restriction.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



View low (1)
🟡 **Low**
4. Wildcard grants fail the permission check 🐞 Bug
Description
comm -23 requires literal equality between the emitted group, resource and verb triples rather
than checking whether an installer rule covers a bundle rule. If the installer role is changed to
use wildcard grants, the check reports concrete bundle permissions as missing despite those grants
covering them.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Literal triple comparison rejects installer wildcard rules that cover concrete bundle permissions.
## Fix Focus Areas
- hack/check-installer-rbac.sh[69-74]
- hack/check-installer-rbac.sh[101-103]
## Recommended Fix
Evaluate each required permission against covering installer rules, treating wildcard groups, resources and verbs as coverage rather than requiring identical strings.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



@rhdh-qodo-merge

Copy link
Copy Markdown

Important

The /generate_labels command by Qodo was deprecated on the 1st of October 2026 and is no longer available. We recommend switching to the latest Qodo review capabilities. Learn more

@github-actions

Copy link
Copy Markdown
Contributor

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image Full tag PR tag
Operator quay.io/rhdh-community/operator:2.2.0-pr-3923-c206cb1 quay.io/rhdh-community/operator:2.2.0-pr-3923
Bundle quay.io/rhdh-community/operator-bundle:2.2.0-pr-3923-c206cb1 quay.io/rhdh-community/operator-bundle:2.2.0-pr-3923
Catalog quay.io/rhdh-community/operator-catalog:2.2.0-pr-3923-c206cb1 quay.io/rhdh-community/operator-catalog:2.2.0-pr-3923

zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 24, 2026
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 24, 2026
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
@zdrapela
zdrapela force-pushed the fix/olm-v1-installer-rbac branch from c206cb1 to bc43323 Compare October 24, 2026 10:46
prepare-restricted-environment.sh renders an installer ClusterRole for the
OLM v1 path. OLM installs the bundle with that ServiceAccount, and the API
server refuses to create RBAC that grants permissions the installer does not
hold itself ("attempting to grant RBAC permissions not currently held").

The role was not updated when the operator gained access to pods, apiservers,
ingresses and the DevHubPluginCatalog CRD, so OLM v1 installs fail with:

  failed to create resource: clusterroles.rbac.authorization.k8s.io
  "rhdh-operator.v2.2-..." is forbidden: ... attempting to grant RBAC
  permissions not currently held: {pods}, {apiservers}, {ingresses},
  {devhubplugincatalogs}, {devhubplugincatalogs/status}

The Helm release fails halfway: the CRD, the Deployment and the
ClusterRoleBinding are created, so the operator pod runs without any
permissions and reports "forbidden" for every resource it watches.

Add the missing rules, and a check (make check-installer-rbac) that compares
the installer role with the bundle CSV and the ClusterRoles shipped in the
bundle so they cannot drift again.

Assisted-by: OpenCode
@zdrapela
zdrapela force-pushed the fix/olm-v1-installer-rbac branch from bc43323 to 332e0c9 Compare October 24, 2026 10:50
@github-actions

Copy link
Copy Markdown
Contributor

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image Full tag PR tag
Operator quay.io/rhdh-community/operator:2.2.0-pr-3923-bc43323 quay.io/rhdh-community/operator:2.2.0-pr-3923
Bundle quay.io/rhdh-community/operator-bundle:2.2.0-pr-3923-bc43323 quay.io/rhdh-community/operator-bundle:2.2.0-pr-3923
Catalog quay.io/rhdh-community/operator-catalog:2.2.0-pr-3923-bc43323 quay.io/rhdh-community/operator-catalog:2.2.0-pr-3923

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

✅ PR images built and pushed successfully!

Images are available for testing (expires in 7 days):

Image Full tag PR tag
Operator quay.io/rhdh-community/operator:2.2.0-pr-3923-332e0c9 quay.io/rhdh-community/operator:2.2.0-pr-3923
Bundle quay.io/rhdh-community/operator-bundle:2.2.0-pr-3923-332e0c9 quay.io/rhdh-community/operator-bundle:2.2.0-pr-3923
Catalog quay.io/rhdh-community/operator-catalog:2.2.0-pr-3923-332e0c9 quay.io/rhdh-community/operator-catalog:2.2.0-pr-3923

@openshift-ci openshift-ci Bot added the lgtm label Oct 24, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit e22cc98 into redhat-developer:main Oct 24, 2026
14 checks passed
zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 26, 2026
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 26, 2026
redhat-developer/rhdh-operator#3923 is merged, so the
prepare-restricted-environment.sh fetched from rhdh-operator main renders an
installer ClusterRole that covers the bundle permissions (pods,
apiservers/ingresses, devhubplugincatalogs). The disconnected operator job
downloads that script at runtime, so the workaround is no longer needed on
main.

release-2.1 keeps the workaround until redhat-developer#3924 merges, because that job fetches
the script from rhdh-operator release-2.1.

Assisted-by: OpenCode
zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 26, 2026
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
@zdrapela

Copy link
Copy Markdown
Member Author

/cherry-pick release-2.1

@openshift-cherrypick-robot

Copy link
Copy Markdown
Contributor

@zdrapela: new pull request created: #3927

Details

In response to this:

/cherry-pick release-2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 26, 2026
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 26, 2026
redhat-developer/rhdh-operator#3923 is merged, so the
prepare-restricted-environment.sh fetched from rhdh-operator main renders an
installer ClusterRole that covers the bundle permissions (pods,
apiservers/ingresses, devhubplugincatalogs). The disconnected operator job
downloads that script at runtime, so the workaround is no longer needed on
main.

release-2.1 keeps the workaround until redhat-developer#3924 merges, because that job fetches
the script from rhdh-operator release-2.1.

Assisted-by: OpenCode
zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 26, 2026
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
zdrapela added a commit to zdrapela/rhdh that referenced this pull request Oct 26, 2026
redhat-developer/rhdh-operator#3923 is merged, so the
prepare-restricted-environment.sh fetched from rhdh-operator main renders an
installer ClusterRole that covers the bundle permissions (pods,
apiservers/ingresses, devhubplugincatalogs). The disconnected operator job
downloads that script at runtime, so the workaround is no longer needed on
main.

release-2.1 keeps the workaround until redhat-developer#3924 merges, because that job fetches
the script from rhdh-operator release-2.1.

Assisted-by: OpenCode
openshift-merge-bot Bot pushed a commit to redhat-developer/rhdh that referenced this pull request Oct 27, 2026
* fix(ci): unblock disconnected OCP nightlies

The Helm and Operator disconnected nightlies have not passed for over five
weeks. Three independent gaps kept them red:

- Backend egress to the mirror registry. The chart and the Operator both
  default-deny backend egress except DNS, HTTPS, Redis, and PostgreSQL, while
  the CI mirror serves plain HTTP on port 5000. install-dynamic-plugins timed
  out pulling the catalog index from the bastion.
- Operator catalog. A filtered oc-mirror run against the published Red Hat
  catalog rejects it since rhdh-operator#3822 removed the default `fast`
  channel, and that catalog carries no 2.x bundles anyway. CI runs now mirror
  the RHDH CI catalog (quay.io/rhdh/iib:<stream>-v<ocp>-x86_64); its
  registry.redhat.io/rhdh relatedImages are redirected to quay.io/rhdh with
  CONTAINERS_REGISTRIES_CONF for the prepare-restricted-environment.sh call.
- Smoke plugin set. On the 2.x New Frontend System hub the sign-in page,
  sidebar, SCM integrations, and guest auth provider are dynamic plugins, so
  the homepage-only set could not render a login. Mirror the minimal digest
  pinned set and configure the NFS extensions and signInPage in the app
  configs.

The 2.x hub image and the new plugin set were verified on a live OCP cluster:
guest sign-in, sidebar, and the "Welcome back!" homepage heading all render.

Assisted-by: OpenCode

* fix(ci): address disconnected review feedback

- Pick the operator catalog from the chart build type instead of the image
  registry alone: a CI chart (x.y-N-CI) mirrors the RHDH CI catalog, a
  released chart (x.y.z) keeps the published catalog. A pinned RC/GA
  verification on main (TAG_NAME and CHART_VERSION 1.9.4) was handed a `next`
  catalog for a 1.9 channel filter.
- Export the smoke plugin list as a space-separated scalar. bash cannot export
  arrays, so the `export -f` helpers saw an empty list in a child shell and
  would have mirrored every digest-pinned plugin. Fail when no package
  variable was exported.
- Document why the mirror egress rule is scoped by port and not by
  destination: the bastion address is assigned per run and reachable by DNS
  name only, so an ipBlock would break whenever the bastion is replaced.

Assisted-by: OpenCode

* fix(ci): mount the Helm registry auth secret under the chart name

The chart mounts <fullname>-dynamic-plugins-registry-auth, and fullname is
<release>-<nameOverride> with nameOverride pinned to developer-hub. The job
created rhdh-dynamic-plugins-registry-auth instead, so the optional volume was
never mounted, install-dynamic-plugins ran without auth.json, and skopeo failed
the mirror pull with "authentication required". Pin nameOverride in the values
and create the secret under the chart's name.

Verified on a live OCP 4.21 cluster (LOCAL_DISCONNECTED): the init container
now pulls the catalog index and all five smoke plugins through the mirror.

Assisted-by: OpenCode

* fix(ci): stop MCP drains mid-deploy and fail fast on OLM v1 RBAC errors

Helm: after the IDMS/ITMS apply, `oc wait machineconfigpool
--for=condition=Updated=True` returned while the MCO had not rendered the new
config yet. The nodes were drained a few seconds later, while the hub and
PostgreSQL pods were starting, so they were evicted mid-startup. The hub
recovery path then ran `oc rollout restart`, starting a second backend next to
the first against a freshly initialised database; the two raced on the knex
migration lock and the losing backend stayed up with readiness 503 forever.

- snapshot each pool's rendered config before the IDMS/ITMS and
  additionalTrustedCA changes, and wait for the MCO to render before waiting
  for convergence (poll Updated + updatedMachineCount; warn and continue when
  nothing is rolling)
- start the hub restart grace period only once the backend container has
  started, and replace the rollout restart with scale to 0, wait for the pods
  to be gone, scale to 1, so two backends never overlap

Operator: OLM v1 installs the bundle with the rhdh-operator-installer
ServiceAccount, which cannot grant RBAC it does not hold itself. The installer
ClusterRole rendered by prepare-restricted-environment.sh lags behind
config/rbac/role.yaml (pods, apiservers/ingresses, devhubplugincatalogs), so
the operator ClusterRole creation is refused and the operator ends up running
with no permissions. The half-applied release still creates the CRD, so the
CRD-only wait passed and the job failed later with a misleading message.

- require ClusterExtension Installed=True as well as the CRD, log the install
  condition at info level, and dump OLM v1 status when the Backstage CR
  produces no deployment
- TEMPORARY: grant the installer the operator's own ClusterRole rules from the
  same release branch before prepare runs, so the first install attempt
  succeeds. Remove once redhat-developer/rhdh-operator ships the fix.

Verified the RBAC workaround on a live OCP cluster: the same ClusterRole
creation is refused before the grant and succeeds after it.

Assisted-by: OpenCode

* chore(ci): reference the installer RBAC fix and ticket

Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport #3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode

* chore(ci): drop the temporary OLM v1 installer RBAC workaround

redhat-developer/rhdh-operator#3923 is merged, so the
prepare-restricted-environment.sh fetched from rhdh-operator main renders an
installer ClusterRole that covers the bundle permissions (pods,
apiservers/ingresses, devhubplugincatalogs). The disconnected operator job
downloads that script at runtime, so the workaround is no longer needed on
main.

release-2.1 keeps the workaround until #3924 merges, because that job fetches
the script from rhdh-operator release-2.1.

Assisted-by: OpenCode

* fix(ci): derive the disconnected operator filter from the release stream

The operator job took the bundle filter from CHART_VERSION while the catalog tag
came from RELEASE_VERSION, so a pinned run on main (for example an RC tag) asked
oc-mirror for bundles the `next` catalog does not carry.

Both now come from the branch: RELEASE_VERSION for the catalog tag (next on
main, 2.1 on release-2.1) and common::get_release_stream for the filter, which
needs a major.minor and resolves main to the package.json stream (2.2).

Move that helper out of lib/helm.sh while at it: the operator and upgrade jobs
use it too, so it lives in lib/common.sh as common::get_release_stream.

Assisted-by: OpenCode

* chore(ci): trim disconnected code comments

Drop the comments that only describe the PR history or restate the code
(merged rhdh-operator PRs, the NFS plugin rationale repeated in three files,
snapshot double-comments, a duplicated secret-mount paragraph). Keep the notes
that explain non-obvious behaviour: the guest sign-in page, nameOverride
coupling, CI vs released catalogs, the registries.conf redirect and the
additionalTrustedCA node roll.

Also log the configured MachineConfigPool timeout instead of a hardcoded 20m,
and correct the mirror egress rationale: MIRROR_REGISTRY_URL is the bastion's
public EC2 DNS name, which resolves differently from inside the cluster, so
there is no stable ipBlock to use.

Assisted-by: OpenCode

* fix(ci): take the CI operator catalog tag from the release stream

The operator job picked the catalog tag from RELEASE_VERSION and the
--filter-versions value from common::get_release_stream. On main those are
"next" and the package.json stream (2.2), which agree only while rhdh and
rhdh-operator bump together; when one side lags, prepare keeps a fast-X.Y
channel the catalog does not carry, or oc-mirror rejects the catalog because
its default channel is filtered out.

Use filter_versions for the tag as well. Both catalog tags resolve to the same
digest today (iib:next-v4.21 == iib:2.2-v4.21), so nightlies are unchanged,
and a stream without a published catalog now fails at the skopeo check with a
readable error.

Assisted-by: OpenCode

* chore(ci): tidy disconnected helpers

- list jq in common.sh dependencies: the moved get_release_stream uses it
- rename mcp_all_converged to the underscore prefix the other private
  helpers use
- drop the unreachable exported -eq 0 branch in the smoke plugin export
- document the hub grace period as it behaves: it covers the backend start
  wait, then the Available wait

Assisted-by: OpenCode

* refactor(ci): always mirror the branch's CI operator catalog

The operator job fell back to the published Red Hat catalog whenever the image
registry was not registry.redhat.io or the chart was not a CI tag. That path
cannot work: prepare-restricted-environment.sh writes only fast-X.Y channels
into the ImageSetConfiguration, so oc-mirror rejects the published catalog
because its default fast channel is filtered out, and the catalog carries no
2.x bundles at all.

Every disconnected run is branch-aligned, and the hub image comes from the
bundle's RELATED_IMAGE_backstage, so TAG_NAME, IMAGE_* and CHART_VERSION never
reached the deployment. Use the CI catalog
(quay.io/rhdh/iib:<stream>-v<ocp>-x86_64) and its registries.conf redirect
unconditionally, and log that only the newest bundle of the branch's stream is
tested.

Assisted-by: OpenCode

* fix(ci): fail disconnected deploys on errors instead of waiting

Putting deploy_rhdh_operator on the left of || turns errexit off for its whole
body, so a failed envsubst, yq or oc apply fell through to the 5-minute
deployment poll and then reported a missing deployment, which pointed at OLM.

Return from deploy_rhdh_operator as soon as rendering or applying the CR fails,
and use a neutral message in the operator job (the OLM v1 dump stays as
diagnostics).

The Helm hub bounce scaled the Deployment back up even when the old pod was
still present after 180s, recreating the two-backend race it is meant to
prevent. Treat that timeout as a recovery failure.

Assisted-by: OpenCode

* fix(ci): retry MachineConfigPool reads and drop the MCP abbreviation

disconnected::snapshot_mcp_rendered_configs hid oc failures by returning an
empty success: a failed pre-change read skipped render detection, and a failed
read during detection looked like a changed config. Retry the read three times
and fail on an error or an empty answer, skip failed reads in the render loop,
and require the pre-change snapshot. Both jobs now stop when it cannot be taken.

Rename the MCP identifiers to machineconfigpool, since MCP reads as Model
Context Protocol in this repository.

Document why the IDMS and ITMS applies share one wait: a pool's spec moves to
the newest rendered config, so a config rendered during the first rollout is
still awaited.

Assisted-by: OpenCode

* refactor(ci): derive smoke plugin checks from the plugin list

The smoke set was spelled out in six places: the array, the case in
_smoke_plugin_package_var, two require_vars lists and two templates. A template
entry the array did not know about rendered an empty package: and only failed at
pod runtime.

Render both templates through one helper that maps the array to the package
variables, requires them, and rejects a package: left empty by envsubst.

Assisted-by: OpenCode

* docs(ci): correct disconnected comments

- the CI mirror serves TLS on port 5000, not plain HTTP: the openshift/release
  step curls https://<mirror>/v2/ with the vault CA
- nameOverride pins the chart fullname for the registry-auth secret; the egress
  NetworkPolicy selects on pod labels
- the smoke app config is a subset of dynamic-plugins-config.yaml, not a mirror
- the Backstage CR installs the minimal NFS plugin set, not only the homepage

Assisted-by: OpenCode

* fix(ci): keep the published catalog for GA verification

A GA verification run (--image-registry registry.redhat.io with a released x.y.z
chart) should pair with the published Red Hat catalog, and the existing image and
chart overrides already select it. Restore that branch.

It cannot keep the version filter: the published catalog declares defaultChannel
fast, and oc-mirror rejects a channels list that leaves the default channel out,
which is why GA runs failed at mirroring before. Pass --filter-versions '*' for
the published catalog (prepare then mirrors every rhdh channel, still scoped to
the rhdh package) and keep the branch-stream filter for the CI catalog, so the
nightlies are unchanged.

Assisted-by: OpenCode
openshift-merge-bot Bot pushed a commit to redhat-developer/rhdh that referenced this pull request Oct 27, 2026
* fix(ci): unblock disconnected OCP nightlies

The Helm and Operator disconnected nightlies have not passed for over five
weeks. Three independent gaps kept them red:

- Backend egress to the mirror registry. The chart and the Operator both
  default-deny backend egress except DNS, HTTPS, Redis, and PostgreSQL, while
  the CI mirror serves plain HTTP on port 5000. install-dynamic-plugins timed
  out pulling the catalog index from the bastion.
- Operator catalog. A filtered oc-mirror run against the published Red Hat
  catalog rejects it since rhdh-operator#3822 removed the default `fast`
  channel, and that catalog carries no 2.x bundles anyway. CI runs now mirror
  the RHDH CI catalog (quay.io/rhdh/iib:<stream>-v<ocp>-x86_64); its
  registry.redhat.io/rhdh relatedImages are redirected to quay.io/rhdh with
  CONTAINERS_REGISTRIES_CONF for the prepare-restricted-environment.sh call.
- Smoke plugin set. On the 2.x New Frontend System hub the sign-in page,
  sidebar, SCM integrations, and guest auth provider are dynamic plugins, so
  the homepage-only set could not render a login. Mirror the minimal digest
  pinned set and configure the NFS extensions and signInPage in the app
  configs.

The 2.x hub image and the new plugin set were verified on a live OCP cluster:
guest sign-in, sidebar, and the "Welcome back!" homepage heading all render.

Assisted-by: OpenCode

* fix(ci): address disconnected review feedback

- Pick the operator catalog from the chart build type instead of the image
  registry alone: a CI chart (x.y-N-CI) mirrors the RHDH CI catalog, a
  released chart (x.y.z) keeps the published catalog. A pinned RC/GA
  verification on main (TAG_NAME and CHART_VERSION 1.9.4) was handed a `next`
  catalog for a 1.9 channel filter.
- Export the smoke plugin list as a space-separated scalar. bash cannot export
  arrays, so the `export -f` helpers saw an empty list in a child shell and
  would have mirrored every digest-pinned plugin. Fail when no package
  variable was exported.
- Document why the mirror egress rule is scoped by port and not by
  destination: the bastion address is assigned per run and reachable by DNS
  name only, so an ipBlock would break whenever the bastion is replaced.

Assisted-by: OpenCode

* fix(ci): mount the Helm registry auth secret under the chart name

The chart mounts <fullname>-dynamic-plugins-registry-auth, and fullname is
<release>-<nameOverride> with nameOverride pinned to developer-hub. The job
created rhdh-dynamic-plugins-registry-auth instead, so the optional volume was
never mounted, install-dynamic-plugins ran without auth.json, and skopeo failed
the mirror pull with "authentication required". Pin nameOverride in the values
and create the secret under the chart's name.

Verified on a live OCP 4.21 cluster (LOCAL_DISCONNECTED): the init container
now pulls the catalog index and all five smoke plugins through the mirror.

Assisted-by: OpenCode

* fix(ci): stop MCP drains mid-deploy and fail fast on OLM v1 RBAC errors

Helm: after the IDMS/ITMS apply, `oc wait machineconfigpool
--for=condition=Updated=True` returned while the MCO had not rendered the new
config yet. The nodes were drained a few seconds later, while the hub and
PostgreSQL pods were starting, so they were evicted mid-startup. The hub
recovery path then ran `oc rollout restart`, starting a second backend next to
the first against a freshly initialised database; the two raced on the knex
migration lock and the losing backend stayed up with readiness 503 forever.

- snapshot each pool's rendered config before the IDMS/ITMS and
  additionalTrustedCA changes, and wait for the MCO to render before waiting
  for convergence (poll Updated + updatedMachineCount; warn and continue when
  nothing is rolling)
- start the hub restart grace period only once the backend container has
  started, and replace the rollout restart with scale to 0, wait for the pods
  to be gone, scale to 1, so two backends never overlap

Operator: OLM v1 installs the bundle with the rhdh-operator-installer
ServiceAccount, which cannot grant RBAC it does not hold itself. The installer
ClusterRole rendered by prepare-restricted-environment.sh lags behind
config/rbac/role.yaml (pods, apiservers/ingresses, devhubplugincatalogs), so
the operator ClusterRole creation is refused and the operator ends up running
with no permissions. The half-applied release still creates the CRD, so the
CRD-only wait passed and the job failed later with a misleading message.

- require ClusterExtension Installed=True as well as the CRD, log the install
  condition at info level, and dump OLM v1 status when the Backstage CR
  produces no deployment
- TEMPORARY: grant the installer the operator's own ClusterRole rules from the
  same release branch before prepare runs, so the first install attempt
  succeeds. Remove once redhat-developer/rhdh-operator ships the fix.

Verified the RBAC workaround on a live OCP cluster: the same ClusterRole
creation is refused before the grant and succeeds after it.

Assisted-by: OpenCode

* chore(ci): reference the installer RBAC fix and ticket

Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport #3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode

* chore(ci): drop the temporary OLM v1 installer RBAC workaround

redhat-developer/rhdh-operator#3923 is merged, so the
prepare-restricted-environment.sh fetched from rhdh-operator main renders an
installer ClusterRole that covers the bundle permissions (pods,
apiservers/ingresses, devhubplugincatalogs). The disconnected operator job
downloads that script at runtime, so the workaround is no longer needed on
main.

release-2.1 keeps the workaround until #3924 merges, because that job fetches
the script from rhdh-operator release-2.1.

Assisted-by: OpenCode

* fix(ci): derive the disconnected operator filter from the release stream

The operator job took the bundle filter from CHART_VERSION while the catalog tag
came from RELEASE_VERSION, so a pinned run on main (for example an RC tag) asked
oc-mirror for bundles the `next` catalog does not carry.

Both now come from the branch: RELEASE_VERSION for the catalog tag (next on
main, 2.1 on release-2.1) and common::get_release_stream for the filter, which
needs a major.minor and resolves main to the package.json stream (2.2).

Move that helper out of lib/helm.sh while at it: the operator and upgrade jobs
use it too, so it lives in lib/common.sh as common::get_release_stream.

Assisted-by: OpenCode

* chore(ci): trim disconnected code comments

Drop the comments that only describe the PR history or restate the code
(merged rhdh-operator PRs, the NFS plugin rationale repeated in three files,
snapshot double-comments, a duplicated secret-mount paragraph). Keep the notes
that explain non-obvious behaviour: the guest sign-in page, nameOverride
coupling, CI vs released catalogs, the registries.conf redirect and the
additionalTrustedCA node roll.

Also log the configured MachineConfigPool timeout instead of a hardcoded 20m,
and correct the mirror egress rationale: MIRROR_REGISTRY_URL is the bastion's
public EC2 DNS name, which resolves differently from inside the cluster, so
there is no stable ipBlock to use.

Assisted-by: OpenCode

* fix(ci): take the CI operator catalog tag from the release stream

The operator job picked the catalog tag from RELEASE_VERSION and the
--filter-versions value from common::get_release_stream. On main those are
"next" and the package.json stream (2.2), which agree only while rhdh and
rhdh-operator bump together; when one side lags, prepare keeps a fast-X.Y
channel the catalog does not carry, or oc-mirror rejects the catalog because
its default channel is filtered out.

Use filter_versions for the tag as well. Both catalog tags resolve to the same
digest today (iib:next-v4.21 == iib:2.2-v4.21), so nightlies are unchanged,
and a stream without a published catalog now fails at the skopeo check with a
readable error.

Assisted-by: OpenCode

* chore(ci): tidy disconnected helpers

- list jq in common.sh dependencies: the moved get_release_stream uses it
- rename mcp_all_converged to the underscore prefix the other private
  helpers use
- drop the unreachable exported -eq 0 branch in the smoke plugin export
- document the hub grace period as it behaves: it covers the backend start
  wait, then the Available wait

Assisted-by: OpenCode

* refactor(ci): always mirror the branch's CI operator catalog

The operator job fell back to the published Red Hat catalog whenever the image
registry was not registry.redhat.io or the chart was not a CI tag. That path
cannot work: prepare-restricted-environment.sh writes only fast-X.Y channels
into the ImageSetConfiguration, so oc-mirror rejects the published catalog
because its default fast channel is filtered out, and the catalog carries no
2.x bundles at all.

Every disconnected run is branch-aligned, and the hub image comes from the
bundle's RELATED_IMAGE_backstage, so TAG_NAME, IMAGE_* and CHART_VERSION never
reached the deployment. Use the CI catalog
(quay.io/rhdh/iib:<stream>-v<ocp>-x86_64) and its registries.conf redirect
unconditionally, and log that only the newest bundle of the branch's stream is
tested.

Assisted-by: OpenCode

* fix(ci): fail disconnected deploys on errors instead of waiting

Putting deploy_rhdh_operator on the left of || turns errexit off for its whole
body, so a failed envsubst, yq or oc apply fell through to the 5-minute
deployment poll and then reported a missing deployment, which pointed at OLM.

Return from deploy_rhdh_operator as soon as rendering or applying the CR fails,
and use a neutral message in the operator job (the OLM v1 dump stays as
diagnostics).

The Helm hub bounce scaled the Deployment back up even when the old pod was
still present after 180s, recreating the two-backend race it is meant to
prevent. Treat that timeout as a recovery failure.

Assisted-by: OpenCode

* fix(ci): retry MachineConfigPool reads and drop the MCP abbreviation

disconnected::snapshot_mcp_rendered_configs hid oc failures by returning an
empty success: a failed pre-change read skipped render detection, and a failed
read during detection looked like a changed config. Retry the read three times
and fail on an error or an empty answer, skip failed reads in the render loop,
and require the pre-change snapshot. Both jobs now stop when it cannot be taken.

Rename the MCP identifiers to machineconfigpool, since MCP reads as Model
Context Protocol in this repository.

Document why the IDMS and ITMS applies share one wait: a pool's spec moves to
the newest rendered config, so a config rendered during the first rollout is
still awaited.

Assisted-by: OpenCode

* refactor(ci): derive smoke plugin checks from the plugin list

The smoke set was spelled out in six places: the array, the case in
_smoke_plugin_package_var, two require_vars lists and two templates. A template
entry the array did not know about rendered an empty package: and only failed at
pod runtime.

Render both templates through one helper that maps the array to the package
variables, requires them, and rejects a package: left empty by envsubst.

Assisted-by: OpenCode

* docs(ci): correct disconnected comments

- the CI mirror serves TLS on port 5000, not plain HTTP: the openshift/release
  step curls https://<mirror>/v2/ with the vault CA
- nameOverride pins the chart fullname for the registry-auth secret; the egress
  NetworkPolicy selects on pod labels
- the smoke app config is a subset of dynamic-plugins-config.yaml, not a mirror
- the Backstage CR installs the minimal NFS plugin set, not only the homepage

Assisted-by: OpenCode

* fix(ci): keep the published catalog for GA verification

A GA verification run (--image-registry registry.redhat.io with a released x.y.z
chart) should pair with the published Red Hat catalog, and the existing image and
chart overrides already select it. Restore that branch.

It cannot keep the version filter: the published catalog declares defaultChannel
fast, and oc-mirror rejects a channels list that leaves the default channel out,
which is why GA runs failed at mirroring before. Pass --filter-versions '*' for
the published catalog (prepare then mirrors every rhdh channel, still scoped to
the rhdh package) and keep the branch-stream filter for the CI catalog, so the
nightlies are unchanged.

Assisted-by: OpenCode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants