Skip to content

[release-2.1] fix(ci): unblock disconnected OCP nightlies - #5710

Merged
openshift-merge-bot[bot] merged 16 commits into
redhat-developer:release-2.1from
zdrapela:fix/ci-disconnected-nightlies-2.1
Oct 27, 2026
Merged

openshift-merge-bot[bot] merged 16 commits into
redhat-developer:release-2.1from
zdrapela:fix/ci-disconnected-nightlies-2.1

Conversation

@zdrapela

@zdrapela zdrapela commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Cherry-pick of #5709 to release-2.1, rebased on the current release-2.1. The disconnected job/lib/value/resource files are identical between the two branches.

Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4056

The disconnected OCP Helm and Operator nightlies have not passed for over five weeks. These gaps were blocking them:

  1. Backend egress to the CI mirror registry — the chart and the Operator default-deny backend egress except DNS, HTTPS, Redis and PostgreSQL, while the bastion mirror serves TLS on port 5000 (not 443). disconnected::allow_mirror_registry_egress applies a targeted rule for the mirror's port (Helm + Operator pod selectors); skipped on 443 (LOCAL_DISCONNECTED).
  2. Operator job: use the RHDH CI catalog — since rhdh-operator#3821 (this branch) a filtered oc-mirror run rejects the published catalog (default fast channel filtered out), and it has no 2.x bundles. CI runs now use quay.io/rhdh/iib:2.1-v<ocp>-x86_64 with the registry.redhat.io/rhdh relatedImages redirected to quay.io/rhdh via CONTAINERS_REGISTRIES_CONF. The tag follows the same common::get_release_stream value passed to --filter-versions 2.1, so the catalog and its fast-2.1 channel cannot disagree. CI runs mirror the newest CI bundle of the branch's stream: the hub image comes from the bundle, so pinned TAG_NAME, IMAGE_* and CHART_VERSION settings do not apply to them. A GA run (registry.redhat.io with a released x.y.z chart) uses the published catalog without a version filter, because filtering would drop its default fast channel.
  3. Smoke setup for the 2.x NFS hub — mirror app-auth, app-defaults, app-integrations, guest provider and homepage, plus the NFS app.extensions config and signInPage: guest.
  4. Helm registry auth secret name — the chart mounts <fullname>-dynamic-plugins-registry-auth; the job created a differently named secret, so install-dynamic-plugins had no auth.json and the mirror pull failed with "authentication required".
  5. MachineConfigPool wait returned before the MCO rendered the IDMS/ITMS config, so nodes were drained mid-deploy and the hub/PostgreSQL pods were evicted. It now snapshots the rendered config before the change (retrying failed reads) and waits for the MCO to pick it up before waiting for convergence.
  6. Hub recovery started two backends (oc rollout restart), which raced on the knex migration lock and left a backend at readiness 503 forever. The bounce is now scale-to-zero then scale-to-one, the wait gives a pod that is still extracting plugins time to start its backend first, and the bounce refuses to scale up while an old pod remains.
  7. OLM v1 install failures were masked — the wait now requires ClusterExtension Installed=True as well as the CRD, logs the install condition, and dumps OLM v1 status when deploying the Backstage CR fails.

get_release_stream moved from helm::get_chart_stream to common::get_release_stream, since the operator and upgrade jobs use it too.

The OLM v1 installer ClusterRole in prepare-restricted-environment.sh was missing permissions (pods, apiservers/ingresses, devhubplugincatalogs). It is fixed on rhdh-operator main (#3923) and on release-2.1 (#3927). This branch downloads the script from rhdh-operator release-2.1, so it now installs with the fixed installer role; the temporary verification workaround has been removed.

The disconnected jobs exist only on main and release-2.1, and both now target 2.x hubs: the smoke plugin set is the minimal New Frontend System set and the operator stream follows the branch. Verification of 1.x releases is not covered by these jobs.

Verified on a live OCP 4.21 cluster with the 2.x hub image: guest sign-in, sidebar and the Welcome back! homepage heading render; plugin list/package export exercised against the real :2.1 catalog.

The nightly jobs on this branch additionally need the upi-installer base_image fix in openshift/release (tracked separately).

Ref: #5709

@openshift-ci
openshift-ci Bot requested review from gustavolira and psrna October 23, 2026 14:11
@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Unblock disconnected OCP Helm and Operator nightlies

🐞 Bug fix ⚙️ Configuration changes 🕐 40+ Minutes


AI Description

• Allow backend egress to CI mirror registries serving on non-HTTPS ports.
• Mirror the appropriate Operator catalog and redirect CI bundle image pulls.
• Install and configure the minimal dynamic plugin set needed for 2.x smoke tests.
Diagram

graph TD
  Jobs["Disconnected jobs"] --> Catalog["Catalog preparation"] --> Mirror[("Mirror registry")] --> Hub["Hub backend"]
  Jobs --> Plugins["Smoke plugin list"] --> Mirror
  Jobs --> Policy["Mirror egress policy"] --> Hub
  Plugins --> Config["Smoke configuration"] --> Hub
Loading
High-Level Assessment

Keep the targeted approach: the published Operator catalog lacks the required CI bundles, and mirroring the entire plugin catalog would reintroduce fragile tag references. Scoping the image redirect to catalog preparation and egress to the mirror port avoids broadening unrelated traffic.

Files changed (11) +292 / -71

Bug fix (5) +223 / -57
ocp-disconnected-helm.shPrepare mirror access and complete Helm smoke plugin values +7/-3

Prepare mirror access and complete Helm smoke plugin values

• Applies mirror-registry egress policy before deployment and requires all five digest-pinned plugin packages when rendering Helm smoke values.

.ci/pipelines/jobs/ocp-disconnected-helm.sh

ocp-disconnected-operator.shSelect the CI Operator catalog and complete smoke deployment +57/-8

Select the CI Operator catalog and complete smoke deployment

• Uses the RHDH CI catalog for non-GA runs and scopes related-image registry redirects to Operator preparation. Applies mirror egress policy and creates the expanded smoke plugin ConfigMap.

.ci/pipelines/jobs/ocp-disconnected-operator.sh

plugins.shMirror the smoke plugin set in local runs +5/-7

Mirror the smoke plugin set in local runs

• Replaces the homepage-only list with the shared digest-pinned smoke list while retaining local registry retries and tagging.

.ci/pipelines/lib/disconnected/local/plugins.sh

namespace.shPermit backend egress to non-443 mirror ports +69/-0

Permit backend egress to non-443 mirror ports

• Creates port-specific NetworkPolicies for Helm and Operator backend pods when the mirror uses an explicit non-443 port. Skips the extra rule for mirrors served on 443.

.ci/pipelines/lib/disconnected/namespace.sh

plugins.shResolve and configure five digest-pinned smoke plugins +85/-39

Resolve and configure five digest-pinned smoke plugins

• Filters the catalog to the minimal 2.x smoke plugin set and exports each package reference. Both mirroring paths and the Operator ConfigMap now consume that set.

.ci/pipelines/lib/disconnected/plugins.sh

Refactor (1) +5 / -2
disconnected.shExport shared smoke and mirror-egress helpers +5/-2

Export shared smoke and mirror-egress helpers

• Exports the new plugin-list, package-resolution, ConfigMap, and network-policy helpers for disconnected jobs.

.ci/pipelines/lib/disconnected.sh

Documentation (2) +5 / -3
local.shClarify the local plugin-mirroring scope +1/-1

Clarify the local plugin-mirroring scope

• Updates the local-disconnected module description from homepage-only to the shared smoke plugin set.

.ci/pipelines/lib/disconnected/local.sh

rhdh-start-disconnected-smoke.yamlClarify CI and GA hub image sources +4/-2

Clarify CI and GA hub image sources

• Documents how the selected Operator catalog supplies different CI and GA hub images for the disconnected Backstage deployment.

.ci/pipelines/resources/rhdh-operator/rhdh-start-disconnected-smoke.yaml

Other (3) +59 / -9
app-config-rhdh-disconnected-smoke.yamlConfigure Operator smoke guest login and homepage +13/-0

Configure Operator smoke guest login and homepage

• Sets the explicit guest sign-in page and NFS homepage extensions, disabling the competing community home page.

.ci/pipelines/resources/disconnected/app-config-rhdh-disconnected-smoke.yaml

dynamic-plugins-smoke.yamlDefine the Operator smoke plugin set +18/-0

Define the Operator smoke plugin set

• Adds an Operator ConfigMap template enabling app-auth, app-defaults, app-integrations, guest provider, and homepage from digest-pinned packages.

.ci/pipelines/resources/disconnected/dynamic-plugins-smoke.yaml

values_disconnected-smoke.yamlEnable Helm's 2.x smoke plugins and NFS extensions +28/-9

Enable Helm's 2.x smoke plugins and NFS extensions

• Adds four required plugins alongside homepage and configures explicit guest sign-in and a single homepage destination.

.ci/pipelines/value_files/values_disconnected-smoke.yaml

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (f90866e) to head (7e9d963).
⚠️ Report is 1 commits behind head on release-2.1.

Additional details and impacted files
@@               Coverage Diff               @@
##           release-2.1    #5710      +/-   ##
===============================================
- Coverage        75.70%   69.00%   -6.70%     
===============================================
  Files               65       58       -7     
  Lines              708      555     -153     
  Branches           100       83      -17     
===============================================
- Hits               536      383     -153     
  Misses             171      171              
  Partials             1        1              
Flag Coverage Δ
rhdh 69.00% <ø> (-6.70%) ⬇️
Components Coverage Δ
Backend plugins ∅ <ø> (∅)
Backend app 78.94% <ø> (ø)
Frontend app 61.03% <ø> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f90866e...7e9d963. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1)



View medium (1)
🟠 **Medium**
1. Backend pods can reach unrelated services 🐞 Bug
Description
allow_mirror_registry_egress limits the new NetworkPolicies by TCP port but omits an egress
destination. When the mirror uses port 5000, the selected backend pods can connect to any reachable
service on that port, not just the mirror registry.
Code

.ci/pipelines/lib/disconnected/namespace.sh[R159-162]

+  egress:
+    - ports:
+        - port: ${registry_port}
+          protocol: TCP
Evidence
The function extracts only the mirror's port. Both new egress rules specify ports without to, so
neither restricts destinations; the repository's existing policy documentation confirms that these
allowances are additive to the backend policies.

.ci/pipelines/lib/disconnected/namespace.sh[130-142]
.ci/pipelines/lib/disconnected/namespace.sh[153-179]
.ci/pipelines/resources/network_policy/netpol-ci-allow-backend-egress.yaml[1-7]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new mirror-registry policies permit traffic to every destination on the mirror's TCP port, weakening the backend's default-deny egress policy.

## Fix Focus Areas
- .ci/pipelines/lib/disconnected/namespace.sh[130-186]

## Recommended Fix
Constrain both egress rules to the mirror destination as well as its port. Resolve and validate a stable mirror address or CIDR for an `ipBlock`, or use an appropriate selector when the mirror is inside the cluster.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Resolved findings
1. Smoke plugin list is empty inside child shells ✓ Resolved
Description
The single exported DISCONNECTED_HOMEPAGE_PLUGIN_NAME constant is replaced by `readonly -a
DISCONNECTED_SMOKE_PLUGIN_NAMES, and bash cannot export arrays. Yet write_smoke_plugin_list` and
set_smoke_plugin_packages_from_list are still exported with export -f so they can run in child
shells (timeout bash -c). In such a child shell the array is empty, so write_digest_plugin_list
gets an empty filter and keeps every digest-pinned plugin in the catalog. The export loop then sets
no *_PLUGIN_PACKAGE variables and still returns success.
Code

.ci/pipelines/lib/disconnected/plugins.sh[R12-18]

+readonly -a DISCONNECTED_SMOKE_PLUGIN_NAMES=(
+  "red-hat-developer-hub-backstage-plugin-app-auth"
+  "red-hat-developer-hub-backstage-plugin-app-defaults"
+  "red-hat-developer-hub-backstage-plugin-app-integrations"
+  "red-hat-developer-hub-backstage-plugin-homepage"
+  "backstage-plugin-auth-backend-module-guest-provider"
+)
Evidence
disconnected.sh says the functions are exported for subshell use and exports the new helpers. The
old constant was exported explicitly so child shells could see it. The new array has no export path.
The jobs currently call mirror_plugins directly, so this is a latent regression of the documented
subshell contract rather than a failure in today's jobs.

.ci/pipelines/lib/disconnected.sh[46-76]
.ci/pipelines/lib/disconnected/plugins.sh[199-241]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`DISCONNECTED_SMOKE_PLUGIN_NAMES` is a bash array. Arrays cannot be exported, so the exported functions `write_smoke_plugin_list` and `set_smoke_plugin_packages_from_list` see an empty list when they run in a child shell. The old homepage constant was exported for exactly this case.

## Fix Focus Areas
- .ci/pipelines/lib/disconnected/plugins.sh[12-18]
- .ci/pipelines/lib/disconnected/plugins.sh[203-241]

## Recommended Fix
Store the names in an exported space-separated string, e.g. `export DISCONNECTED_SMOKE_PLUGIN_NAMES="a b c ..."`. Loop over it with `for name in ${DISCONNECTED_SMOKE_PLUGIN_NAMES}`, or split it into a local array with `read -ra`. Also make `set_smoke_plugin_packages_from_list` fail when it exports zero packages.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@zdrapela zdrapela changed the title fix(ci): unblock disconnected OCP nightlies [release-2.1] fix(ci): unblock disconnected OCP nightlies Oct 26, 2026
@zdrapela
zdrapela force-pushed the fix/ci-disconnected-nightlies-2.1 branch from 6aba6d0 to 44f3693 Compare October 26, 2026 07:57
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

The Helm and Operator disconnected nightlies have not passed for over five
weeks. Three independent gaps kept them red:

- Backend egress to the mirror registry. The chart and the Operator both
  default-deny backend egress except DNS, HTTPS, Redis, and PostgreSQL, while
  the CI mirror serves plain HTTP on port 5000. install-dynamic-plugins timed
  out pulling the catalog index from the bastion.
- Operator catalog. A filtered oc-mirror run against the published Red Hat
  catalog rejects it since rhdh-operator#3822 removed the default `fast`
  channel, and that catalog carries no 2.x bundles anyway. CI runs now mirror
  the RHDH CI catalog (quay.io/rhdh/iib:<stream>-v<ocp>-x86_64); its
  registry.redhat.io/rhdh relatedImages are redirected to quay.io/rhdh with
  CONTAINERS_REGISTRIES_CONF for the prepare-restricted-environment.sh call.
- Smoke plugin set. On the 2.x New Frontend System hub the sign-in page,
  sidebar, SCM integrations, and guest auth provider are dynamic plugins, so
  the homepage-only set could not render a login. Mirror the minimal digest
  pinned set and configure the NFS extensions and signInPage in the app
  configs.

The 2.x hub image and the new plugin set were verified on a live OCP cluster:
guest sign-in, sidebar, and the "Welcome back!" homepage heading all render.

Assisted-by: OpenCode
- Pick the operator catalog from the chart build type instead of the image
  registry alone: a CI chart (x.y-N-CI) mirrors the RHDH CI catalog, a
  released chart (x.y.z) keeps the published catalog. A pinned RC/GA
  verification on main (TAG_NAME and CHART_VERSION 1.9.4) was handed a `next`
  catalog for a 1.9 channel filter.
- Export the smoke plugin list as a space-separated scalar. bash cannot export
  arrays, so the `export -f` helpers saw an empty list in a child shell and
  would have mirrored every digest-pinned plugin. Fail when no package
  variable was exported.
- Document why the mirror egress rule is scoped by port and not by
  destination: the bastion address is assigned per run and reachable by DNS
  name only, so an ipBlock would break whenever the bastion is replaced.

Assisted-by: OpenCode
The chart mounts <fullname>-dynamic-plugins-registry-auth, and fullname is
<release>-<nameOverride> with nameOverride pinned to developer-hub. The job
created rhdh-dynamic-plugins-registry-auth instead, so the optional volume was
never mounted, install-dynamic-plugins ran without auth.json, and skopeo failed
the mirror pull with "authentication required". Pin nameOverride in the values
and create the secret under the chart's name.

Verified on a live OCP 4.21 cluster (LOCAL_DISCONNECTED): the init container
now pulls the catalog index and all five smoke plugins through the mirror.

Assisted-by: OpenCode
Helm: after the IDMS/ITMS apply, `oc wait machineconfigpool
--for=condition=Updated=True` returned while the MCO had not rendered the new
config yet. The nodes were drained a few seconds later, while the hub and
PostgreSQL pods were starting, so they were evicted mid-startup. The hub
recovery path then ran `oc rollout restart`, starting a second backend next to
the first against a freshly initialised database; the two raced on the knex
migration lock and the losing backend stayed up with readiness 503 forever.

- snapshot each pool's rendered config before the IDMS/ITMS and
  additionalTrustedCA changes, and wait for the MCO to render before waiting
  for convergence (poll Updated + updatedMachineCount; warn and continue when
  nothing is rolling)
- start the hub restart grace period only once the backend container has
  started, and replace the rollout restart with scale to 0, wait for the pods
  to be gone, scale to 1, so two backends never overlap

Operator: OLM v1 installs the bundle with the rhdh-operator-installer
ServiceAccount, which cannot grant RBAC it does not hold itself. The installer
ClusterRole rendered by prepare-restricted-environment.sh lags behind
config/rbac/role.yaml (pods, apiservers/ingresses, devhubplugincatalogs), so
the operator ClusterRole creation is refused and the operator ends up running
with no permissions. The half-applied release still creates the CRD, so the
CRD-only wait passed and the job failed later with a misleading message.

- require ClusterExtension Installed=True as well as the CRD, log the install
  condition at info level, and dump OLM v1 status when the Backstage CR
  produces no deployment
- TEMPORARY: grant the installer the operator's own ClusterRole rules from the
  same release branch before prepare runs, so the first install attempt
  succeeds. Remove once redhat-developer/rhdh-operator ships the fix.

Verified the RBAC workaround on a live OCP cluster: the same ClusterRole
creation is refused before the grant and succeeds after it.

Assisted-by: OpenCode
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
redhat-developer/rhdh-operator#3923 is merged, so the
prepare-restricted-environment.sh fetched from rhdh-operator main renders an
installer ClusterRole that covers the bundle permissions (pods,
apiservers/ingresses, devhubplugincatalogs). The disconnected operator job
downloads that script at runtime, so the workaround is no longer needed on
main.

release-2.1 keeps the workaround until redhat-developer#3924 merges, because that job fetches
the script from rhdh-operator release-2.1.

Assisted-by: OpenCode
@zdrapela
zdrapela force-pushed the fix/ci-disconnected-nightlies-2.1 branch from 44f3693 to 78be608 Compare October 26, 2026 12:08
@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

The operator job took the bundle filter from CHART_VERSION while the catalog tag
came from RELEASE_VERSION, so a pinned run on main (for example an RC tag) asked
oc-mirror for bundles the `next` catalog does not carry.

Both now come from the branch: RELEASE_VERSION for the catalog tag (next on
main, 2.1 on release-2.1) and common::get_release_stream for the filter, which
needs a major.minor and resolves main to the package.json stream (2.2).

Move that helper out of lib/helm.sh while at it: the operator and upgrade jobs
use it too, so it lives in lib/common.sh as common::get_release_stream.

Assisted-by: OpenCode
Drop the comments that only describe the PR history or restate the code
(merged rhdh-operator PRs, the NFS plugin rationale repeated in three files,
snapshot double-comments, a duplicated secret-mount paragraph). Keep the notes
that explain non-obvious behaviour: the guest sign-in page, nameOverride
coupling, CI vs released catalogs, the registries.conf redirect and the
additionalTrustedCA node roll.

Also log the configured MachineConfigPool timeout instead of a hardcoded 20m,
and correct the mirror egress rationale: MIRROR_REGISTRY_URL is the bastion's
public EC2 DNS name, which resolves differently from inside the cluster, so
there is no stable ipBlock to use.

Assisted-by: OpenCode
@zdrapela
zdrapela force-pushed the fix/ci-disconnected-nightlies-2.1 branch from ccb352e to f897695 Compare October 26, 2026 13:55
@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

The operator job picked the catalog tag from RELEASE_VERSION and the
--filter-versions value from common::get_release_stream. On main those are
"next" and the package.json stream (2.2), which agree only while rhdh and
rhdh-operator bump together; when one side lags, prepare keeps a fast-X.Y
channel the catalog does not carry, or oc-mirror rejects the catalog because
its default channel is filtered out.

Use filter_versions for the tag as well. Both catalog tags resolve to the same
digest today (iib:next-v4.21 == iib:2.2-v4.21), so nightlies are unchanged,
and a stream without a published catalog now fails at the skopeo check with a
readable error.

Assisted-by: OpenCode
- list jq in common.sh dependencies: the moved get_release_stream uses it
- rename mcp_all_converged to the underscore prefix the other private
  helpers use
- drop the unreachable exported -eq 0 branch in the smoke plugin export
- document the hub grace period as it behaves: it covers the backend start
  wait, then the Available wait

Assisted-by: OpenCode
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

The operator job fell back to the published Red Hat catalog whenever the image
registry was not registry.redhat.io or the chart was not a CI tag. That path
cannot work: prepare-restricted-environment.sh writes only fast-X.Y channels
into the ImageSetConfiguration, so oc-mirror rejects the published catalog
because its default fast channel is filtered out, and the catalog carries no
2.x bundles at all.

Every disconnected run is branch-aligned, and the hub image comes from the
bundle's RELATED_IMAGE_backstage, so TAG_NAME, IMAGE_* and CHART_VERSION never
reached the deployment. Use the CI catalog
(quay.io/rhdh/iib:<stream>-v<ocp>-x86_64) and its registries.conf redirect
unconditionally, and log that only the newest bundle of the branch's stream is
tested.

Assisted-by: OpenCode
Putting deploy_rhdh_operator on the left of || turns errexit off for its whole
body, so a failed envsubst, yq or oc apply fell through to the 5-minute
deployment poll and then reported a missing deployment, which pointed at OLM.

Return from deploy_rhdh_operator as soon as rendering or applying the CR fails,
and use a neutral message in the operator job (the OLM v1 dump stays as
diagnostics).

The Helm hub bounce scaled the Deployment back up even when the old pod was
still present after 180s, recreating the two-backend race it is meant to
prevent. Treat that timeout as a recovery failure.

Assisted-by: OpenCode
disconnected::snapshot_mcp_rendered_configs hid oc failures by returning an
empty success: a failed pre-change read skipped render detection, and a failed
read during detection looked like a changed config. Retry the read three times
and fail on an error or an empty answer, skip failed reads in the render loop,
and require the pre-change snapshot. Both jobs now stop when it cannot be taken.

Rename the MCP identifiers to machineconfigpool, since MCP reads as Model
Context Protocol in this repository.

Document why the IDMS and ITMS applies share one wait: a pool's spec moves to
the newest rendered config, so a config rendered during the first rollout is
still awaited.

Assisted-by: OpenCode
The smoke set was spelled out in six places: the array, the case in
_smoke_plugin_package_var, two require_vars lists and two templates. A template
entry the array did not know about rendered an empty package: and only failed at
pod runtime.

Render both templates through one helper that maps the array to the package
variables, requires them, and rejects a package: left empty by envsubst.

Assisted-by: OpenCode
- the CI mirror serves TLS on port 5000, not plain HTTP: the openshift/release
  step curls https://<mirror>/v2/ with the vault CA
- nameOverride pins the chart fullname for the registry-auth secret; the egress
  NetworkPolicy selects on pod labels
- the smoke app config is a subset of dynamic-plugins-config.yaml, not a mirror
- the Backstage CR installs the minimal NFS plugin set, not only the homepage

Assisted-by: OpenCode
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

A GA verification run (--image-registry registry.redhat.io with a released x.y.z
chart) should pair with the published Red Hat catalog, and the existing image and
chart overrides already select it. Restore that branch.

It cannot keep the version filter: the published catalog declares defaultChannel
fast, and oc-mirror rejects a channels list that leaves the default channel out,
which is why GA runs failed at mirroring before. Pass --filter-versions '*' for
the published catalog (prepare then mirrors every rhdh channel, still scoped to
the rhdh package) and keep the branch-stream filter for the CI catalog, so the
nightlies are unchanged.

Assisted-by: OpenCode
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@openshift-ci openshift-ci Bot added the lgtm label Oct 27, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit fc6478d into redhat-developer:release-2.1 Oct 27, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants