Repository navigation
[release-2.1] fix(ci): unblock disconnected OCP nightlies - #5710
openshift-merge-bot[bot] merged 16 commits into
Conversation
PR Summary by QodoUnblock disconnected OCP Helm and Operator nightlies
AI Description
Diagram
High-Level Assessment
Files changed (11)
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release-2.1 #5710 +/- ##
===============================================
- Coverage 75.70% 69.00% -6.70%
===============================================
Files 65 58 -7
Lines 708 555 -153
Branches 100 83 -17
===============================================
- Hits 536 383 -153
Misses 171 171
Partials 1 1
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Code Review by Qodo
View medium (1)🟠 **Medium** 1. Backend pods can reach unrelated services
|
6aba6d0 to
44f3693
Compare
|
/test e2e-ocp-disconnected-helm-nightly |
The Helm and Operator disconnected nightlies have not passed for over five weeks. Three independent gaps kept them red: - Backend egress to the mirror registry. The chart and the Operator both default-deny backend egress except DNS, HTTPS, Redis, and PostgreSQL, while the CI mirror serves plain HTTP on port 5000. install-dynamic-plugins timed out pulling the catalog index from the bastion. - Operator catalog. A filtered oc-mirror run against the published Red Hat catalog rejects it since rhdh-operator#3822 removed the default `fast` channel, and that catalog carries no 2.x bundles anyway. CI runs now mirror the RHDH CI catalog (quay.io/rhdh/iib:<stream>-v<ocp>-x86_64); its registry.redhat.io/rhdh relatedImages are redirected to quay.io/rhdh with CONTAINERS_REGISTRIES_CONF for the prepare-restricted-environment.sh call. - Smoke plugin set. On the 2.x New Frontend System hub the sign-in page, sidebar, SCM integrations, and guest auth provider are dynamic plugins, so the homepage-only set could not render a login. Mirror the minimal digest pinned set and configure the NFS extensions and signInPage in the app configs. The 2.x hub image and the new plugin set were verified on a live OCP cluster: guest sign-in, sidebar, and the "Welcome back!" homepage heading all render. Assisted-by: OpenCode
- Pick the operator catalog from the chart build type instead of the image registry alone: a CI chart (x.y-N-CI) mirrors the RHDH CI catalog, a released chart (x.y.z) keeps the published catalog. A pinned RC/GA verification on main (TAG_NAME and CHART_VERSION 1.9.4) was handed a `next` catalog for a 1.9 channel filter. - Export the smoke plugin list as a space-separated scalar. bash cannot export arrays, so the `export -f` helpers saw an empty list in a child shell and would have mirrored every digest-pinned plugin. Fail when no package variable was exported. - Document why the mirror egress rule is scoped by port and not by destination: the bastion address is assigned per run and reachable by DNS name only, so an ipBlock would break whenever the bastion is replaced. Assisted-by: OpenCode
The chart mounts <fullname>-dynamic-plugins-registry-auth, and fullname is <release>-<nameOverride> with nameOverride pinned to developer-hub. The job created rhdh-dynamic-plugins-registry-auth instead, so the optional volume was never mounted, install-dynamic-plugins ran without auth.json, and skopeo failed the mirror pull with "authentication required". Pin nameOverride in the values and create the secret under the chart's name. Verified on a live OCP 4.21 cluster (LOCAL_DISCONNECTED): the init container now pulls the catalog index and all five smoke plugins through the mirror. Assisted-by: OpenCode
Helm: after the IDMS/ITMS apply, `oc wait machineconfigpool --for=condition=Updated=True` returned while the MCO had not rendered the new config yet. The nodes were drained a few seconds later, while the hub and PostgreSQL pods were starting, so they were evicted mid-startup. The hub recovery path then ran `oc rollout restart`, starting a second backend next to the first against a freshly initialised database; the two raced on the knex migration lock and the losing backend stayed up with readiness 503 forever. - snapshot each pool's rendered config before the IDMS/ITMS and additionalTrustedCA changes, and wait for the MCO to render before waiting for convergence (poll Updated + updatedMachineCount; warn and continue when nothing is rolling) - start the hub restart grace period only once the backend container has started, and replace the rollout restart with scale to 0, wait for the pods to be gone, scale to 1, so two backends never overlap Operator: OLM v1 installs the bundle with the rhdh-operator-installer ServiceAccount, which cannot grant RBAC it does not hold itself. The installer ClusterRole rendered by prepare-restricted-environment.sh lags behind config/rbac/role.yaml (pods, apiservers/ingresses, devhubplugincatalogs), so the operator ClusterRole creation is refused and the operator ends up running with no permissions. The half-applied release still creates the CRD, so the CRD-only wait passed and the job failed later with a misleading message. - require ClusterExtension Installed=True as well as the CRD, log the install condition at info level, and dump OLM v1 status when the Backstage CR produces no deployment - TEMPORARY: grant the installer the operator's own ClusterRole rules from the same release branch before prepare runs, so the first install attempt succeeds. Remove once redhat-developer/rhdh-operator ships the fix. Verified the RBAC workaround on a live OCP cluster: the same ClusterRole creation is refused before the grant and succeeds after it. Assisted-by: OpenCode
Point the temporary OLM v1 installer workaround at redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so the patch is easy to find and remove once the fix is released. Assisted-by: OpenCode
redhat-developer/rhdh-operator#3923 is merged, so the prepare-restricted-environment.sh fetched from rhdh-operator main renders an installer ClusterRole that covers the bundle permissions (pods, apiservers/ingresses, devhubplugincatalogs). The disconnected operator job downloads that script at runtime, so the workaround is no longer needed on main. release-2.1 keeps the workaround until redhat-developer#3924 merges, because that job fetches the script from rhdh-operator release-2.1. Assisted-by: OpenCode
44f3693 to
78be608
Compare
The operator job took the bundle filter from CHART_VERSION while the catalog tag came from RELEASE_VERSION, so a pinned run on main (for example an RC tag) asked oc-mirror for bundles the `next` catalog does not carry. Both now come from the branch: RELEASE_VERSION for the catalog tag (next on main, 2.1 on release-2.1) and common::get_release_stream for the filter, which needs a major.minor and resolves main to the package.json stream (2.2). Move that helper out of lib/helm.sh while at it: the operator and upgrade jobs use it too, so it lives in lib/common.sh as common::get_release_stream. Assisted-by: OpenCode
Drop the comments that only describe the PR history or restate the code (merged rhdh-operator PRs, the NFS plugin rationale repeated in three files, snapshot double-comments, a duplicated secret-mount paragraph). Keep the notes that explain non-obvious behaviour: the guest sign-in page, nameOverride coupling, CI vs released catalogs, the registries.conf redirect and the additionalTrustedCA node roll. Also log the configured MachineConfigPool timeout instead of a hardcoded 20m, and correct the mirror egress rationale: MIRROR_REGISTRY_URL is the bastion's public EC2 DNS name, which resolves differently from inside the cluster, so there is no stable ipBlock to use. Assisted-by: OpenCode
ccb352e to
f897695
Compare
The operator job picked the catalog tag from RELEASE_VERSION and the --filter-versions value from common::get_release_stream. On main those are "next" and the package.json stream (2.2), which agree only while rhdh and rhdh-operator bump together; when one side lags, prepare keeps a fast-X.Y channel the catalog does not carry, or oc-mirror rejects the catalog because its default channel is filtered out. Use filter_versions for the tag as well. Both catalog tags resolve to the same digest today (iib:next-v4.21 == iib:2.2-v4.21), so nightlies are unchanged, and a stream without a published catalog now fails at the skopeo check with a readable error. Assisted-by: OpenCode
- list jq in common.sh dependencies: the moved get_release_stream uses it - rename mcp_all_converged to the underscore prefix the other private helpers use - drop the unreachable exported -eq 0 branch in the smoke plugin export - document the hub grace period as it behaves: it covers the backend start wait, then the Available wait Assisted-by: OpenCode
|
/test e2e-ocp-disconnected-helm-nightly |
The operator job fell back to the published Red Hat catalog whenever the image registry was not registry.redhat.io or the chart was not a CI tag. That path cannot work: prepare-restricted-environment.sh writes only fast-X.Y channels into the ImageSetConfiguration, so oc-mirror rejects the published catalog because its default fast channel is filtered out, and the catalog carries no 2.x bundles at all. Every disconnected run is branch-aligned, and the hub image comes from the bundle's RELATED_IMAGE_backstage, so TAG_NAME, IMAGE_* and CHART_VERSION never reached the deployment. Use the CI catalog (quay.io/rhdh/iib:<stream>-v<ocp>-x86_64) and its registries.conf redirect unconditionally, and log that only the newest bundle of the branch's stream is tested. Assisted-by: OpenCode
Putting deploy_rhdh_operator on the left of || turns errexit off for its whole body, so a failed envsubst, yq or oc apply fell through to the 5-minute deployment poll and then reported a missing deployment, which pointed at OLM. Return from deploy_rhdh_operator as soon as rendering or applying the CR fails, and use a neutral message in the operator job (the OLM v1 dump stays as diagnostics). The Helm hub bounce scaled the Deployment back up even when the old pod was still present after 180s, recreating the two-backend race it is meant to prevent. Treat that timeout as a recovery failure. Assisted-by: OpenCode
disconnected::snapshot_mcp_rendered_configs hid oc failures by returning an empty success: a failed pre-change read skipped render detection, and a failed read during detection looked like a changed config. Retry the read three times and fail on an error or an empty answer, skip failed reads in the render loop, and require the pre-change snapshot. Both jobs now stop when it cannot be taken. Rename the MCP identifiers to machineconfigpool, since MCP reads as Model Context Protocol in this repository. Document why the IDMS and ITMS applies share one wait: a pool's spec moves to the newest rendered config, so a config rendered during the first rollout is still awaited. Assisted-by: OpenCode
The smoke set was spelled out in six places: the array, the case in _smoke_plugin_package_var, two require_vars lists and two templates. A template entry the array did not know about rendered an empty package: and only failed at pod runtime. Render both templates through one helper that maps the array to the package variables, requires them, and rejects a package: left empty by envsubst. Assisted-by: OpenCode
- the CI mirror serves TLS on port 5000, not plain HTTP: the openshift/release step curls https://<mirror>/v2/ with the vault CA - nameOverride pins the chart fullname for the registry-auth secret; the egress NetworkPolicy selects on pod labels - the smoke app config is a subset of dynamic-plugins-config.yaml, not a mirror - the Backstage CR installs the minimal NFS plugin set, not only the homepage Assisted-by: OpenCode
|
/test e2e-ocp-disconnected-helm-nightly |
A GA verification run (--image-registry registry.redhat.io with a released x.y.z chart) should pair with the published Red Hat catalog, and the existing image and chart overrides already select it. Restore that branch. It cannot keep the version filter: the published catalog declares defaultChannel fast, and oc-mirror rejects a channels list that leaves the default channel out, which is why GA runs failed at mirroring before. Pass --filter-versions '*' for the published catalog (prepare then mirrors every rhdh channel, still scoped to the rhdh package) and keep the branch-stream filter for the CI catalog, so the nightlies are unchanged. Assisted-by: OpenCode
|
/test e2e-ocp-disconnected-helm-nightly |
|
|
The container image build workflow finished with status: |
fc6478d
into
redhat-developer:release-2.1



Cherry-pick of #5709 to release-2.1, rebased on the current
release-2.1. The disconnected job/lib/value/resource files are identical between the two branches.Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4056
The disconnected OCP Helm and Operator nightlies have not passed for over five weeks. These gaps were blocking them:
disconnected::allow_mirror_registry_egressapplies a targeted rule for the mirror's port (Helm + Operator pod selectors); skipped on 443 (LOCAL_DISCONNECTED).fastchannel filtered out), and it has no 2.x bundles. CI runs now usequay.io/rhdh/iib:2.1-v<ocp>-x86_64with theregistry.redhat.io/rhdhrelatedImages redirected toquay.io/rhdhviaCONTAINERS_REGISTRIES_CONF. The tag follows the samecommon::get_release_streamvalue passed to--filter-versions 2.1, so the catalog and itsfast-2.1channel cannot disagree. CI runs mirror the newest CI bundle of the branch's stream: the hub image comes from the bundle, so pinnedTAG_NAME,IMAGE_*andCHART_VERSIONsettings do not apply to them. A GA run (registry.redhat.iowith a releasedx.y.zchart) uses the published catalog without a version filter, because filtering would drop its defaultfastchannel.app.extensionsconfig andsignInPage: guest.<fullname>-dynamic-plugins-registry-auth; the job created a differently named secret, soinstall-dynamic-pluginshad noauth.jsonand the mirror pull failed with "authentication required".oc rollout restart), which raced on the knex migration lock and left a backend at readiness 503 forever. The bounce is now scale-to-zero then scale-to-one, the wait gives a pod that is still extracting plugins time to start its backend first, and the bounce refuses to scale up while an old pod remains.ClusterExtension Installed=Trueas well as the CRD, logs the install condition, and dumps OLM v1 status when deploying the Backstage CR fails.get_release_streammoved fromhelm::get_chart_streamtocommon::get_release_stream, since the operator and upgrade jobs use it too.The OLM v1 installer ClusterRole in
prepare-restricted-environment.shwas missing permissions (pods,apiservers/ingresses,devhubplugincatalogs). It is fixed on rhdh-operatormain(#3923) and onrelease-2.1(#3927). This branch downloads the script from rhdh-operatorrelease-2.1, so it now installs with the fixed installer role; the temporary verification workaround has been removed.The disconnected jobs exist only on
mainandrelease-2.1, and both now target 2.x hubs: the smoke plugin set is the minimal New Frontend System set and the operator stream follows the branch. Verification of 1.x releases is not covered by these jobs.Verified on a live OCP 4.21 cluster with the 2.x hub image: guest sign-in, sidebar and the
Welcome back!homepage heading render; plugin list/package export exercised against the real:2.1catalog.The nightly jobs on this branch additionally need the
upi-installerbase_image fix in openshift/release (tracked separately).Ref: #5709