Skip to content

fix(ci): override stale RELATED_IMAGE_backstage digest in the operator CSV - #5435

Open
gustavolira wants to merge 6 commits into
redhat-developer:mainfrom
gustavolira:fix/ci-operator-image-override
Open

fix(ci): override stale RELATED_IMAGE_backstage digest in the operator CSV#5435
gustavolira wants to merge 6 commits into
redhat-developer:mainfrom
gustavolira:fix/ci-operator-image-override

Conversation

@gustavolira

@gustavolira gustavolira commented Sep 17, 2026

Copy link
Copy Markdown
Member

Description

All operator-based nightlies (e2e-osd-gcp-operator-nightly, e2e-gke-operator-nightly, e2e-aks-operator-nightly) have been red for weeks with zero tests executed: the install-dynamic-plugins init container crash-loops with

InstallException: Cannot resolve ref:// reference: no plugin named
'backstage-community-plugin-catalog-backend-module-keycloak' found in included plugins

Root cause

  • The operator bundle CSV pins RELATED_IMAGE_backstage to the hub digest captured at bundle build time. The current next bundle (built 2026-08-31) still pins rhdh-hub-rhel9@sha256:a89235... — the 2026-08-04 build, which bundles install-dynamic-plugins 0.4.0. That version resolves ref:// references only after the pre-merge disabled pass, so refs pointing at plugins that the catalog index ships disabled (keycloak, scaffolder-relation-processor, rbac…) can never resolve. Fixed in 0.4.1 (rhdh-plugins#4410, 2026-08-20), which today's :next tag already carries — the same config installs cleanly against quay.io/rhdh/rhdh-hub-rhel9:next (verified locally with the exact CI ConfigMap).
  • The Backstage CR deployment.patch already overrides both containers to $IMAGE_REGISTRY/$IMAGE_REPO:$TAG_NAME, but the installed operator applies the RELATED_IMAGE_backstage env to the init container after the patch, so the stale digest wins there (pod events confirm the init container pulls the digest while backstage-backend gets :next).

Fix

After installing the operator, patch the CSV's RELATED_IMAGE_backstage to the image under test and wait for OLM to roll the operator deployment. This keeps every operator job testing the same image the helm jobs test, on all branches (next and -v X.Y installs alike).

The jq patch and the poll condition were validated against the real rhdh-operator-bundle:next CSV.

Follow-ups (out of scope here)

  • The next operator bundle pinning a 4-week-old hub digest is a productization/midstream issue worth its own report.
  • The operator ignoring deployment.patch for init containers while honoring it for containers looks like an rhdh-operator bug.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.04%. Comparing base (b4eab29) to head (5cfc575).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff             @@
##             main    #5435       +/-   ##
===========================================
- Coverage   74.21%   64.04%   -10.17%     
===========================================
  Files          69       57       -12     
  Lines         799      573      -226     
  Branches      125       99       -26     
===========================================
- Hits          593      367      -226     
  Misses        205      205               
  Partials        1        1
Flag Coverage Δ
rhdh 64.04% <ø> (-10.17%) ⬇️
Components Coverage Δ
Backend plugins ∅ <ø> (∅)
Backend app 71.91% <ø> (ø)
Frontend app 60.50% <ø> (ø)
Plugin utils ∅ <ø> (∅)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4eab29...5cfc575. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@gustavolira

gustavolira commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

/test ?

@gustavolira

gustavolira commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@gustavolira

gustavolira commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

/test ?

@gustavolira

gustavolira commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@gustavolira

gustavolira commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

/retest

@gustavolira

gustavolira commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

1 similar comment
@gustavolira

gustavolira commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@gustavolira

gustavolira commented Sep 18, 2026 •
edited by openshift-ci Bot
Loading

Copy link
Copy Markdown
Member Author

The osd-gcp validation run (build 2100932305752166400) revealed a gap: on OCP/OSD ≥ 4.18 the installer auto-detects OLM v1 and installs via ClusterExtension, so there is no CSV and the override takes its skip guard — the stale digest stayed in effect and the ref:// crash-loop persisted (RHDH installed via OLM v1 ClusterExtension; skipping CSV backstage image override in the step log).

Pushed a follow-up commit that passes --olm-version v0 to install-rhdh-catalog-source.sh, keeping the standard operator jobs on the CSV-based path the override patches. OLM v1 coverage keeps its dedicated flows (disconnected jobs / RHIDP-14647). The osd-gcp job needs a re-run to validate end to end.

@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@gustavolira

gustavolira commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@gustavolira
gustavolira force-pushed the fix/ci-operator-image-override branch from 21dd1b2 to 5cfc575 Compare September 18, 2026 18:25
@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@gustavolira

gustavolira commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

gustavolira and others added 5 commits September 18, 2026 17:40
@gustavolira @claude
fix(ci): override stale RELATED_IMAGE_backstage digest in the operato…
9ec4301 
…r CSV

The RHDH operator bundle CSV pins RELATED_IMAGE_backstage to the hub image
digest captured at bundle build time. The current next bundle (built
2026-08-31) still pins the 2026-08-04 hub digest, whose bundled
install-dynamic-plugins 0.4.0 resolves ref:// references only after the
pre-merge disabled pass and crash-loops with:

  InstallException: Cannot resolve ref:// reference: no plugin named
  'backstage-community-plugin-catalog-backend-module-keycloak' found in
  included plugins

The Backstage CR deployment.patch already overrides both containers to
$IMAGE_REGISTRY/$IMAGE_REPO:$TAG_NAME, but the installed operator applies
the RELATED_IMAGE_backstage env to the install-dynamic-plugins init
container after patching, so the stale digest wins there. This has kept
showcase-operator deployments red on OSD-GCP, GKE and AKS since early
August (zero tests executed).

Patch the CSV env right after operator installation so OLM propagates the
image under test to the operator deployment, and wait for the rollout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gustavolira @claude
fix(ci): guard command substitutions in the CSV override under errexit
13bf173 
openshift-ci-tests.sh runs with set -o errexit, so an empty grep result or
a missing deployment in the assignment pipelines would abort the whole job
instead of reaching the intended skip guards.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gustavolira @claude
fix(ci): harden the CSV backstage image override against races and OL…
c390bb5 
…M v1

Review follow-ups:
- install-rhdh-catalog-source.sh returns right after creating the
  Subscription, so poll for the CSV to appear before patching instead of
  silently skipping on the common race.
- Skip immediately (with a log) when RHDH was installed via an OLM v1
  ClusterExtension, which has no CSV to patch.
- Fail instead of skipping when the CSV exists but cannot be read, so an
  API error is not masked as a legitimate skip.
- Guard the rollout wait with an explicit log::error, and jq exit status
  can no longer abort the script via errexit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gustavolira @claude
fix(ci): force the OLM v0 install path so the CSV override can apply
891b43e 
The osd-gcp run on this PR showed install-rhdh-catalog-source.sh
auto-detecting OLM v1 on OCP/OSD >= 4.18 and installing via
ClusterExtension — no CSV exists there, override_operator_backstage_image
takes its skip guard, and the stale RELATED_IMAGE_backstage digest stays
in effect (init container still crash-loops on ref:// resolution).

Pass --olm-version v0 so the standard operator jobs keep the CSV-based
install that the override patches. OLM v1 coverage keeps its dedicated
flows (disconnected jobs, RHIDP-14647).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gustavolira @claude
fix(ci): wait for the image-registry default route before the operato…
02510b0 
…r install

Same RHDHBUGS-3758 hardening as on release-1.9: the install script
patches the registry and reads default-route before OpenShift creates
it, intermittently failing the install. Expose and wait up front;
warn-only on timeout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gustavolira
gustavolira force-pushed the fix/ci-operator-image-override branch from 5cfc575 to 02510b0 Compare September 18, 2026 20:40
@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@gustavolira @claude
fix(ci): probe the registry through the default route before installing
b93ca4e 
The Route object existing is not enough: right after creation the
router/registry data path can still refuse blob uploads (EOF on the
operator bundle push, RHDHBUGS-3759). Poll the registry API through the
route until it answers 200/401 before handing off to the install script.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gustavolira

gustavolira commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

End-to-end validation on e2e-osd-gcp-operator-nightly (build 2101025969811755008) — the full chain worked:

  • Image registry default-route is ready → operator installed via OLM v0 on attempt 1
  • RHDH CSV present → Overriding RELATED_IMAGE_backstage in ...rhdh-operator.v2.0.0 with 'quay.io/rhdh-community/rhdh:next' → Operator deployment picked up backstage image override → rolled out
  • The Cannot resolve ref:// InstallException is gone; install-dynamic-plugins completed (Exit 0) against rhdh-community/rhdh:next (the stale Aug-4 digest appears nowhere)
  • showcase-operator: 0 failures · showcase-operator-rbac: 0 failures

The job's red comes solely from showcase-runtime's deploy setup (backend never passed readiness, pre-existing on this env and unrelated to this PR — no pod logs were collected on that path, worth a follow-up). Ready for review.

@sonarqubecloud

sonarqubecloud Bot commented Sep 18, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@gustavolira

gustavolira commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown

@gustavolira: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-osd-gcp-operator-nightly b93ca4e link false /test e2e-osd-gcp-operator-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HusneShabbir HusneShabbir Awaiting requested review from HusneShabbir
@zdrapela zdrapela Awaiting requested review from zdrapela

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gustavolira