Skip to content

[release-1.10] chore(ci): replace Crunchy Postgres operator with RHEL PostgreSQL StatefulSet - #5835

Merged
openshift-merge-bot[bot] merged 1 commit into
redhat-developer:release-1.10from
zdrapela:cherry-pick-5820-release-1.10
Nov 13, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
redhat-developer:release-1.10from
zdrapela:cherry-pick-5820-release-1.10

Conversation

@zdrapela

@zdrapela zdrapela commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Manual backport of #5820 to release-1.10.

Replaces the Crunchy Postgres operator with a plain StatefulSet for the external
PostgreSQL used by showcase-rbac and showcase-runtime. The Crunchy operator is not
published for OCP 5.0, so every OCP job fails before the tests start.

Adaptations for release-1.10:

  • PostgreSQL 16 (registry.redhat.io/rhel9/postgresql-16): 1.10 supports
    PostgreSQL 14-17 and Red Hat does not ship a rhel9/postgresql-17 image.
    ssl_ecdh_curve replaces ssl_groups, which only exists in PostgreSQL 18.
  • The branch installs OpenShift Pipelines on OCP, so cluster_setup_ocp_helm,
    cluster_setup_ocp_operator and their call sites are kept and only the Crunchy
    install is removed.
  • Dropped the wait for the postgress-external-db Deployment, which no longer
    exists, and kept the external DB setup for the operator runtime tests.
  • No changes to files that do not exist on this branch (ocp-fips-helm.sh,
    runtime-deploy.ts, layer-migration-matrix.md) or to the cloud value files.
  • pr-triage content is sourced from .rulesync/commands here, so the edits were
    applied there and the generated copies regenerated.

Validation

Job OCP Result Notes
ci/prow/e2e-ocp-helm 4.x ✅ pass required check
e2e-ocp-operator-nightly 4.x ✅ pass external PostgreSQL + schema-mode
e2e-ocp-v5-0-helm-nightly 5.0 ❌ 2h timeout 1.10-on-OCP-5.0 test failure, see below

Local: shellcheck (-x -S warning), bash -n, prettier --check, eslint,
tsc, playwright test --list, rulesync generate --check.

The PostgreSQL 16 StatefulSet works. The operator nightly deployed it and
ran the schema-mode tests against it:

ℹ Deploying external PostgreSQL in namespace postgress-external-db...
✓ External PostgreSQL database configured successfully!
ℹ Schema-mode (operator): no in-cluster Postgres Service in showcase-runtime; using external PostgreSQL in postgress-external-db
ℹ Schema-mode env configured (operator): Playwright will port-forward pod/postgress-external-db-0

The v5-0-helm-nightly failure is not caused by this change. showcase runs on
its own in-cluster database there, and the smoke test failed all 11 attempts in
both showcase and showcase-rbac: the home page renders, but
waitForSelector("nav a") never matches and the browser reports Failed to fetch.
Everything else in those projects was then skipped, and the runtime suite hit the
2h pod timeout. This needs a separate look at 1.10 on OCP 5.0.

Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4113

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v5-0-helm-nightly

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Replace Crunchy PostgreSQL with a StatefulSet in release-1.10 CI

🐞 Bug fix ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Replace unavailable Crunchy operator dependencies with a TLS-enabled RHEL PostgreSQL 16
 StatefulSet for OCP jobs.
• Reuse database credentials across RBAC and runtime tests, and wait for PostgreSQL readiness before
 deployment.
• Align schema-mode tests, TLS test naming, and CI guidance with the new database.
Diagram

graph TD
  CI["OCP CI jobs"] --> Setup["Database setup"] --> Secrets["TLS and credentials"] --> DB[("PostgreSQL StatefulSet")] --> Service["Primary Service"] --> RHDH["RBAC and runtime"] --> Tests["E2E tests"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Use an externally managed PostgreSQL service
  • ➕ Avoids maintaining PostgreSQL manifests and storage in CI.
  • ➖ Requires service provisioning and network access for every job.
  • ➖ Makes isolated CI runs dependent on external infrastructure.

Recommendation: Keep the self-contained StatefulSet for ephemeral CI: it removes the unavailable operator without sacrificing external-database coverage. A managed service would add provisioning dependencies that do not fit these jobs.

Files changed (16) +343 / -295

Bug fix (2) +4 / -21
operator.shRemove Crunchy CRD expectations +2/-17

Remove Crunchy CRD expectations

• Drops the PostgresCluster CRD wait and diagnostic query. Checks only the built-in StatefulSet when the RHDH operator creates its database.

.ci/pipelines/install-methods/operator.sh

ocp-operator.shStop gating deployments on Crunchy +2/-4

Stop gating deployments on Crunchy

• Removes Crunchy CRD waits from OCP and OSD-GCP operator deployments. Retains external database setup for runtime and schema-mode tests.

.ci/pipelines/jobs/ocp-operator.sh

Tests (3) +16 / -15
schema-mode-env.shDiscover the StatefulSet for schema-mode tests +11/-10

Discover the StatefulSet for schema-mode tests

• Reads the new rhdh user secret and selects a Running PostgreSQL pod by StatefulSet label for port-forwarding.

.ci/pipelines/lib/schema-mode-env.sh

playwright.config.tsSelect the renamed PostgreSQL TLS test +4/-4

Select the renamed PostgreSQL TLS test

• Updates Playwright project matchers to include the external-postgres TLS spec instead of its Crunchy-named predecessor.

e2e-tests/playwright.config.ts

verify-tls-config-with-external-postgres.spec.tsUse provider-neutral TLS test naming +1/-1

Use provider-neutral TLS test naming

• Renames the test suite description to external PostgreSQL; its database-backed UI assertion remains unchanged.

e2e-tests/playwright/e2e/external-database/verify-tls-config-with-external-postgres.spec.ts

Documentation (8) +53 / -56
README.mdRefresh operator library function list +2/-3

Refresh operator library function list

• Replaces obsolete PostgreSQL operator helpers with the supported Pipelines and Tekton functions.

.ci/pipelines/lib/README.md

pr-triage.mdRecognize StatefulSet database failures +9/-9

Recognize StatefulSet database failures

• Updates generated triage examples and guidance to classify external PostgreSQL readiness failures.

.claude/commands/pr-triage.md

pr-triage.mdRecognize StatefulSet database failures +9/-9

Recognize StatefulSet database failures

• Updates generated triage examples, fingerprints, and investigation guidance for external PostgreSQL failures.

.cursor/commands/pr-triage.md

pr-triage.mdRecognize StatefulSet database failures +9/-9

Recognize StatefulSet database failures

• Updates generated triage examples and the DEPLOY_POSTGRES guidance to reference StatefulSet diagnostics.

.opencode/command/pr-triage.md

pr-triage.mdUpdate source triage guidance +9/-9

Update source triage guidance

• Changes the source command's PostgreSQL failure fingerprints, search examples, and recommended investigation from Crunchy to the StatefulSet.

.rulesync/commands/pr-triage.md

pgdb.yamlDescribe the new PostgreSQL resource +4/-4

Describe the new PostgreSQL resource

• Replaces Crunchy-specific catalog text and links with StatefulSet and PostgreSQL references.

catalog-entities/resources/pgdb.yaml

CI-medic-guide.mdDocument PostgreSQL CI failure diagnosis +6/-8

Document PostgreSQL CI failure diagnosis

• Describes platform-specific Pipelines setup and external PostgreSQL readiness, including image pull, storage, and TLS-secret checks.

docs/e2e-tests/CI-medic-guide.md

README.mdDocument schema-mode database discovery +5/-5

Document schema-mode database discovery

• Explains the new rhdh secret, pod selector, and TLS requirement for port-forwarded schema-mode tests.

e2e-tests/playwright/e2e/plugin-division-mode-schema/README.md

Other (3) +270 / -203
operators.shRemove Crunchy installation helpers +0/-34

Remove Crunchy installation helpers

• Deletes the Crunchy subscription, status polling, and PostgresCluster CRD wait functions while retaining other operator setup.

.ci/pipelines/lib/operators.sh

postgres.yamlDeploy TLS-enabled RHEL PostgreSQL 16 +170/-64

Deploy TLS-enabled RHEL PostgreSQL 16

• Replaces PostgresCluster with ConfigMaps, a Service, and a persistent single-replica StatefulSet. Configures TLS-only TCP authentication, an rhdh database role, and registry authentication.

.ci/pipelines/resources/postgres-db/postgres.yaml

utils.shProvision and await external PostgreSQL +100/-105

Provision and await external PostgreSQL

• Generates certificates, creates or reuses database secrets, deploys the StatefulSet, and waits for rollout before distributing connection secrets. Removes Crunchy installation and obsolete Deployment waits while preserving OpenShift Pipelines setup.

.ci/pipelines/utils.sh

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.69%. Comparing base (8f5a570) to head (35550a9).
⚠️ Report is 1 commits behind head on release-1.10.

Additional details and impacted files
@@                Coverage Diff                @@
##           release-1.10    #5835       +/-   ##
=================================================
+ Coverage         41.03%   69.69%   +28.65%     
=================================================
  Files               121      111       -10     
  Lines              2220     4715     +2495     
  Branches            563      535       -28     
=================================================
+ Hits                911     3286     +2375     
- Misses             1303     1428      +125     
+ Partials              6        1        -5     
Flag Coverage Δ
install-dynamic-plugins 92.48% <ø> (?)
rhdh 38.81% <ø> (-2.23%) ⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f5a570...35550a9. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Code Review by Qodo


Action required

Nice, no open findings for this group


Remediation recommended

Nice, no open findings for this group


Informational

Nice, no open findings for this group


@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

…tefulSet

The Crunchy Postgres operator is not available in the OCP 5.0
certified-operators catalog (crunchy-postgres-operator is published for
OCP 4.15-4.22 only), so every OCP job fails before the tests start. On top
of that, Crunchy announced that the Developer Program images our
PostgresCluster pulls will stop receiving updates on 2026-12-31.

Deploy the external PostgreSQL used by showcase-rbac as a plain
StatefulSet running registry.redhat.io/rhel9/postgresql-16 instead:

- TLS server certificate generated by the CI script (CA + cert with SANs
  for the Service DNS names), served from the same
  postgress-external-db-cluster-cert secret the RBAC values mount.
- The start hook restricts network connections to TLS and creates the
  rhdh SUPERUSER from postgress-external-db-pguser-rhdh.
- The registry.redhat.io pull secret is created from the existing
  REGISTRY_REDHAT_IO_SERVICE_ACCOUNT_DOCKERCONFIGJSON CI secret.

The Crunchy operator install and the PostgresCluster CRD waits are
removed. Also rename the external DB TLS spec and update the CI docs
accordingly.

Adaptations for release-1.10:
- Use PostgreSQL 16: 1.10 supports PostgreSQL 14-17 and Red Hat does not
  ship a rhel9/postgresql-17 image. Use ssl_ecdh_curve instead of
  ssl_groups, which only exists in PostgreSQL 18 and would prevent the
  server from starting.
- Keep the OpenShift Pipelines install in cluster_setup_ocp_helm and
  cluster_setup_ocp_operator, and their call sites, since the branch
  installs Pipelines on OCP. Only the Crunchy install is dropped, so
  operator::install_subscription and operator::install_pipelines stay in
  lib/operators.sh while the Crunchy helper and the OLM status helper it
  used are removed.
- Drop the k8s_wait::deployment wait for postgress-external-db, which no
  longer exists as a Deployment.
- The operator job runtime tests also use the external PostgreSQL, so
  keep configuring it there and keep the second call idempotent through
  the reused certificates and credentials.
- Do not add changes to files that do not exist on this branch
  (ocp-fips-helm.sh, runtime-deploy.ts, layer-migration-matrix.md) or to
  the cloud value files, which have no Crunchy reference here.
- The pr-triage content lives in .rulesync/commands here, so the edits are
  applied to that source and the generated copies are regenerated.

Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4113
Assisted-by: OpenCode
(cherry picked from commit 6e6cd04)
@zdrapela
zdrapela force-pushed the cherry-pick-5820-release-1.10 branch from 410f750 to 35550a9 Compare November 13, 2026 09:40
@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@albarbaro

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci

openshift-ci Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

@zdrapela: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-ocp-v5-0-helm-nightly 410f750 link false /test e2e-ocp-v5-0-helm-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@zdrapela

Copy link
Copy Markdown
Member Author

/retest

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@openshift-merge-bot
openshift-merge-bot Bot merged commit ce171b3 into redhat-developer:release-1.10 Nov 13, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants