Skip to content

[release-2.1] chore(ci): replace Crunchy Postgres operator with RHEL PostgreSQL StatefulSet - #5834

Merged
openshift-merge-bot[bot] merged 1 commit into
redhat-developer:release-2.1from
zdrapela:cherry-pick-5820-release-2.1
Nov 13, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
redhat-developer:release-2.1from
zdrapela:cherry-pick-5820-release-2.1

Conversation

@zdrapela

@zdrapela zdrapela commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Manual backport of #5820 to release-2.1.

Replaces the Crunchy Postgres operator with a plain StatefulSet running
registry.redhat.io/rhel9/postgresql-18 for the external PostgreSQL used by
showcase-rbac. The Crunchy operator is not published for OCP 5.0, so every OCP
job fails before the tests start.

Adaptations for release-2.1:

  • No ocp-fips-helm.sh change, the job does not exist on this branch.
  • Kept this branch's Playwright projects and test ignores, only the external DB
    spec was renamed.
  • Kept this branch's layer-migration-matrix rows, only row 25 was renamed.
  • pr-triage content is sourced from .rulesync/commands here, so the edits were
    applied there and the generated copies regenerated.

Validation

Results below are from commit 23fd2a381, same tree as HEAD: the commit was
re-created to re-run a flaky markdown-link-check (transient GitHub 502 on an
untouched link).

Job OCP Result Notes
ci/prow/e2e-ocp-helm 4.x ✅ pass required check
e2e-ocp-operator-nightly 4.x ❌ 2h timeout pre-existing, see below
e2e-ocp-v5-0-helm-nightly 5.0 ❌ 2h timeout known test failures, see below

Local: shellcheck (-x -S warning), bash -n, oxfmt --check, oxlint, tsc,
playwright test --list, rulesync generate --check.

External PostgreSQL works on both OCP versions. The StatefulSet deployed in
both nightly runs (✓ External PostgreSQL database configured successfully!,
~30s), and showcase-rbac - the only suite using the external DB - passed on
OCP 5.0 (16 passed).

The two nightly failures are not caused by this change:

Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4113

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v5-0-helm-nightly

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Replace Crunchy PostgreSQL with a StatefulSet for OCP CI

🐞 Bug fix ⚙️ Configuration changes 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Replace the unavailable Crunchy operator with RHEL PostgreSQL so OCP jobs can reach the tests.
• Provision TLS and credentials for showcase-rbac, and wait for the database before deployment.
• Align schema-mode discovery, test naming, and CI troubleshooting guidance with the new database.
Diagram

graph TD
  Jobs["OCP jobs"] --> Provision["DB provisioning"] --> Secrets["TLS and credentials"] --> Database[("PostgreSQL StatefulSet")] --> RBAC["Showcase RBAC"] --> Test["TLS connection test"]
  Manifest["Kubernetes resources"] --> Database
  Database --> Schema["Schema-mode tests"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Use another supported PostgreSQL operator
  • ➕ Retains operator-managed database lifecycle and backups.
  • ➖ Adds a catalog dependency and setup time to CI; availability on OCP 5.0 would still need verification.
2. Use a hosted PostgreSQL service
  • ➕ Avoids deploying a database inside each test cluster.
  • ➖ Adds external networking, credentials, cost, and cleanup dependencies to CI.

Recommendation: The single-instance StatefulSet is the better fit for this ephemeral CI database: it removes the unavailable operator dependency while preserving the existing service and certificate contracts. Review should focus on image startup behavior, TLS enforcement, and secret reuse across deployments.

Files changed (25) +349 / -346

Bug fix (8) +113 / -148
operator.shRemove Crunchy-specific operator deployment checks +2/-17

Remove Crunchy-specific operator deployment checks

• Removes the PostgresCluster CRD wait and debug query. The RHDH operator deployment now waits only for its database StatefulSet.

.ci/pipelines/install-methods/operator.sh

ocp-localization.shSkip obsolete OCP Helm cluster setup +0/-1

Skip obsolete OCP Helm cluster setup

• Removes the cluster setup call that installed Crunchy before the localization deployment.

.ci/pipelines/jobs/ocp-localization.sh

ocp-nightly.shSkip Crunchy setup in nightly jobs +0/-2

Skip Crunchy setup in nightly jobs

• Removes the obsolete OCP Helm cluster setup call before nightly deployments.

.ci/pipelines/jobs/ocp-nightly.sh

ocp-operator.shRemove Crunchy prerequisites from operator jobs +0/-4

Remove Crunchy prerequisites from operator jobs

• Drops PostgresCluster CRD waits from RBAC deployments and the obsolete OCP operator cluster setup call.

.ci/pipelines/jobs/ocp-operator.sh

ocp-pull.shSkip Crunchy setup in OCP pull jobs +0/-1

Skip Crunchy setup in OCP pull jobs

• Removes the OCP Helm cluster setup call before PR deployments.

.ci/pipelines/jobs/ocp-pull.sh

upgrade.shSkip Crunchy setup during OCP Helm upgrades +0/-2

Skip Crunchy setup during OCP Helm upgrades

• Removes the obsolete cluster setup call from the upgrade workflow.

.ci/pipelines/jobs/upgrade.sh

schema-mode-env.shDiscover the StatefulSet database for schema-mode tests +11/-10

Discover the StatefulSet database for schema-mode tests

• Uses the external database's service, pod label, and rhdh user secret when no runtime-namespace database is present.

.ci/pipelines/lib/schema-mode-env.sh

utils.shProvision and wait for external PostgreSQL directly +100/-111

Provision and wait for external PostgreSQL directly

• Generates or reuses TLS and credential secrets, deploys the StatefulSet, waits for rollout, and supplies connection secrets to showcase-rbac. Removes Crunchy installation shims and cluster setup hooks.

.ci/pipelines/utils.sh

Refactor (1) +0 / -63
operators.shRemove Crunchy installation and OLM subscription helpers +0/-63

Remove Crunchy installation and OLM subscription helpers

• Deletes the Crunchy installer and the subscription and status helpers used to install it.

.ci/pipelines/lib/operators.sh

Tests (2) +5 / -5
playwright.config.tsRetarget Playwright projects to the renamed TLS spec +4/-4

Retarget Playwright projects to the renamed TLS spec

• Updates the existing project matches and ignores to use the external PostgreSQL spec name.

e2e-tests/playwright.config.ts

verify-tls-config-with-external-postgres.spec.tsRename the external database TLS test +1/-1

Rename the external database TLS test

• Renames the Crunchy-specific spec and suite description to refer to external PostgreSQL; test behavior is unchanged.

e2e-tests/playwright/e2e/external-database/verify-tls-config-with-external-postgres.spec.ts

Documentation (12) +59 / -65
README.mdRefresh operator helper documentation +1/-3

Refresh operator helper documentation

• Lists the remaining Tekton and OLM helpers instead of removed PostgreSQL subscription helpers.

.ci/pipelines/lib/README.md

diff-values_showcase-rbac_AKS.yamlClarify AKS database TLS override +1/-1

Clarify AKS database TLS override

• Updates the comment explaining why the AKS configuration removes external-database TLS settings.

.ci/pipelines/value_files/diff-values_showcase-rbac_AKS.yaml

diff-values_showcase-rbac_EKS.yamlClarify EKS database TLS override +1/-1

Clarify EKS database TLS override

• Updates the comment explaining why the EKS configuration removes external-database TLS settings.

.ci/pipelines/value_files/diff-values_showcase-rbac_EKS.yaml

diff-values_showcase-rbac_GKE.yamlClarify GKE database TLS override +1/-1

Clarify GKE database TLS override

• Updates the comment explaining why the GKE configuration removes external-database TLS settings.

.ci/pipelines/value_files/diff-values_showcase-rbac_GKE.yaml

pr-triage.mdRecognize StatefulSet PostgreSQL failures in Claude triage +9/-9

Recognize StatefulSet PostgreSQL failures in Claude triage

• Replaces Crunchy failure examples, fingerprints, and remediation guidance with external PostgreSQL readiness failures.

.claude/commands/pr-triage.md

pr-triage.mdRecognize StatefulSet PostgreSQL failures in Cursor triage +9/-9

Recognize StatefulSet PostgreSQL failures in Cursor triage

• Updates generated triage examples and guidance to classify external PostgreSQL readiness failures.

.cursor/commands/pr-triage.md

pr-triage.mdRecognize StatefulSet PostgreSQL failures in OpenCode triage +9/-9

Recognize StatefulSet PostgreSQL failures in OpenCode triage

• Updates generated triage examples and guidance to classify external PostgreSQL readiness failures.

.opencode/commands/pr-triage.md

pr-triage.mdUpdate source triage guidance for external PostgreSQL +9/-9

Update source triage guidance for external PostgreSQL

• Changes failure patterns, sample fingerprints, Jira keywords, and remediation guidance in the source triage command.

.rulesync/commands/pr-triage.md

pgdb.yamlDescribe the showcase PostgreSQL StatefulSet +4/-4

Describe the showcase PostgreSQL StatefulSet

• Replaces Crunchy-specific resource text and links with PostgreSQL and RHEL container references.

catalog-entities/resources/pgdb.yaml

CI-medic-guide.mdDocument the revised CI setup and failure signals +9/-13

Document the revised CI setup and failure signals

• Removes Crunchy setup troubleshooting and documents external PostgreSQL StatefulSet readiness during deployment. Clarifies that operator runtime-schema tests use the operator's runtime database.

docs/e2e-tests/CI-medic-guide.md

layer-migration-matrix.mdRename the external database matrix entry +1/-1

Rename the external database matrix entry

• Changes row 25 from Crunchy PostgreSQL to RHEL PostgreSQL without changing other migration rows.

docs/e2e-tests/layer-migration-matrix.md

README.mdDocument schema-mode StatefulSet discovery +5/-5

Document schema-mode StatefulSet discovery

• Describes the new user secret, pod selector, and TLS requirement for external-database port forwarding.

e2e-tests/playwright/e2e/plugin-division-mode-schema/README.md

Other (2) +172 / -65
postgres.yamlDefine a TLS-enabled RHEL PostgreSQL StatefulSet +171/-64

Define a TLS-enabled RHEL PostgreSQL StatefulSet

• Replaces the Crunchy PostgresCluster with ConfigMaps, a Service, and a single-replica StatefulSet. The manifest configures TLS-only network authentication, startup role creation, registry pull access, and persistent storage.

.ci/pipelines/resources/postgres-db/postgres.yaml

runtime-deploy.tsAlign placeholder database credentials with rhdh +1/-1

Align placeholder database credentials with rhdh

• Changes the placeholder postgres-cred username from janus-idp to rhdh.

e2e-tests/playwright/utils/runtime-deploy.ts

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.38%. Comparing base (39e8d31) to head (3013621).
⚠️ Report is 1 commits behind head on release-2.1.

Additional details and impacted files
@@               Coverage Diff               @@
##           release-2.1    #5834      +/-   ##
===============================================
- Coverage        76.02%   69.38%   -6.65%     
===============================================
  Files               64       57       -7     
  Lines              705      552     -153     
  Branches           100       83      -17     
===============================================
- Hits               536      383     -153     
  Misses             168      168              
  Partials             1        1              
Flag Coverage Δ
rhdh 69.38% <ø> (-6.65%) ⬇️
Components Coverage Δ
Backend plugins ∅ <ø> (∅)
Backend app 79.91% <ø> (ø)
Frontend app 61.03% <ø> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39e8d31...3013621. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Code Review by Qodo


View low (2)
Informational
1. Failed DB setup leaves the CI CA key on disk 🐞 Bug
Description
configure_external_postgres_db creates cert_dir with mktemp -d and fills it with ca.key,
tls.key and the CSR, but rm -rf "${cert_dir}" runs only on the success path. Every early `return
1 (failed cert generation, secret creation, oc apply`, rollout timeout, extract, or a password
read failure) skips the cleanup, so the CA signing key and server private key stay in the runner's
temp directory for the rest of the job.
Code

.ci/pipelines/utils.sh[R226-227]

+  local cert_dir
+  cert_dir=$(mktemp -d)
Relevance

● Weak

Same finding was rejected on the original PR #5820 (merged 2026-11-12) for the same utils.sh
cert_dir cleanup-on-failure issue.

PR-#5820

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The cleanup at line 299 comes after all the return 1 branches between lines 230 and 297, so none
of the failure paths remove the temp directory. The CA key is also never needed after signing, yet
it stays in the directory until the end.

.ci/pipelines/utils.sh[216-312]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`configure_external_postgres_db` deletes its temporary certificate directory (which holds `ca.key` and `tls.key`) only when it succeeds. Every failure path returns early and leaves the directory behind.

## Fix Focus Areas
- .ci/pipelines/utils.sh[226-299]

## Recommended Fix
Right after `cert_dir=$(mktemp -d)`, add `trap 'rm -rf "${cert_dir}"' RETURN`, or move the logic into a helper and run `rm -rf` before each `return`. Also check that `mktemp -d` succeeded. You can delete `ca.key`, `ca.srl` and `tls.csr` as soon as `generate_external_postgres_certs` finishes.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

2. Release-branch CI tracks a floating database image 🐞 Bug
Description
The StatefulSet's init and main containers both use registry.redhat.io/rhel9/postgresql-18:latest,
while the Crunchy manifest it replaces pinned crunchy-postgres:ubi9-18.1-2547. A new image push (a
18.x minor bump, changed start hooks, a new check-container or pg_hba handling) silently changes
the showcase-rbac DB on release-2.1, and old CI runs can no longer be reproduced.
Code

.ci/pipelines/resources/postgres-db/postgres.yaml[110]

+          image: registry.redhat.io/rhel9/postgresql-18:latest
Relevance

● Weak

Same :latest-vs-pinned-tag suggestion on this exact StatefulSet was rejected in the upstream PR,
which merged with :latest anyway.

PR-#5820

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Both containers reference :latest (lines 85 and 110). The removed manifest pinned a specific build
tag, so this backport loses that pinning. The setup also relies on image internals:
/usr/libexec/check-container, the postgresql-start sourcing and the postgresql-cfg/*.conf
merge.

.ci/pipelines/resources/postgres-db/postgres.yaml[81-160]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The external PostgreSQL StatefulSet uses `registry.redhat.io/rhel9/postgresql-18:latest` in both the init container and the main container. This makes release-branch CI depend on whatever image is pushed next.

## Fix Focus Areas
- .ci/pipelines/resources/postgres-db/postgres.yaml[85-85]
- .ci/pipelines/resources/postgres-db/postgres.yaml[110-110]

## Recommended Fix
Replace `:latest` with a specific versioned tag (e.g. `postgresql-18:1-<build>`) or an `@sha256:` digest. Use the same reference for both containers.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@openshift-ci

openshift-ci Bot commented Nov 12, 2026

Copy link
Copy Markdown

@zdrapela: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-ocp-operator-nightly 5350445 link false /test e2e-ocp-operator-nightly
ci/prow/e2e-ocp-v5-0-helm-nightly 5350445 link false /test e2e-ocp-v5-0-helm-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@zdrapela
zdrapela force-pushed the cherry-pick-5820-release-2.1 branch from 5350445 to 23fd2a3 Compare November 13, 2026 09:40
@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

…tefulSet

The Crunchy Postgres operator is not available in the OCP 5.0
certified-operators catalog (crunchy-postgres-operator is published for
OCP 4.15-4.22 only), so every OCP job fails before the tests start. On top
of that, Crunchy announced that the Developer Program images our
PostgresCluster pulls will stop receiving updates on 2026-12-31.

Deploy the external PostgreSQL used by showcase-rbac as a plain
StatefulSet running registry.redhat.io/rhel9/postgresql-18 instead:

- TLS server certificate generated by the CI script (CA + cert with SANs
  for the Service DNS names), served from the same
  postgress-external-db-cluster-cert secret the RBAC values mount.
- The start hook restricts network connections to TLS and creates the
  rhdh SUPERUSER from postgress-external-db-pguser-rhdh.
- The registry.redhat.io pull secret is created from the existing
  REGISTRY_REDHAT_IO_SERVICE_ACCOUNT_DOCKERCONFIGJSON CI secret.

The Crunchy operator install, the PostgresCluster CRD waits and the OLM
Subscription helpers (only used for Crunchy) are removed, together with
the now empty cluster_setup_ocp_* hooks. Also rename the external DB TLS
spec and update the CI docs accordingly.

Adaptations for release-2.1:
- Do not add the ocp-fips-helm.sh change: the job does not exist on this
  branch.
- Keep the branch's Playwright projects and test ignores (auth-providers,
  no SHOWCASE_FIPS project) and only update the external DB spec name.
- Keep the branch's layer-migration-matrix rows and only rename row 25.
- The pr-triage content lives in .rulesync/commands here, so the edits are
  applied to that source and the generated copies are regenerated.

Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4113
Assisted-by: OpenCode
(cherry picked from commit 6e6cd04)
@zdrapela
zdrapela force-pushed the cherry-pick-5820-release-2.1 branch from 23fd2a3 to 3013621 Compare November 13, 2026 10:09
@sonarqubecloud

Copy link
Copy Markdown

@albarbaro

Copy link
Copy Markdown
Member

/lgtm

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@openshift-merge-bot
openshift-merge-bot Bot merged commit 7eb3a2e into redhat-developer:release-2.1 Nov 13, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants