Skip to content

test(e2e): add porstgres and redis tests to fips job - #5726

Open
albarbaro wants to merge 3 commits into
redhat-developer:mainfrom
albarbaro:fips-porstgres-redis
Open

albarbaro wants to merge 3 commits into
redhat-developer:mainfrom
albarbaro:fips-porstgres-redis

Conversation

@albarbaro

@albarbaro albarbaro commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds external PostgreSQL and Redis TLS connection tests for FIPS environment.

PostgreSQL:

  • Fixed startup failure in FIPS mode: PostgreSQL 18 defaults ssl_groups to X25519:prime256v1, but X25519 is not FIPS-approved (NIST FIPS 186-4)
  • Created postgres-fips.yaml with FIPS-compatible setting: ssl_groups: "prime256v1"
  • Added optional fips_mode parameter to configure_external_postgres_db()
  • Uses Crunchy's auto-generated CA (tests multi-CA scenario: FIPS CA for ingress/Redis, Crunchy CA for PostgreSQL)

Redis:

  • Added simple PING test to fips-compliance.spec.ts to validate TLS connection

Testing:

  • PostgreSQL: Backstage startup with external DB over TLS proves connection works
  • Redis: Simple PING validates TLS connection, auth, and server response

Which issue(s) does this PR fix

Evidence

  • Before: <screenshot / failing test run / output>
  • After: <screenshot / passing test run / output>

Testing

How the change was tested: the commands that ran, their output, and what the checks would miss.

Merge Danger

Door: one-way or two-way (can we walk back through it?)

Blast radius:

How to revert

<revert command, flag to turn off, or rollback step>

PR acceptance criteria

Please make sure that the following steps are complete:

  • GitHub Actions are completed and successful
  • Unit Tests are updated and passing
  • E2E Tests are updated and passing
  • Documentation is updated if necessary (requirement for new features)
  • Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer

@openshift-ci

openshift-ci Bot commented Oct 26, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@albarbaro

Copy link
Copy Markdown
Member Author

/test ?

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.33%. Comparing base (a4bc046) to head (66f7c1f).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5726      +/-   ##
==========================================
- Coverage   78.97%   73.33%   -5.65%     
==========================================
  Files          66       59       -7     
  Lines         723      570     -153     
  Branches      100       83      -17     
==========================================
- Hits          571      418     -153     
  Misses        151      151              
  Partials        1        1              
Flag Coverage Δ *Carryforward flag
rhdh 69.38% <ø> (-6.65%) ⬇️
rhdh-e2e-frontend 71.23% <ø> (ø) Carriedforward from a4bc046

*This pull request uses carry forward flags. Click here to find out more.

Components Coverage Δ
Backend plugins ∅ <ø> (∅)
Backend app 79.91% <ø> (ø)
Frontend app 68.40% <ø> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4bc046...66f7c1f. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@albarbaro albarbaro changed the title Fips porstgres redis test(e2e):add porstgres and redis tests to fips job Oct 29, 2026
@albarbaro albarbaro changed the title test(e2e):add porstgres and redis tests to fips job test(e2e): add porstgres and redis tests to fips job Oct 29, 2026
@albarbaro
albarbaro marked this pull request as ready for review October 29, 2026 13:27
@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@albarbaro

Copy link
Copy Markdown
Member Author

fips nightly job is failing on unrelated test cases (report, mostly scaffolder actions). The other tests are all green.

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@zdrapela

zdrapela commented Nov 3, 2026

Copy link
Copy Markdown
Member

/agentic_review

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Code Review by Qodo


Action required

1. New FIPS database and Redis tests all fail at setup 🐞 Bug ⭐ New
Description
beforeAll runs oc port-forward svc/postgress-external-db-primary, but Crunchy PGO v5 creates
that primary Service without a selector, and oc port-forward refuses such Services with "Service
is defined without a selector". The port-forward never prints Forwarding from, so
pgPortForward.start() rejects and the shared beforeAll fails, which takes down all six tests in
this describe block, including the Redis and RHDH-in-FIPS-mode checks.
Code

e2e-tests/playwright/e2e/fips-compliance.spec.ts[R293-298]

+    pgPortForward = portForward(
+      pgNamespace,
+      "svc/postgress-external-db-primary",
+      POSTGRES_LOCAL_PORT,
+      5432,
+    );
Relevance

●● Moderate

No direct precedent for port-forward target changes. Repo's schema-mode helper already resolves
PostgreSQL via pod, which supports the fix, but the claim depends on cluster behavior.

PR-#4288

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The spec port-forwards the Crunchy -primary Service. Crunchy's maintainers confirm that this
Service has no selector and that kubectl/oc port-forward fails on it. The repo's existing
schema-mode helper avoids this on purpose: when it finds postgress-external-db-primary, it looks
up the Running postgres pod by label and forwards pod/<name> (forward_via_pod=1) instead of the
svc.

.ci/pipelines/lib/schema-mode-env.sh[67-89]
e2e-tests/playwright/e2e/fips-compliance.spec.ts[236-246]
🌐 kubectl port-forward -n crunchy-db1 svc/crunchy-db1-primary 5432 → error: cannot attach to *v1.Service: invalid service &#x27;crunchy-db1-primary&#x27;: Service is defined without a selector; maintainer: the primary services no longer use selectors.

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The FIPS spec port-forwards `svc/postgress-external-db-primary`. Crunchy creates that Service without a selector, so `oc port-forward` rejects it and the whole describe block fails in `beforeAll`.

## Fix Focus Areas
- e2e-tests/playwright/e2e/fips-compliance.spec.ts[287-301]

## Recommended Fix
Before starting the PostgreSQL port-forward, use `kubeClient.coreV1Api.listNamespacedPod(pgNamespace, ..., labelSelector='postgres-operator.crunchydata.com/cluster=postgress-external-db,postgres-operator.crunchydata.com/role=master')` to find the primary pod. If the `role` label is not available, fall back to the `data=postgres` label and pick a Running pod. Then call `portForward(pgNamespace, `pod/${podName}`, POSTGRES_LOCAL_PORT, 5432)`, which matches `.ci/pipelines/lib/schema-mode-env.sh`. Throw a clear error if no Running pod is found.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

2. Redis cannot read its TLS certificate 🐞 Bug
Description
The Redis deployment mounts redis-certs with defaultMode: 0400 while running without an explicit
user or group that can read root-owned Secret files. On an OpenShift restricted pod running as a
non-root UID, redis-server cannot open the certificate and key required by its TLS-only listener,
so the service never becomes available.
Code

.ci/pipelines/resources/redis-cache/redis-deployment-tls.yaml[R61-64]

+        - name: redis-certs
+          secret:
+            secretName: redis-certs
+            defaultMode: 0400
Relevance

●● Moderate

The OpenShift non-root Secret-permission failure is plausible, but no closely matching repository
precedent exists.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new Redis command requires all three Secret-mounted certificate files. Unlike the existing
non-TLS deployment, the new deployment introduces a root-only Secret volume without configuring
access for a non-root runtime.

.ci/pipelines/resources/redis-cache/redis-deployment-tls.yaml[40-64]
.ci/pipelines/resources/redis-cache/redis-deployment.yaml[15-53]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The TLS Secret uses root-only file permissions, but Redis must read those files under OpenShift's non-root runtime.
## Fix Focus Areas
- .ci/pipelines/resources/redis-cache/redis-deployment-tls.yaml[40-64]
## Recommended Fix
Configure a compatible pod security context and Secret permissions so the Redis runtime UID or group can read the TLS files without making the private key world-readable.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


View medium (3)
Remediation recommended
3. Failed setup leaves database tunnels running 🐞 Bug ⭐ New
Description
The new afterAll hook calls pg.end() unconditionally, even when beforeAll fails before
assigning pg. If the PostgreSQL or Redis port-forward fails to start, teardown throws before
stopping either tunnel, masking the setup error and leaving any successfully started tunnel running
until the worker exits.
Code

e2e-tests/playwright/e2e/fips-compliance.spec.ts[R318-320]

+    await pg.end();
+    await pgPortForward?.stop();
+    await redisPortForward?.stop();
Relevance

●● Moderate

Closest precedent (cleanup must run even if beforeAll fails) was rejected in PR 3987, but that is
roughly nine months old and was a broader suggestion. Evidence is weak either way.

PR-#3987
PR-#5027

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The setup starts the PostgreSQL tunnel before the Redis tunnel and assigns pg only afterward.
Teardown calls pg.end() before either tunnel's stop(), so a failure between those operations
prevents tunnel cleanup.

e2e-tests/playwright/e2e/fips-compliance.spec.ts[259-262]
e2e-tests/playwright/e2e/fips-compliance.spec.ts[293-321]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The FIPS suite's teardown assumes PostgreSQL setup completed. A partial setup failure makes `pg.end()` throw before either port-forward is stopped.

## Fix Focus Areas
- e2e-tests/playwright/e2e/fips-compliance.spec.ts[287-321]

## Recommended Fix
Make the PostgreSQL client optional until it connects, and guard its cleanup. Stop both port-forward sessions independently, even if closing the client or one tunnel fails, while preserving the original setup failure.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

4. Failed setup leaves the CA key on disk 🐞 Bug
Description
fips_configure_custom_ca_ingress now preserves FIPS_CERT_TMPDIR, including rootCA.key, but
cleanup runs only at the end of fips_deployment. If the newly added PostgreSQL or Redis setup
returns early, that cleanup is skipped and the private key remains in the temporary directory.
Code

.ci/pipelines/jobs/ocp-fips-helm.sh[R329-330]

+  # NOTE: tmpdir is preserved (FIPS_CERT_TMPDIR) for reuse in PostgreSQL and Redis cert generation
+  # It will be cleaned up at the end of fips_deployment()
Relevance

●●● Strong

Recent FIPS precedent accepted cleanup fixes for temporary certificate material and exit-path leaks.

PR-#5579

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The ingress setup writes the CA private key into the retained directory; both newly added setup
calls can return before the only new cleanup block is reached.

.ci/pipelines/jobs/ocp-fips-helm.sh[247-260]
.ci/pipelines/jobs/ocp-fips-helm.sh[326-330]
.ci/pipelines/jobs/ocp-fips-helm.sh[41-75]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Preserving the FIPS certificate directory for Redis signing leaves its CA private key behind when database or Redis setup returns early.
## Fix Focus Areas
- .ci/pipelines/jobs/ocp-fips-helm.sh[41-48]
- .ci/pipelines/jobs/ocp-fips-helm.sh[70-75]
- .ci/pipelines/jobs/ocp-fips-helm.sh[281-330]
## Recommended Fix
Register cleanup immediately after creating the certificate directory and run it on every success and failure path, including early returns, without masking the original failure.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

5. Backstage can't verify the Redis certificate 🐞 Bug
Description
The FIPS values file puts the Redis CA under backend.cache.tlsOptions, but Backstage's cache
config has no such key, so the setting is silently ignored. The backend pod runs with
NODE_TLS_REJECT_UNAUTHORIZED=1, and its NODE_EXTRA_CA_CERTS (from postgres-cred) points to the
PostgreSQL tls.crt, not the FIPS root CA, so the backend's rediss://redis:6379 connection should
fail certificate verification. Cache errors are only logged when a plugin uses the cache, so the pod
still starts and passes its probes. The new PING test port-forwards and brings its own CA, so it
passes even though Backstage itself never talks to Redis over TLS.
Code

.ci/pipelines/value_files/diff-values_showcase-fips.yaml[R27-31]

+      # Redis TLS configuration
+      tlsOptions:
+        rejectUnauthorized: true
+        ca:
+          $file: /opt/app-root/src/redis-ca.pem
Relevance

●● Moderate

Substantive semantic config-correctness claim with no close accepted or rejected precedent. A
similar 'key silently ignored' finding was dismissed as a false positive, so keep uncertain.

PR-#5641

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Backstage's backend.cache schema for store: redis accepts only connection, defaultTtl and
redis.client/redis.cluster; there is no tlsOptions. create_postgres_cred_secret sets
NODE_EXTRA_CA_CERTS=/opt/app-root/src/postgres-crt.pem, and the FIPS values mount that path from
the Crunchy secret's tls.crt. The Redis cert is signed by the FIPS root CA
(fips_deploy_redis_with_tls), which the backend pod never trusts. verify-redis-cache.spec.ts is
now ignored for this project, so nothing checks Backstage's own Redis connection.

.ci/pipelines/utils.sh[205-218]
.ci/pipelines/value_files/diff-values_showcase-fips.yaml[33-56]
.ci/pipelines/jobs/ocp-fips-helm.sh[412-432]
🌐 The redis cache store schema defines only connection, defaultTtl and redis.client/redis.cluster; it has no TLS options key.

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Backstage ignores `backend.cache.tlsOptions`, and the backend pod's NODE_EXTRA_CA_CERTS points to the PostgreSQL leaf cert. As a result, the backend's TLS connection to Redis can't verify the FIPS-CA-signed certificate.

## Fix Focus Areas
- .ci/pipelines/value_files/diff-values_showcase-fips.yaml[23-31]
- .ci/pipelines/value_files/diff-values_showcase-fips.yaml[33-56]

## Recommended Fix
1. Remove the unsupported `tlsOptions` block and the deprecated `useRedisSets` key.
2. Build a combined CA file in the pod (FIPS root CA + Crunchy ca.crt), for example with a projected volume or a secret that holds both.
3. Point `NODE_EXTRA_CA_CERTS` at that file through `extraEnv`. Because `extraEnvFrom` with `postgres-cred` also sets this variable, set it in a way that wins (explicit `env` takes precedence over `envFrom`).
4. Add a check that Backstage's cache really works over TLS, for example by keeping a FIPS variant of the redis-cache test or by asserting that the backend logs contain no cache errors.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


View low (1)
Informational
6. A missing PostgreSQL CA goes undetected 🐞 Bug ⭐ New
Description
The oc get secret ... | base64 -d > crunchy_ca_file pipeline reports only base64's exit status
(no pipefail), and the redirect always creates the file. If oc fails, the fallback branch is
skipped, an empty file is appended to the combined bundle, and the later [[ -f crunchy_ca_file ]]
check is still true, so the failure shows up only as a misleading certutil warning.
Code

.ci/pipelines/jobs/ocp-fips-helm.sh[R173-177]

+  if ! oc get secret postgress-external-db-cluster-cert -n "${NAME_SPACE}" -o jsonpath='{.data.ca\.crt}' | base64 -d > "${crunchy_ca_file}"; then
+    log::warning "Failed to extract Crunchy CA - PostgreSQL might not be deployed yet"
+    # Fall back to FIPS CA only
+    cp "${root_ca_file}" "${combined_ca_file}"
+  else
Relevance

●●● Strong

Shell error-handling hardening is consistently accepted here (pipefail/set -e and '|| return 1'
suggestions were accepted); the fix is local and deterministic.

PR-#5479
PR-#5709

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Without pipefail, a pipeline's status is the status of its last command. base64 -d on empty input
exits 0, and > file creates the file whatever happens, so neither the fallback branch nor the -f
guard detects the missing CA. Only the top-level script sets set -o errexit, not pipefail, and
that is suspended here anyway because the function is called inside || {}.

.ci/pipelines/jobs/ocp-fips-helm.sh[168-205]
.ci/pipelines/openshift-ci-tests.sh[1-5]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Crunchy CA extraction pipeline cannot detect an `oc` failure, and the `-f` check is always true.

## Fix Focus Areas
- .ci/pipelines/jobs/ocp-fips-helm.sh[172-205]

## Recommended Fix
Run `ca_b64=$(oc get secret ... -o jsonpath='{.data.ca\.crt}')` first. If the command fails or `ca_b64` is empty, log a warning, `rm -f` the crunchy file and fall back to the FIPS CA. Otherwise decode it into the file. Change the later guard to `[[ -s "${crunchy_ca_file}" ]]`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


@zdrapela zdrapela left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, I think the Qodo findings about Redis are worth checking, and I'd try to minimize the duplication if possible. WDYT?

Comment thread .ci/pipelines/jobs/ocp-fips-helm.sh Resolved
Comment thread .ci/pipelines/jobs/ocp-fips-helm.sh Resolved
@albarbaro

Copy link
Copy Markdown
Member Author

/agentic_review

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@albarbaro

Copy link
Copy Markdown
Member Author

/test ?

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@zdrapela

Copy link
Copy Markdown
Member

/agentic_review

Comment thread .ci/pipelines/jobs/ocp-fips-helm.sh Resolved
Comment thread .ci/pipelines/jobs/ocp-fips-helm.sh Resolved
@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@zdrapela

Copy link
Copy Markdown
Member

/agentic_review

@albarbaro

Copy link
Copy Markdown
Member Author

/agentic_review

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@albarbaro

Copy link
Copy Markdown
Member Author

/retest

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@albarbaro

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-22-fips-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@sonarqubecloud

Copy link
Copy Markdown

@albarbaro

Copy link
Copy Markdown
Member Author

/agentic_review

1 similar comment
@zdrapela

Copy link
Copy Markdown
Member

/agentic_review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants