Skip to content

fix(ci): merge the AWS RDS global CA bundle into the RDS trust store - #5447

Open
gustavolira wants to merge 3 commits into
redhat-developer:mainfrom
gustavolira:fix/rds-ca-global-bundle
Open

gustavolira wants to merge 3 commits into
redhat-developer:mainfrom
gustavolira:fix/rds-ca-global-bundle

Conversation

@gustavolira

Copy link
Copy Markdown
Member

Description

The RDS TLS e2e tests in showcase-runtime fail with self-signed certificate in certificate chain on all four RDS instances (latest .. latest-3) — seen on e2e-osd-gcp-helm-nightly, e2e-ocp-helm-nightly and the operator variants (RHDHBUGS-3411, RHDHBUGS-3691).

Root cause

RDS_DB_CERTIFICATES_PATH points at the Vault copy of the RDS CA bundle. AWS rotates RDS CAs, and GSM cannot store the full ~170KB global bundle (RHDHBUGS-3744), so the stored copy is a trimmed snapshot that goes stale — the harness pg client (clearDatabase) and RHDH itself (via the postgres-crt secret) then reject the servers' chains.

Fix

At env-setup time, download the official AWS RDS global trust bundle and merge the Vault copy into it, pointing RDS_DB_CERTIFICATES_PATH at the merged file. Every current and future RDS CA generation/region validates, custom CAs from Vault are preserved, and if the download fails the behavior falls back to today's (Vault copy only, with a warning).

🤖 Generated with Claude Code

@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.04%. Comparing base (b1e2f8a) to head (e06d346).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #5447       +/-   ##
===========================================
- Coverage   74.21%   64.04%   -10.17%     
===========================================
  Files          69       57       -12     
  Lines         799      573      -226     
  Branches      125       99       -26     
===========================================
- Hits          593      367      -226     
  Misses        205      205               
  Partials        1        1               
Flag Coverage Δ
rhdh 64.04% <ø> (-10.17%) ⬇️
Components Coverage Δ
Backend plugins ∅ <ø> (∅)
Backend app 71.91% <ø> (ø)
Frontend app 60.50% <ø> (ø)
Plugin utils ∅ <ø> (∅)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1e2f8a...e06d346. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@gustavolira

Copy link
Copy Markdown
Member Author

/tes e2e-osd-gcp-helm-nightly

@gustavolira

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

1 similar comment
@gustavolira

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@gustavolira
gustavolira force-pushed the fix/rds-ca-global-bundle branch from 3aec0e3 to cde1ff6 Compare September 18, 2026 18:25
@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@gustavolira

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

gustavolira and others added 3 commits September 18, 2026 17:40
The RDS TLS e2e tests fail with SELF_SIGNED_CERT_IN_CHAIN on all four RDS
instances: the Vault copy of rds-db-certificates.pem no longer validates
the servers' certificate chains (AWS rotates RDS CAs, and GSM cannot hold
the full global bundle — RHDHBUGS-3744 — so the stored copy is a trimmed
snapshot that goes stale).

Download the official AWS RDS global trust bundle at env-setup time and
merge the Vault copy into it, pointing RDS_DB_CERTIFICATES_PATH at the
merged file. Both consumers pick it up: the harness pg client
(clearDatabase) and the postgres-crt secret mounted into RHDH
(configurePostgresCertificate). If the download fails, fall back to the
Vault copy alone (current behavior).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up: under errexit a double failure of the cat/cp fallback
chain would abort the whole job before the cleanup trap is installed;
guard it and keep pointing at the Vault copy in that case.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SonarCloud: curl -L could follow a redirect to a non-HTTPS target;
restrict both the initial request and any redirect to HTTPS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gustavolira
gustavolira force-pushed the fix/rds-ca-global-bundle branch from cde1ff6 to e06d346 Compare September 18, 2026 20:40
@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant