[anth-volk]

Fixes #306

Incident

On 2026-10-20 it was disclosed that the private keys of eight eMudhra (emSign) intermediate CA certificates can be recovered from the public certificates (https://emudhra.we-have-a-problem.com). Anyone can therefore issue a certificate for any domain that chains to "emSign ECC Root CA - G3" or, through Sectigo's cross-signatures, to "AAA Certificate Services". Browsers already block them; what stays exposed is software that carries its own CA list, or reads an operating-system CA store that has not been updated.

What in this repository trusted the affected roots

1. The /uk/chat/api/proxy/* route on Vercel. frontend/src/app/api/proxy/[...slug]/route.ts runs as a Vercel Function (runtime = "nodejs", project policyengine-uk-chat, Node.js 24.x) and fetches the Modal backend, forwarding the user's Authorization and X-User-Id headers. Vercel Functions ran Node.js 24.21.0 when last measured (another PolicyEngine project's preview function on 2026-10-26), and the CA list compiled into that release includes "emSign ECC Root CA - G3". Vercel offers only 24.x and 22.x, so no project setting changes this. The production deployment is still serving (the backend behind it is stopped, Production backend stopped until auth issues #128, #130 and #131 are fixed #275, so the route currently returns Modal's 404).
2. The Modal backend image (modal_app.py, app policyengine-uk-chat and the pe-uk-chat-<PR> previews):
   • modal.Image.debian_slim(python_version="3.13") is built from python:3.13.3-slim-bookworm, which carries Debian ca-certificates 20230311 (includes both roots), and no step upgrades it.
   • backend/requirements.txt does not pin certifi, so the image has whatever certifi was current when Modal last built the requirements step. The last production deploy was 2026-09-02 (run 33631583740), before certifi 2026.10.19, the first release without the emSign root.
   • policyengine-observability[google] writes logs to Google Cloud Logging through google-cloud-logging, which uses gRPC. gRPC verifies against the CA list bundled in grpcio, which includes both roots in every release so far. Previews use this destination by default; production currently sets OBSERVABILITY_LOG_DESTINATIONS=stdout.
3. The Modal eval image (modal_eval_app.py) sends the eval requests with httpx, which uses certifi, again unpinned in backend/requirements-modal-eval.txt.

Not affected or not changed:

• stripe 16.1.0 bundles its own 12-certificate CA list; neither root is in it (checked by SHA-256 fingerprint).
• OpenTelemetry OTLP export is off: policyengine-observability enables it only when OTEL_EXPORTER_OTLP_ENDPOINT is set, and sync-modal-secret.sh does not set it.
• backend/Dockerfile, frontend/Dockerfile and docker-compose.yml are local development only.

Also changed: the frontend CI jobs (tests.yml and deploy.yml) ran Node.js 20.19, which gets no release without the emSign root. Run under Node.js 20.19.6 and npm 10.8.2 (the versions CI installed), npm ci --loglevel http fetched 386 package tarballs, which the lockfile's integrity hashes check, and also sent requests whose responses no hash covers: the audit request POST https://registry.npmjs.org/-/npm/v1/security/advisories/bulk (CI's log shows it: "audited 383 packages … 11 vulnerabilities") and package-metadata requests for 16 packages the audit reports on (for example next, vite, ws, undici), plus npm's own update check, which npm skips when CI is set. strace -e trace=connect recorded no network connections at all from npm run test:coverage and npm run build (a control fetch under the same tracing recorded its port-443 and DNS connections); the Google Fonts stylesheet in layout.tsx is a <link> the browser loads.

• No code in the repository disables certificate verification or sets a custom CA bundle.

Changes

• frontend/src/utils/compromisedCaRoots.ts (new): removeCompromisedEmudhraRoots() removes from Node's default CA list every certificate whose SHA-256 fingerprint (X509Certificate#fingerprint256) is one of:

  • "emSign ECC Root CA - G3": 86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B
  • "AAA Certificate Services": D7:A7:A0:FB:5D:7E:27:31:D7:71:E9:48:4E:BC:DE:F7:1D:5F:0C:3E:0A:29:48:78:2B:C8:3E:E0:EA:69:9E:F4, removed only as a defensive measure. Node's compiled-in list does not contain it (Mozilla trusts it for email only); it can reach the default list only through NODE_EXTRA_CA_CERTS or --use-system-ca.

  It uses tls.getCACertificates("default") and tls.setDefaultCACertificates. On each server start it logs, with console.info, how many certificates it removed, the list size before and after, and process.version. It logs console.warn when either Node.js function is missing. An error is logged with console.error, not rethrown, because an error from instrumentation's register() makes every function-served route fail. A comment says to delete the module once Vercel Functions run Node.js 24.22.1 / 22.23.4 or later. This follows the reviewed design in Stop trusting the compromised eMudhra CA certificates uk-cgt-reform#19.

• frontend/src/utils/compromisedCaRoots.test.ts (new): Vitest cases (Node environment) with a stub tls and logger: removing G3 while keeping an unrelated root, removing AAA and repeated copies, a list with neither root, the early return for each missing function, and errors from reading the list, parsing a certificate and setting the list.

• frontend/src/instrumentation.ts (new): register() imports the module dynamically and calls it only when process.env.NEXT_RUNTIME === "nodejs". Next.js 15.5.14 runs instrumentation without a configuration flag. @types/node 22.19.15 already declares both TLS functions.

• modal_app.py: after pip_install_from_requirements, the image now runs .apt_install("ca-certificates") (Modal runs apt-get update then apt-get install -y ca-certificates), .pip_install("certifi>=2026.10.19"), and .env({"GRPC_DEFAULT_SSL_ROOTS_FILE_PATH": "/etc/ssl/certs/ca-certificates.crt"}), so gRPC reads the refreshed Debian bundle instead of grpcio's. They go after the requirements step so Modal can reuse that step's cached build instead of re-resolving every unpinned package. certifi is a minimum, not a pin, so a rebuild picks up later removals.

• modal_eval_app.py: .pip_install("certifi>=2026.10.19") after its requirements step.

• .github/workflows/tests.yml and .github/workflows/deploy.yml: the frontend job's setup-node node-version "20.19" → "24", the release series the Vercel project runs (24.22.2 today, which has the fixed CA list).

• backend/tests/test_modal_app.py: the image-step assertions for both apps include the new steps and their arguments; the fake image gains env.

Checks

• npx vitest run src/utils/compromisedCaRoots.test.ts: 8 passed.
• npx tsc --noEmit -p . in frontend/: no errors.
• next build with Node.js 24.21.0: succeeded.
• Built standalone server (node .next/standalone/server.js) on Node.js 24.21.0: at startup it logs [compromisedCaRoots] Removed 1 certificate(s) of the compromised roots (emSign ECC Root CA - G3) from Node's default CA list (121 -> 120 certificates) on Node.js v24.21.0. With POLICYENGINE_UK_CHAT_BACKEND_URL set to each host, GET /uk/chat/api/proxy/health gave:
  • https://dv-g3b-emudhra.we-have-a-problem.com and https://dv-g3b-aaa-emudhra.we-have-a-problem.com: HTTP 500 fetch failed (the request to the test host failed);
  • https://pypi.org and https://policyengine--policyengine-uk-chat-web.modal.run: the upstream 404 passed through, so TLS to other hosts still works.
• Plain fetch on Node.js 24.21.0, importing the module directly: without it, the emSign test host answered HTTP 200 and the AAA host failed (UNABLE_TO_GET_ISSUER_CERT_LOCALLY); with it, both failed with UNABLE_TO_GET_ISSUER_CERT_LOCALLY and https://pypi.org/ answered HTTP 200.
• Modal image steps reproduced in python:3.13.3-slim-bookworm, starting from certifi 2026.7.22 and ca-certificates 20230311:
  • before: httpx with certifi got HTTP 200 from the emSign test host; the operating-system store got HTTP 200 from both test hosts;
  • after apt-get update && apt-get install -y ca-certificates and pip install 'certifi>=2026.10.19' (ca-certificates 20250419~deb12u2, certifi 2026.10.23): both test hosts failed with CERTIFICATE_VERIFY_FAILED through both CA lists, and https://pypi.org/ answered HTTP 200.
  • That gRPC reads GRPC_DEFAULT_SSL_ROOTS_FILE_PATH was verified for the Public API in Stop trusting the compromised eMudhra CA certificates policyengine-api#3999; it was not re-tested here.
• pytest backend/tests/test_modal_app.py::test_modal_deployment_definition_imports_without_remote_calls backend/tests/test_modal_app.py::test_modal_eval_app_fans_out_twenty_cases_with_a_25_container_cap: 2 passed.
• Importing modal_app and modal_eval_app with modal 1.6.3 constructs both images.
• Frontend CI commands on Node.js 24 (node:24-bookworm, Node.js 24.22.2, npm 11.19.1, CI=true): npm ci succeeded (npm 11 skips sharp's install script by default and reports it; the build does not need it, since sharp's binaries come from the locked @img/sharp-* packages), npx tsc --noEmit -p . reported no errors, npx vitest run src/utils/compromisedCaRoots.test.ts passed 8 tests, and npm run build succeeded. The frontend has no lint script.

Operator actions after merge

• Vercel deploys main to production as usual; nothing else is needed for the frontend. After the deploy, the function logs should show the [compromisedCaRoots] Removed 1 certificate(s) message with the running Node.js version.
• The production Modal app is stopped and the "Deploy to Modal" workflow is disabled (Production backend stopped until auth issues #128, #130 and #131 are fixed #275). When the backend is brought back with that workflow, check in the Deploy to Modal step that the new image steps run (ca-certificates 20250419~deb12u2 or later, certifi 2026.10.19 or later). The same applies to PR previews once pr-beta-deploy.yml deploys them again.
• The next modal run modal_eval_app.py builds the certifi step automatically.

Environment-variable verification

This PR changes Modal image definitions, so every variable the changed files read or declare was traced:

Variable	Read or declared by	Provider	Wiring	Status
GRPC_DEFAULT_SSL_ROOTS_FILE_PATH (new)	modal_app.py .env(); read by gRPC in the backend	Fixed value in the image	Image environment; sync-modal-secret.sh puts no variable of that name in the Modal secret, so nothing overrides it	verified
NEXT_RUNTIME (new reader)	frontend/src/instrumentation.ts	Set by Next.js in each server runtime	No configuration	verified
POLICYENGINE_UK_CHAT_MODAL_APP_NAME, POLICYENGINE_UK_CHAT_MODAL_SECRET_NAME (unchanged)	modal_app.py at deploy time	Shell running modal deploy	deploy.yml sets neither, so the defaults apply; deploy-modal-preview.sh sets both from the workflow-level MODAL_PREVIEW_* values	verified
ALEMBIC_DATABASE_URL (unchanged)	modal_app.py functions	Modal secret, filled from the GitHub Actions secret POLICYENGINE_UK_CHAT_DATABASE_URL (repository and organization level)	sync-modal-secret.sh step env: → modal secret create; attached with secrets=[chat_secrets]	verified (wiring); the secret value was not read
POLICYENGINE_UK_CHAT_EVAL_MODAL_APP_NAME, ..._SECRET_NAME, ..._VOLUME_NAME (unchanged)	modal_eval_app.py	Shell running modal run	Defaults apply when unset	verified
UK_CHAT_EVAL_TOKEN (unchanged)	modal_eval_app.py	Modal secret policyengine-uk-chat-secrets, filled from the GitHub Actions repository secret UK_CHAT_EVAL_TOKEN	sync-modal-secret.sh	verified (wiring)
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET (unchanged)	modal deploy in deploy.yml and pr-beta-deploy.yml	GitHub Actions organization secrets available to this repository	Step env: from secrets.*	verified (names available to the repository)
POLICYENGINE_UK_CHAT_BACKEND_URL, VERCEL_ENV, VERCEL_GIT_PULL_REQUEST_ID (unchanged; on the path this protects)	frontend/src/app/api/proxy/backend-url.ts	Vercel project environment variable; Vercel system variables	Vercel runtime	unverified: the project's environment variables were not listed
NODE_EXTRA_CA_CERTS, NODE_USE_SYSTEM_CA, NODE_OPTIONS (not read; they would change the list the module filters)	Node.js	Vercel project environment variables, if set	Vercel runtime	unverified for this project; the module removes the roots whichever way they arrive

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-uk-chat	Ready	Preview	Oct 30, 2026 3:21pm UTC

[codecov]

Codecov Report

❌ Patch coverage is 88.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.12%. Comparing base (b624984) to head (1c648d1).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
frontend/src/instrumentation.ts	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #305      +/-   ##
==========================================
+ Coverage   74.04%   75.12%   +1.07%     
==========================================
  Files         153      157       +4     
  Lines       11769    11914     +145     
  Branches     2010     2047      +37     
==========================================
+ Hits         8714     8950     +236     
+ Misses       2313     2209     -104     
- Partials      742      755      +13     

Flag	Coverage Δ
backend	82.15% <ø> (+1.16%)	⬆️
frontend	38.13% <88.00%> (+0.66%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
frontend/src/utils/compromisedCaRoots.ts	100.00% <100.00%> (ø)
modal_app.py	34.61% <ø> (ø)
frontend/src/instrumentation.ts	0.00% <0.00%> (ø)

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.