Skip to content

feat(cloudsql): support GCP workload identity auth - #4718

Draft
jkaflik wants to merge 3 commits into
mainfrom
kk/clp-959-peerdb-cloudsql-wi
Draft

feat(cloudsql): support GCP workload identity auth#4718
jkaflik wants to merge 3 commits into
mainfrom
kk/clp-959-peerdb-cloudsql-wi

Conversation

@jkaflik

@jkaflik jkaflik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Problem

Cloud SQL-backed PostgreSQL and MySQL peers currently require static database passwords. ClickPipes needs to authenticate with short-lived IAM database login tokens minted through the tenant GSA workload-identity chain.

Linear: https://linear.app/clickhouse/issue/CLP-959/cloud-sql-source-workload-identity-iam-auth

Solution

  • Add dedicated PostgreSQL and MySQL Cloud SQL IAM auth enum values.
  • Extract the existing GCP workload-identity exchange and tenant-GSA impersonation into a shared utility.
  • Mint a fresh sqlservice.login token for each ordinary, replication, and binlog connection.
  • Preserve configured database usernames and existing SSH/RPE networking.
  • Require TLS with certificate verification for Cloud SQL IAM auth.
  • Keep tokens in memory only and reject nil or empty token responses.

Testing

  • Focused tests passed for connectors/utils, connectors/bigquery, connectors/postgres, connectors/mysql, and internal.
  • Compile-only tests passed for affected packages.
  • golangci-lint passed with 0 issues.
  • git diff --check passed.

Risks and follow-up

  • Live Cloud SQL PostgreSQL logical-replication and MySQL binlog smoke tests are still required.
  • The MySQL path relies on Cloud SQL requesting mysql_clear_password over verified TLS.
  • After merge, publish a new flow-api-client version for the dependent clickpipes-platform PR.

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: All 5 failures on the single failing matrix leg are identical 60-second wall-clock timeouts (UNEXPECTED STATUS TIMEOUT STATUS_SETUP/STATUS_SNAPSHOT from flow/e2e/test_utils.go:296) spread across three unrelated source connectors sharing only the ClickHouse cluster destination, with no underlying errors logged and the other two matrix legs fully green — a resource-contention stall rather than a bug in this Cloud SQL workload-identity PR.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: Three unrelated e2e tests (MySQL→ClickHouse and Postgres) all hit 60s WaitFor timeouts within the same ~90-second window on only one of three matrix legs, while the PR itself only adds GCP workload-identity auth for CloudSQL — a shared CI infrastructure stall, not a code defect.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant