ci: fix broken pg_isready readiness probe on the WarehousePG CI lane - #323
Merged
Conversation
pip-audit>=2.7 (a direct dev dependency, used by the CI security job and the local pre-commit hook) pulls in pip_api -> pip 26.1.2, which has a known vulnerability (PYSEC-2026-3721, fixed in 26.2). Discovered because it was blocking the local pre-commit hook's own dependency audit step on an unrelated commit. `uv lock --upgrade-package pip` bumps just the one pinned entry; `uv run pip-audit --strict` (the exact invocation the CI security job uses) now reports no known vulnerabilities. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMBaAe32ggci6ukABssDKj
The `warehousepg-latest` matrix entry (Tests (PG warehousepg-latest))
was taking ~9-10 minutes per run vs. ~3.5-4.5 minutes for every other
PG version — confirmed via `gh run view` step timings on two recent
runs. The gap traced entirely to the "Start PostgreSQL (pgvector +
PostGIS)" step (~6-6.5 min there vs. ~20-30s elsewhere); the pytest
step itself was essentially identical in duration to every other lane.
Root cause, from raw run logs: the readiness-poll loop does
`docker exec mcpg-db pg_isready -U gpadmin`, up to 90 times, 4s apart.
Every single attempt across both inspected runs failed identically:
OCI runtime exec failed: exec failed: unable to start container
process: exec: "pg_isready": executable file not found in $PATH
`woblerr/warehousepg:7.4.1-WHPG` doesn't ship `pg_isready` on PATH
(not documented upstream either). The loop could never observe
success, so it always burned its full 90 * 4s = 360s budget as dead
time on every run, regardless of how fast the database actually came
up underneath.
Fix: probe with `psql` instead, connecting through the published port
the same way pytest itself does (the runner's default client already
works here — see the "Install PostgreSQL client tools" step, which
deliberately skips a custom client for this lane). This is also a
strictly stronger check than `pg_isready`: it only succeeds once the
target database accepts queries, not just once the postmaster process
is up. The 90-attempt/4s ceiling stays as a safety net for a
genuinely slow single-node Greenplum-family init, but the loop now
breaks as soon as it's actually ready instead of always waiting out
the full ceiling. Every other matrix lane's pg_isready-based probe is
untouched.
Expected effect: the warehousepg-latest lane should drop from ~9-10
min to something close to the other lanes' ~4 min, unless the
underlying Greenplum-family init genuinely needs more than a few tens
of seconds (unverified — no local docker/CI access to time it
directly; validate on the next actual CI run).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMBaAe32ggci6ukABssDKj
Adds [Unreleased] entries for the two preceding commits, per this repo's PR template checklist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMBaAe32ggci6ukABssDKj
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the
warehousepg-latestCI matrix lane (Tests (PG warehousepg-latest))taking ~9-10 minutes per run vs. ~3.5-4.5 minutes for every other PG version.
Diagnosis (via
gh run viewstep timings on two recent runs): the gaptraced entirely to the "Start PostgreSQL (pgvector + PostGIS)" step
(~6-6.5 min there vs. ~20-30s elsewhere) — the
pyteststep itself wasalready the same duration as any other lane. Raw logs showed the
readiness-poll loop's
docker exec mcpg-db pg_isready -U gpadminfailingidentically on every single one of its 90 attempts:
woblerr/warehousepg:7.4.1-WHPGdoesn't shippg_isreadyon PATH (notdocumented upstream either), so the loop could never observe success and
always burned its full 90 × 4s = 360s budget as dead time, regardless of
how fast the database actually came up underneath.
Fix: probe with
psqlinstead, connecting through the published portthe same way pytest itself does — the runner's default client already
works here (the "Install PostgreSQL client tools" step deliberately skips
a custom client for this lane). It's also a strictly stronger check than
pg_isready: it only succeeds once the target database actually acceptsqueries. The 90-attempt/4s ceiling stays as a safety net; every other
matrix lane's
pg_isready-based probe is untouched.Also includes a small prerequisite fix: bumped a transitive
pipdependency (26.1.2 → 26.2.1, PYSEC-2026-3721, pulled in via
pip-audit'sown
pip_apidep) that was blocking the local pre-commit hook'sdependency audit on an unrelated commit — kept as its own commit rather
than folded into the CI fix.
Caveat: no local docker/CI access to time the fixed probe directly —
the ~9-10 min → ~4 min improvement is inferred from the log evidence
(pg_isready failing on literally every attempt), not re-measured. Worth
confirming on this PR's own CI run.
Roadmap linkage
N/A — CI/infra fix, no roadmap row.
Checklist
application code changed; full local unit suite (2909 passed, 3
skipped) run via the pre-commit hook on every commit
ruff,ruff format, andmypy src/mcpgpassCHANGELOG.mdupdated under[Unreleased]src/mcpg/_vendor/🤖 Generated with Claude Code
Summary by Sourcery
Restore timely WarehousePG CI startup by replacing its unavailable readiness utility with a working database connectivity check.
Bug Fixes:
warehousepg-latestCI lane readiness check so it detects database availability instead of exhausting its full polling timeout.pipdependency to address PYSEC-2026-3721.CI: