diff --git a/.claude/skills/dependency-update-pr-validator/SKILL.md b/.claude/skills/dependency-update-pr-validator/SKILL.md new file mode 100644 index 000000000..c808c9819 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/SKILL.md @@ -0,0 +1,141 @@ +--- +name: dependency-update-pr-validator +description: Validates dependabot dependency-update PRs against quark-engine's CI and drafts a merge/escalate recommendation. Use when asked to review, validate, or triage a dependabot PR on ev-flow/quark-engine, or when the user references issue 18z/QuarkHQ#3. +version: 1.0.0 +allowed-tools: >- + bash(gh pr view:*), + bash(gh pr checks:*), + bash(gh run list:*), + bash(gh run view:*), + bash(gh api:*), + bash(.claude/skills/dependency-update-pr-validator/scripts/*.sh) +--- +# Dependency Update PR Validator + +## Why this exist + +Dependabot spam near-identical PRs at `ev-flow/quark-engine` (one per dep bump). Hand-review each — CI really pass? new version really tested? — repetitive, easy rubber-stamp. Skill automate check, spit recommendation for human act on. Origin ask: [18z/QuarkHQ#3](https://github.com/18z/QuarkHQ/issues/3). + +**Skill never write GitHub.** Never call `gh pr comment`, `gh pr merge`, `gh pr review`. Every output draft for human read/edit/post self — PRs live upstream (`ev-flow/quark-engine`), not user fork, so post/merge there = visible shared-state action, need human loop every time. + +## Core insight: green CI ≠ target version tested + +Repo declare deps two places, drift apart: + +- `setup.py` — consumed by `pip install ".[QuarkAgent]"` in `pytest.yml`. +- `Pipfile` / `Pipfile.lock` — **no workflow consume this.** Nothing run `pipenv install`. + +History lesson (fixed, but why `check_workflow_pin.sh` still exists): `pytest.yml` used to have a standalone line hardcoding `langchain==0.2.11 langchain-core==0.2.23 langchain-openai==0.1.17`, totally ignoring `setup.py`. Dependabot bump `langchain-core` in `setup.py` → still all-green CI, cuz CI quietly kept testing the old hardcoded version, never the bump. Fixed by installing `.[QuarkAgent]` extras instead (commit `9d119c4`) — but **any** workflow step could grow a new hardcoded pin like this again for some other package, so still check for it, don't assume it's gone forever. + +Separate, still-live gotcha: unpinned transitive deps (e.g. `idna`, `requests`) often float to latest at install regardless of which file dependabot touch, and float version frequent already at-or-above target — legit pass, not coincidence to suspect. + +Lesson: **don't guess from which file PR changed.** Always check direct (a) what version actually install in CI run, (b) does some workflow step hardcode this exact package to older version. Two simple greps — trust over filename. + +## Workflow + +Run each step, **print result before next step** — slow part dig CI logs, so show cheap facts (package, current/target version, CI pass/fail) first, not hold all for one final report. + +### Step 1 — Parse PR (cheap, do first) + +```bash +bash .claude/skills/dependency-update-pr-validator/scripts/run_checkpoint.sh +``` + +Print PR title, package name, current version, target version, all CI checks passed (treat `skipping` as fine — e.g. `kali-package.yml` only run on push to master, always skip on PR; not failure). + +Show checkpoint to user now. + +### Step 2 — If `setup.py` changed: confirm new pin actually installable + +Run whenever `setup.py` among PR's changed files (check `changed_files` from Step 1) — **regardless CI pass/fail**. Static check of PR's own diff, not what CI happen test — CI failure Step 3 don't make this irrelevant — PR can fail CI for unrelated reason *and* be independently broken here, want catch both. + +```bash +bash .claude/skills/dependency-update-pr-validator/scripts/check_pip_resolution.sh +``` + +Dependabot PR usually bump one pin inside requirements list (e.g. `quarkAgentRequirements` in `setup.py`) while leave sibling pins same list untouched. Combo can be mutually unsatisfiable — `pip install` for those exact pinned versions together raise `ResolutionImpossible` — even when CI fully green, cuz CI's hardcoded install step (Step 4) test own separate versions, never actually try install what `setup.py` now declare. Stronger, more definitive reason block merge than `is_covered`: if real users can't even `pip install` result, don't matter what CI tested. If check report `pip_resolution=conflict`, recommendation **always** "Needs human review," regardless what CI/is_covered say. + +### Step 3 — If any check failed: find real root cause, don't just report "CI failed" + +Red check on dependabot PR frequent **not caused by bump at all**. Two pattern seen practice (see [examples/escalation_comment.md](examples/escalation_comment.md) for full writeups): + +- **Environment flake** — test fixture fail download valid file (e.g. `zipfile.BadZipFile: File is not a zip file` fetching sample APK). Unrelated any dependency. +- **Stale-branch baseline drift** — `smoke_test.yml`'s hardcoded expected behavior-counts (e.g. "Ahmyth.apk should show 39 behaviors") no longer match cuz master gain new detection rules after dependabot branch cut. Also unrelated bump. + +Find actual cause: + +```bash +gh run view --repo --log-failed +``` + +Read failure, classify "look like real regression from bump" vs "look unrelated" (give reasoning — e.g. failing package test-only tool, can't plausibly affect malware-detection counts), stop here (skip Step 4 — no successful build log read version from). Always escalation regardless classification — only diff what tell human, and Step 2's finding (if any) still apply on top. + +### Step 4 — If CI passed: check what actually installed + +```bash +bash .claude/skills/dependency-update-pr-validator/scripts/check_actual_version.sh +``` + +Find `pytest.yml` ("build") run for PR's branch, grep log for package's `Successfully installed` line — one place reflect what pip genuinely resolve+test, regardless which declaration file PR touch. Script also emit `proof_url` (link to the `build` job page) — link this in table row as "Found in CI log", not just bare version, so human verify same place without re-running anything. + +Then check workflow hardcode package to stale version (explains "No" result, when exist): + +```bash +bash .claude/skills/dependency-update-pr-validator/scripts/check_workflow_pin.sh +``` + +Then compute is_covered: + +```bash +bash .claude/skills/dependency-update-pr-validator/scripts/compare_versions.sh +``` + +### Step 5 — Print result table, then ask before posting anything + +Always finish **one markdown table English**, exact field order, regardless which branch taken: + +``` +| Field | Value | +|---|---| +| Package | | +| Current version | | +| Target version | | +| CI checks | <✅ All passed / ❌ N failing, with a short reason> | +| Actual installed version | | +| Actual ≥ Target | <✅ Yes / ❌ No / — Not verified> | +``` + +Right after table, add prose explain *why* — quote hardcoded pin line if `check_workflow_pin.sh` found one, quote `pip install` conflict if `check_pip_resolution.sh` reported one, or give root-cause classification from Step 3 if CI failed — then one line: **Recommendation: Ready to merge.** or **Recommendation: Needs human review.** `pip_resolution=conflict` result always force "Needs human review," even CI passed and is_covered Yes. See [examples/ready_to_merge.md](examples/ready_to_merge.md) and [examples/escalation_comment.md](examples/escalation_comment.md) full worked examples this exact format. + +Then **ask user**, every time, post this as comment on PR? — e.g. "Would you like me to post this as a comment on PR #923?". Only call `gh pr comment --repo --body "..."` if yes say. Never post auto, never call `gh pr merge` regardless recommendation — merge human's call after read this. + +Present draft to user. Don't post anywhere. + +## Scripts reference + +| Script | Purpose | +|---|---| +| `scripts/run_checkpoint.sh ` | Step 1: package, current/target version, CI pass/fail | +| `scripts/parse_pr.sh ` | Just dependabot body/diff parse part step 1 | +| `scripts/check_ci.sh ` | Just CI-checks part step 1 | +| `scripts/check_pip_resolution.sh ` | Step 2 (only if `setup.py` changed, regardless CI outcome): bumped pin pip-installable alongside sibling pins same requirements list? | +| `scripts/check_actual_version.sh ` | Step 4: actual installed version from pytest.yml build log, plus `proof_url` link to that job | +| `scripts/check_workflow_pin.sh ` | Step 4: workflow hardcode package to older version? | +| `scripts/compare_versions.sh ` | Step 4: is_covered (PEP 440 version comparison) | + +All scripts default to real PR repo when pass `ev-flow/quark-engine` +as `` — that's where every dependabot PR this project live, not user fork. + +## Validated against + +Logic walked manual against five real dependabot PRs before write down here — keep as sanity check if change logic: + +| PR | Package | Outcome | +|---|---|---| +| [#923](https://github.com/ev-flow/quark-engine/pull/923) | idna | Ready to merge — CI green, actual version match target, no pin found | +| [#922](https://github.com/ev-flow/quark-engine/pull/922) | langchain-core | Needs human — CI green but `pytest.yml:53` pin older version; target never install. Also: `pip install` for post-PR `setup.py` (`langchain==0.2.11 langchain-core==1.3.3 langchain-openai==0.1.17`) raise `ResolutionImpossible` — broken regardless CI | +| [#921](https://github.com/ev-flow/quark-engine/pull/921) | langchain | Needs human — CI red, root cause unrelated sample-download flake. Also independently fail `check_pip_resolution.sh`: `langchain==0.3.30` conflict untouched sibling pin `langchain-core==0.2.23` — broken even after clean re-run | +| [#906](https://github.com/ev-flow/quark-engine/pull/906) | pytest (Pipfile.lock) | Needs human — CI red, but root cause smoke-test baseline drift from stale branch, unrelated pytest | +| [#893](https://github.com/ev-flow/quark-engine/pull/893) | requests (Pipfile.lock) | Ready to merge — CI green, actual version (resolved transitively) exceed target, no pin found | + +`check_pip_resolution.sh` step (and `ResolutionImpossible` finding for #922 above) added after eval comparison vs baseline agent without this skill — it independently ran `pip install` on post-PR `setup.py`, caught more definitive, CI-independent reason block PR. See `evals/` this skill dir for eval set validated against. diff --git a/.claude/skills/dependency-update-pr-validator/evals/evals.json b/.claude/skills/dependency-update-pr-validator/evals/evals.json new file mode 100644 index 000000000..ca6ba2797 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/evals/evals.json @@ -0,0 +1,26 @@ +{ + "skill_name": "dependency-update-pr-validator", + "evals": [ + { + "id": 0, + "eval_name": "clean-ready-to-merge", + "prompt": "Please use the dependency-update-pr-validator skill to check ev-flow/quark-engine PR #923 (https://github.com/ev-flow/quark-engine/pull/923) and tell me whether it's ready to merge.", + "expected_output": "Package idna, before 3.11, target 3.15, CI all passed, actual installed version 3.15, is_covered Yes, recommendation Ready to merge, ends by asking whether to post a PR comment.", + "files": [] + }, + { + "id": 1, + "eval_name": "structural-version-pin", + "prompt": "Can you validate this dependabot PR for quark-engine? https://github.com/ev-flow/quark-engine/pull/922 — is it safe to merge?", + "expected_output": "Package langchain-core, before 0.2.23, target 1.3.3, CI all passed, actual installed version still 0.2.23, is_covered No, explanation references the hardcoded pin in pytest.yml (line ~53), recommendation Needs human review, ends by asking whether to post a PR comment.", + "files": [] + }, + { + "id": 2, + "eval_name": "unrelated-ci-failure", + "prompt": "There's a dependabot PR for quark-engine that's failing CI, can you look into why? https://github.com/ev-flow/quark-engine/pull/921", + "expected_output": "Package langchain, before 0.2.11, target 0.3.30, CI failing, root cause identified as an unrelated sample-download/zip failure (not the langchain bump), is_covered not verified, recommendation Needs human review (suggest re-run/rebase), ends by asking whether to post a PR comment.", + "files": [] + } + ] +} diff --git a/.claude/skills/dependency-update-pr-validator/examples/escalation_comment.md b/.claude/skills/dependency-update-pr-validator/examples/escalation_comment.md new file mode 100644 index 000000000..663906210 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/examples/escalation_comment.md @@ -0,0 +1,107 @@ +# Example: Escalation comment drafts + +## Case A — structural version pin (PR #922, langchain-core) + +CI was green, but the green result is misleading. + +``` +| Field | Value | +|---|---| +| Package | langchain-core | +| Current version | 0.2.23 | +| Target version | 1.3.3 | +| CI checks | ✅ All passed (push is skipping, not a failure) | +| Actual installed version | 0.2.23 (unchanged!) | +| Actual ≥ Target | ❌ No | +``` + +`pytest.yml` line 53 hardcodes +`pip install langchain==0.2.11 langchain-core==0.2.23 langchain-openai==0.1.17`, +independent of what `setup.py` declares. This PR only bumped `setup.py`, +so CI kept installing and testing the old version 0.2.23. The green CI +run does not validate langchain-core 1.3.3 at all. + +On top of that, `check_pip_resolution.sh` shows the bump is broken on its +own terms, independent of CI: `setup.py`'s `quarkAgentRequirements` list +now reads `langchain==0.2.11 langchain-core==1.3.3 langchain-openai==0.1.17`, +and `pip install` for that exact combination fails — +`langchain==0.2.11` requires `langchain-core<0.3.0,>=0.2.23`, which +`1.3.3` violates: + +``` +ERROR: Cannot install langchain-core==1.3.3 and langchain==0.2.11 because +these package versions have conflicting dependencies. +ERROR: ResolutionImpossible +``` + +Anyone who runs `pip install -e ".[QuarkAgent]"` after this merges gets a +hard failure. This isn't a CI gap, it's the PR's own diff being internally +inconsistent — `langchain` and `langchain-openai` need to move together +with `langchain-core`, not as an isolated single-pin bump. + +**Recommendation: Needs human review.** + +Would you like me to post this as a comment on PR #922? + +## Case B — CI failure unrelated to the bump (PR #921, langchain) + +``` +| Field | Value | +|---|---| +| Package | langchain | +| Current version | 0.2.11 | +| Target version | 0.3.30 | +| CI checks | ❌ 1 failing (`build`, pytest.yml) | +| Actual installed version | Not verified (CI failed before this check) | +| Actual ≥ Target | — Not verified | +``` + +All 461 test errors trace back to one fixture failing to download a valid +zip/APK file in `tests/core/test_apkinfo.py` +(`zipfile.BadZipFile: File is not a zip file` loading +`13667fe3b0ad496a0cd157f34b7e0c991d72a4db.apk`), not to anything in +`langchain`. This looks like a one-off sample-download flake rather than a +regression from the version bump. + +This PR also changes `setup.py`, so `check_pip_resolution.sh` ran +independently of the CI result above (it's a static check, not tied to +whether CI happened to pass) — and it also found a conflict: +`langchain==0.3.30` together with the untouched sibling pin +`langchain-core==0.2.23` is `ResolutionImpossible`. So even after the CI +flake is resolved with a clean re-run, this PR would still need +`langchain-core` bumped alongside `langchain` before it's mergeable. + +**Recommendation: Needs human review** — suggest re-running CI (or +rebasing the branch) to confirm the flake, and bumping `langchain-core` +together with `langchain` to fix the resolution conflict either way. + +Would you like me to post this as a comment on PR #921? + +## Case C — stale branch baseline drift (PR #906, pytest via Pipfile.lock) + +``` +| Field | Value | +|---|---| +| Package | pytest | +| Current version | 9.0.2 | +| Target version | 9.0.3 | +| CI checks | ❌ 9 failing (`smoke_test.yml`, all OS/Python matrix entries) | +| Actual installed version | Not verified (CI failed before this check) | +| Actual ≥ Target | — Not verified | +``` + +This PR only changes `Pipfile.lock`, which no workflow in this repo +actually installs from (no workflow runs `pipenv install`). The smoke +test failure is a behavior-count mismatch on a malware sample +(`Check Ahmyth Result` got 40, expected 39) — the kind of failure caused +by master adding new detection rules after this dependabot branch was +cut, not by a pytest/pygments/tomli version bump. + +**Recommendation: Needs human review** — suggest rebasing onto current +master and re-running before deciding. + +Would you like me to post this as a comment on PR #906? + +All three end with the explicit question. The skill never calls +`gh pr comment` or `gh pr merge` itself — it only posts if the human says +yes. diff --git a/.claude/skills/dependency-update-pr-validator/examples/ready_to_merge.md b/.claude/skills/dependency-update-pr-validator/examples/ready_to_merge.md new file mode 100644 index 000000000..ce3526b36 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/examples/ready_to_merge.md @@ -0,0 +1,30 @@ +# Example: Ready to merge + +Based on PR #923 (`ev-flow/quark-engine`), bumping `idna`. + +``` +| Field | Value | +|---|---| +| Package | idna | +| Current version | 3.11 | +| Target version | 3.15 | +| CI checks | ✅ All passed (push is skipping, not a failure) | +| Actual installed version | 3.15 ([Found in CI log](https://github.com/ev-flow/quark-engine/actions/runs/24300403080/job/70952658255)) | +| Actual ≥ Target | ✅ Yes | +``` + +CI ran the full test suite against idna 3.15 (confirmed via the +`Successfully installed ... idna-3.15 ...` line in the pytest.yml build +log) and passed. No workflow hardcodes an older idna version, so this +result reflects the target version, not a stale cache. The proof link +points straight at the `build` job log, so a human can verify the +install line without re-running anything. + +**Recommendation: Ready to merge.** + +Would you like me to post this as a comment on PR #923? + +This is the full turn output — the table, the explanation, the +recommendation, then the explicit question. The skill never calls +`gh pr comment` or `gh pr merge` on its own; it only posts if the human +says yes to that last question. diff --git a/.claude/skills/dependency-update-pr-validator/scripts/check_actual_version.sh b/.claude/skills/dependency-update-pr-validator/scripts/check_actual_version.sh new file mode 100755 index 000000000..09515071f --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/scripts/check_actual_version.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Find the actual installed version of a package, as proven by the `build` +# job log (pytest.yml), which is the only workflow that runs `pip install .` +# against this repo's real install_requires (setup.py). Pipfile/Pipfile.lock +# changes are NOT consumed by any current CI workflow — see SKILL.md. +# +# Usage: check_actual_version.sh +set -euo pipefail + +REPO="$1" +HEAD_REF="$2" +PACKAGE="$3" + +RUN_ID=$(gh run list --repo "$REPO" --workflow=pytest.yml --branch="$HEAD_REF" \ + --json databaseId,conclusion -q '.[0]') + +if [ -z "$RUN_ID" ] || [ "$RUN_ID" = "null" ]; then + echo "actual_version=UNKNOWN" + echo "reason=no pytest.yml run found for branch $HEAD_REF" + exit 0 +fi + +DB_ID=$(echo "$RUN_ID" | python3 -c 'import json,sys; print(json.load(sys.stdin)["databaseId"])') +CONCLUSION=$(echo "$RUN_ID" | python3 -c 'import json,sys; print(json.load(sys.stdin)["conclusion"])') + +echo "run_id=$DB_ID" +echo "run_conclusion=$CONCLUSION" + +JOB_ID=$(gh run view "$DB_ID" --repo "$REPO" --json jobs -q '.jobs[] | select(.name=="build") | .databaseId' | head -1) +if [ -n "$JOB_ID" ]; then + echo "proof_url=https://github.com/${REPO}/actions/runs/${DB_ID}/job/${JOB_ID}" +fi + +if [ "$CONCLUSION" != "success" ]; then + echo "actual_version=UNKNOWN" + echo "reason=pytest.yml run did not succeed, install log is not a reliable signal" + exit 0 +fi + +VERSION=$(gh run view "$DB_ID" --repo "$REPO" --log 2>/dev/null \ + | grep -i "Successfully installed" \ + | grep -oE "(^|[[:space:]])${PACKAGE}-[0-9][0-9A-Za-z.+-]*" \ + | head -1 \ + | sed -E "s/^[[:space:]]*${PACKAGE}-//") + +if [ -z "$VERSION" ]; then + echo "actual_version=UNKNOWN" + echo "reason=package not found in 'Successfully installed' line (it may not be installed by pip install ., e.g. only declared in Pipfile)" +else + echo "actual_version=$VERSION" +fi diff --git a/.claude/skills/dependency-update-pr-validator/scripts/check_ci.sh b/.claude/skills/dependency-update-pr-validator/scripts/check_ci.sh new file mode 100755 index 000000000..94350317c --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/scripts/check_ci.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Summarize all status checks for a PR. A PR counts as "all passed" when +# every check is pass or skipping (skipping is normal — e.g. kali-package.yml +# only runs on push to master, so it's always skipped on a PR). +# +# Usage: check_ci.sh +set -euo pipefail + +REPO="$1" +PR="$2" + +CHECKS=$(gh pr checks "$PR" --repo "$REPO" 2>/dev/null) || true + +if [ -z "$CHECKS" ]; then + echo "all_passed=unknown" + echo "reason=no checks reported yet" + exit 0 +fi + +FAILING=$(echo "$CHECKS" | awk -F'\t' '$2 != "pass" && $2 != "skipping" {print}') + +if [ -z "$FAILING" ]; then + echo "all_passed=yes" +else + echo "all_passed=no" + echo "--- failing checks ---" + echo "$FAILING" +fi diff --git a/.claude/skills/dependency-update-pr-validator/scripts/check_pip_resolution.sh b/.claude/skills/dependency-update-pr-validator/scripts/check_pip_resolution.sh new file mode 100755 index 000000000..488d7bed5 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/scripts/check_pip_resolution.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Only meaningful when the PR changed setup.py. A dependabot PR can bump one +# pin in a requirements list (e.g. quarkAgentRequirements) while leaving its +# sibling pins untouched — that combination can be mutually unsatisfiable +# even though pytest.yml's hardcoded install step never notices (it installs +# its own separate hardcoded versions, see check_workflow_pin.sh). This is a +# stronger, more definitive signal than is_covered: if pip itself can't +# resolve the post-PR setup.py, the bump is broken for any real user +# regardless of what CI happened to test. +# +# Scoped to just the requirement list that declares , so an +# unrelated pre-existing conflict elsewhere in setup.py doesn't get blamed +# on this PR. +# +# Usage: check_pip_resolution.sh +set -euo pipefail + +REPO="$1" +HEAD_REF="$2" +PACKAGE="$3" + +TMP_SETUP=$(mktemp) +trap 'rm -f "$TMP_SETUP"' EXIT + +gh api -H "Accept: application/vnd.github.raw" \ + "repos/$REPO/contents/setup.py?ref=$HEAD_REF" > "$TMP_SETUP" + +REQ_LINE=$(python3 - "$TMP_SETUP" "$PACKAGE" <<'PYEOF' +import ast, sys + +path, package = sys.argv[1], sys.argv[2] +with open(path) as f: + tree = ast.parse(f.read()) + +for node in ast.walk(tree): + if isinstance(node, ast.List): + items = [n.value for n in node.elts if isinstance(n, ast.Constant) and isinstance(n.value, str)] + if not items or not all(any(op in s for op in ("==", ">=", "<=")) for s in items): + continue + names = [s.split("==")[0].split(">=")[0].split("<=")[0].strip() for s in items] + if package in names: + print(" ".join(items)) + break +PYEOF +) + +if [ -z "$REQ_LINE" ]; then + echo "pip_resolution=skipped" + echo "reason=$PACKAGE not found in any pinned requirement list in setup.py" + exit 0 +fi + +echo "checking: $REQ_LINE" +if OUT=$(pip install --dry-run --quiet $REQ_LINE 2>&1); then + echo "pip_resolution=ok" +else + echo "pip_resolution=conflict" + echo "$OUT" | tail -15 +fi diff --git a/.claude/skills/dependency-update-pr-validator/scripts/check_workflow_pin.sh b/.claude/skills/dependency-update-pr-validator/scripts/check_workflow_pin.sh new file mode 100755 index 000000000..35f6d5a7b --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/scripts/check_workflow_pin.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Look for a hardcoded version pin of inside this repo's GitHub +# Actions workflow files (e.g. pytest.yml has a standalone +# `pip install langchain==0.2.11 langchain-core==0.2.23 langchain-openai==0.1.17` +# step that is NOT driven by setup.py/Pipfile). If such a pin exists, it +# silently overrides whatever version setup.py/Pipfile declares, so bumping +# the declared version alone will never change what CI actually installs. +# +# Checks the local working tree's workflow files (current master/branch +# state), which is a reasonable approximation since workflow files rarely +# change between a dependabot PR and master. +# +# Usage: check_workflow_pin.sh +set -euo pipefail + +PACKAGE="$1" +REPO_ROOT="$(git rev-parse --show-toplevel)" + +MATCHES=$(grep -rnE "(^|[^A-Za-z0-9_-])${PACKAGE}==[0-9]" "$REPO_ROOT/.github/workflows/" || true) + +if [ -z "$MATCHES" ]; then + echo "workflow_pin=none" +else + echo "workflow_pin=found" + echo "$MATCHES" +fi diff --git a/.claude/skills/dependency-update-pr-validator/scripts/compare_versions.sh b/.claude/skills/dependency-update-pr-validator/scripts/compare_versions.sh new file mode 100755 index 000000000..46b11dbb8 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/scripts/compare_versions.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# is_covered = actual installed version >= target version (PEP 440 semantics). +# +# Usage: compare_versions.sh +set -euo pipefail + +ACTUAL="$1" +TARGET="$2" + +python3 -c " +from packaging.version import Version +import sys +try: + print('yes' if Version('$ACTUAL') >= Version('$TARGET') else 'no') +except Exception as e: + print('unknown', file=sys.stderr) + sys.exit(1) +" diff --git a/.claude/skills/dependency-update-pr-validator/scripts/parse_pr.sh b/.claude/skills/dependency-update-pr-validator/scripts/parse_pr.sh new file mode 100755 index 000000000..0d1f99746 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/scripts/parse_pr.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Parse a dependabot dependency-update PR: package name, current/target +# version, and which files it touched. Which file changed (setup.py vs +# Pipfile/Pipfile.lock) does NOT tell you whether CI actually installs the +# target version — some setup.py-declared packages are overridden by a +# hardcoded pin elsewhere in the workflow (see check_workflow_pin.sh), and +# some Pipfile-only packages are unconstrained transitive deps that CI +# happily floats to the latest version anyway. Use check_actual_version.sh +# + check_workflow_pin.sh for that, not this script's changed_files. +# +# Usage: parse_pr.sh +set -euo pipefail + +REPO="$1" +PR="$2" + +TMP_BODY=$(mktemp) +trap 'rm -f "$TMP_BODY"' EXIT + +gh pr view "$PR" --repo "$REPO" --json body -q '.body' > "$TMP_BODY" +FILES=$(gh pr view "$PR" --repo "$REPO" --json files -q '.files[].path') +HEAD_REF=$(gh pr view "$PR" --repo "$REPO" --json headRefName -q '.headRefName') + +python3 - "$TMP_BODY" <<'PYEOF' +import re, sys + +with open(sys.argv[1]) as f: + body = f.read() + +m = re.search(r"Bumps \[([^\]]+)\].*?\bfrom\s+(\S+)\s+to\s+(\S+)\.", body, re.DOTALL) +if m: + print(f"package={m.group(1)}") + print(f"current_version={m.group(2)}") + print(f"target_version={m.group(3)}") +else: + print("package=UNKNOWN") + print("current_version=UNKNOWN") + print("target_version=UNKNOWN") +PYEOF + +echo "head_ref=$HEAD_REF" +echo "changed_files=$(echo "$FILES" | paste -sd, -)" diff --git a/.claude/skills/dependency-update-pr-validator/scripts/run_checkpoint.sh b/.claude/skills/dependency-update-pr-validator/scripts/run_checkpoint.sh new file mode 100755 index 000000000..0a2f04bc4 --- /dev/null +++ b/.claude/skills/dependency-update-pr-validator/scripts/run_checkpoint.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Prints the cheap checkpoint (no log digging): target package, before +# version, target version, and whether all CI checks passed. Run this first +# and show the result to the user before moving on to the slower step of +# digging through CI logs. +# +# Usage: run_checkpoint.sh +set -euo pipefail + +REPO="$1" +PR="$2" + +echo "=== PR info ===" +gh pr view "$PR" --repo "$REPO" --json title -q '.title' +"$(dirname "$0")/parse_pr.sh" "$REPO" "$PR" + +echo "=== CI status ===" +"$(dirname "$0")/check_ci.sh" "$REPO" "$PR"