Skip to content

Run the PR risk classifier with pi instead of Claude Code - #29

Merged
rgarcia merged 1 commit into
mainfrom
hypeship/pr-risk-pi
Nov 6, 2026
Merged

rgarcia merged 1 commit into
mainfrom
hypeship/pr-risk-pi

Conversation

@rgarcia

@rgarcia rgarcia commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Moves the reusable pr-risk.yml classifier from Claude Code CLI to pi (@earendil-works/pi-coding-agent), following the org pattern: Node 22, PI_PREFERRED_MODEL, and Anthropic/OpenAI keys.

This is a read-only workflow (it only comments), so the credential split stays the same. Only the agent invocation changes.

  • Isolation is unchanged. pi runs in an empty directory with --no-tools --no-mcp --no-extensions --no-skills --no-prompt-templates --no-context-files --no-approve --no-session. Its only credentials are the LLM keys. The kernel-internal token is still minted after the model finishes.
  • The output contract is unchanged. pi has no --json-schema, so the prompt now asks for one JSON object with exactly the five keys. A new pr_risk.py verdict step reads pi's --mode json event stream and writes the same result.json envelope Claude Code produced (is_error / result / structured_output). It checks the reply strictly against VERDICT_SCHEMA: required keys, types, enums, and no extra keys. Provider errors, truncated replies, non-JSON and off-schema replies all become is_error: true, which the unchanged parse_verdict / comment path already renders as "Not assessed". The comment format and the pr-risk:v1 marker do not change.
  • Model input. New model input, defaulting to ${{ vars.PI_PREFERRED_MODEL }}. It replaces vars.CLAUDE_CODE_PREFERRED_MODEL. An empty model fails the step with a clear message instead of falling back to a CLI default.
  • Secrets. OPENAI_API_KEY is added, and both LLM keys are now optional, so callers pass the key for their model's provider. Making ANTHROPIC_API_KEY optional keeps existing callers valid.
  • Thinking and budget. --max-budget-usd 1 has no pi equivalent. Thinking is set to high rather than max to keep a one-shot classification of a PR under 50 lines cheap. A local run cost about $0.02.
  • README: updated the pr-risk section only (caller example and model note). I left the shared "Required secrets/variables" lists alone because they also cover workflows being migrated in parallel.

Caller changes required

PI_PREFERRED_MODEL currently resolves to an OpenAI model. Until callers pass OPENAI_API_KEY, eligible PRs will get "Not assessed" or a failed classify job. These callers need a one-line addition, merged after this PR. Before then, passing an undeclared secret is a workflow error.

  • kernel/kernel .github/workflows/pr-risk.yml
  • kernel/infra .github/workflows/pr-risk.yml
  • kernel/hypeman .github/workflows/pr-risk.yml
  • kernel/kernel-images-private .github/workflows/pr-risk.yml
    secrets:
      ADMIN_APP_ID: ${{ secrets.ADMIN_APP_ID }}
      ADMIN_APP_PRIVATE_KEY: ${{ secrets.ADMIN_APP_PRIVATE_KEY }}
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Validation

  • actionlint (v1.7.7) on pr-risk.yml and pr-risk-tests.yml: clean, including vars in the workflow_call input default.
  • python -m unittest for scripts/test_pr_risk.py: 18 tests pass. New tests cover:
    • the event-stream → envelope conversion: success, fenced JSON, provider error, length stop, missing reply, non-JSON, extra/missing keys, wrong types, bad enums;
    • that conversion feeding comment;
    • the workflow's isolation flags, step env, secrets, and model default;
    • running the actual Classify step script against a fake pi: stdin, cwd, --model, system prompt, result.json, and the empty-model failure.
  • Mutation checks: removing --no-tools, disabling the extra-key check, or breaking the events path each fails a test. test_breakglass.py and test_vuln_remediation.py still pass.
  • Local live run with pi 1.5.0 and openai/gpt-5.6-sol, using a fresh HOME, on two real small merged infra PRs gathered with pr_risk.py gather. Both produced valid verdicts, and both were denied, consistent with the existing comments on those PRs. A bad API key produced is_error: true with the provider's 401 message.

Not run:

  • A live Actions run. The workflow is workflow_call only, triggered by caller PRs.
  • An Anthropic model. No Anthropic key was available locally.

Unverified: org secret/variable visibility for the four caller repos. I couldn't list org secrets/variables with the available permissions. PI_PREFERRED_MODEL's value was read from another repo's workflow log.

@rgarcia
rgarcia requested a review from tnsardesai November 5, 2026 21:18
@rgarcia

rgarcia commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Picked tnsardesai to review: touched the changed code in scripts; 9 pending

@rgarcia
rgarcia merged commit 3844214 into main Nov 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant