Skip to content

Move paid CI runners and harden workflows - #3240

Draft
alexnorell wants to merge 5 commits into
mainfrom
alex/gha-cost-runners
Draft

alexnorell wants to merge 5 commits into
mainfrom
alex/gha-cost-runners

Conversation

@alexnorell

@alexnorell alexnorell commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Moves the paid CI jobs to cheaper runners and applies a security and efficiency baseline across the workflows.

Runner changes

  • GPU jobs move from Roboflow-GPU-VM-Runner to the self-hosted arc-gpu-runner-set (one NVIDIA L4 per job): test.nvidia_t4.yml, test.nvidia_t4_parallel_server.yml, test.colab.gpu.yml, unit_tests_inference_experimental_gpu.yml, integration_tests_inference_experimental_gpu.yml, e2e_tests_inference_experimental_gpu.yml, test_package_install_inference_gpu.yml and test_package_install_inference_gpu_with_extras.yml. .github/actionlint.yaml now accepts the new label instead of the old one.
  • The two jobs on the GitHub 8-core larger runner (integration_tests_inference_server_x86.yml, test.colab.cpu.yml) move to depot-ubuntu-22.04-8 in the public-depot group, matching the other Depot jobs here.
  • ubuntu-latest, macOS, Windows and Jetson jobs are unchanged.

Fork pull requests and self-hosted runners
Every workflow that now uses arc-gpu-runner-set already triggers only on push to main, workflow_dispatch, or workflow_call from dispatch-only or push-only callers. No pull request, and so no fork pull request, can start a job on the self-hosted GPU runners. This PR keeps that as the rule; there is no behaviour change for pull requests.

Security baseline

  • Every action reference in .github/ is pinned to a full commit SHA with an exact # vX.Y.Z comment (about 300 references in 70 files). Tags were resolved to the commit they point at today, so no action version changes.
  • actions/checkout sets persist-credentials: false everywhere; no workflow pushes with the checkout token.
  • Secrets that were interpolated into run: scripts in the touched GPU workflows and in integration_tests_workflows_x86.yml are passed through env: instead.
  • The Colab workflows no longer check out github.head_ref; they only run on push and dispatch, where it is empty.
  • The six test_package_install_* workflows drop an unused workflow-level id-token: write.
  • publish.pypi.yml no longer restores a pip cache while building release wheels.
  • .github/dependabot.yml is added with a monthly, grouped github-actions update and a seven-day cooldown, so the pins stay current.

Efficiency

  • Every job now sets timeout-minutes (15 jobs were missing one). The three self-hosted Jetson test jobs are left unchanged.
  • integration_tests_inference_experimental_cpu.yml skips documentation-only changes, using the same paths-ignore list as the other test workflows.

CLAUDE.md gains a short GitHub Actions section so these rules hold for new workflows.

Type of change

  • Other: CI infrastructure

Testing

  • actionlint 1.7.12 with .github/actionlint.yaml (the version CI uses): no findings.
  • python .github/scripts/check_workflow_concurrency.py: passes.
  • zizmor 1.31.1 on .github/workflows: unpinned-uses findings go from 246 to 0, artipacked from 69 to 0, excessive-permissions (high) from 9 to 3, cache-poisoning from 2 to 1. The rest are pre-existing and listed below.
  • main was merged into this branch at 95d8875 (merge commit 6d996b6). The one conflict was the Windows CPU installer timeout, where this branch keeps the 120 minutes main chose. The new validate-windows-bundle.yml from main got the same pinning comments and persist-credentials: false.
  • After the merge: actionlint and the concurrency check pass, the .github/scripts unit tests that Validate GitHub Actions runs pass (129 tests), and every non-Jetson job has timeout-minutes.
  • All pull request checks at 6d996b6 pass (96 pass, 3 skipped), including Validate Windows bundles (frozen (cpu) and frozen (gpu)).
  • The two moved CPU workflows do not run on pull requests, so they were dispatched on this branch: Google Colab CPU (run 39726926269, 6 min) and INTEGRATION TESTS - Inference Server CPU x86 with use-inference-models=true (run 39726928550, 10 min) both passed, on runners depot-4v5snvzchq and depot-n5lr53sm9d in the public-depot group.
  • The GPU workflows do not run on pull requests either. They need a workflow_dispatch on this branch once arc-gpu-runner-set accepts jobs.

Additional context

  • This stays a draft because arc-gpu-runner-set is not available. As of 2026-10-27 the runner controller is running, but no runner scale set with that name is installed and no L4 nodes are up, so a job with that label cannot start. A smoke-test job on the label was cancelled after 14 minutes in the queue. The runner set must be installed before merge, otherwise every GPU job on main would wait in the queue until GitHub cancels it.
  • Jobs that start GPU containers (make start_test_docker_gpu, docker run --gpus all, --runtime nvidia) in test.nvidia_t4.yml, test.nvidia_t4_parallel_server.yml and test.colab.gpu.yml need the runner's Docker daemon to expose the GPU. Confirm this with a dispatch before merging; the inference-models GPU and package-install jobs only use the GPU from the job itself.
  • The new runners give each job a fresh pod, so nothing written to the runner disk survives between jobs. The persistent model cache that server-tests-gpu gained on main (Persist model weights on the GPU test runner between stages and runs #3232) keeps working there, but it starts empty in every job, so the T4 suite downloads model weights on each run.
  • Not changed, for follow-up: workflow-level write permissions in claude-pr-review.yml and fast_track_ci.yml (both gate to same-repository PRs and read PR code only as data); the archived actions/upload-release-asset in the macOS and Windows installer workflows; the setup-node cache in docs.yml, which also deploys on release; secrets: inherit on reusable-workflow calls.

Move GPU jobs from the GitHub-hosted GPU runner to arc-gpu-runner-set and
the two GitHub 8-core larger-runner jobs to depot-ubuntu-22.04-8. Pin all
actions to commit SHAs with exact version comments, disable persisted
checkout credentials, add missing job timeouts, pass secrets to run steps
through env, drop unused id-token permissions and the release pip cache,
and add a Dependabot entry for github-actions.
@github-actions

Copy link
Copy Markdown
Contributor

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚧 Right now this is a draft, so automated Claude review is paused — nothing is being spent yet. Mark it Ready for review to trigger it.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

  • 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
  • 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

  • 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
  • ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
  • 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
  • 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

  • Prefer to skip automated review entirely? Add the skip-claude-review label.

Keep the 120-minute timeout that main set for the Windows CPU installer
job, and apply the pinned version comments and persist-credentials: false
to the new Windows bundle validation workflow.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant