Repository navigation
Capability gap: scheduled runs cannot build a trusted pull request to try it as a user #4366
Description
🤖 Generated by the Agentic Engineer
Problem
A scheduled run cannot build a pull request's own code on the agent host to try it as a user. On 2026-10-20 the permission layer refused go build of a trusted, devantler-authored KSail pull request (devantler-tech/ksail#7873, head 9b233ae7) as code from an external source. An earlier run built the same pull request without a refusal, so the outcome is inconsistent between runs.
Readiness condition 3 (tried and evaluated as a user) needs that build for any change with a command-line surface. Without it, an otherwise ready pull request (green checks, a green review at its head, no open threads) has to stay a draft.
What is needed
A reviewed way for a scheduled run to build and run a trusted-author pull request head locally, for example a permission rule scoped to go build and the built binary inside the run's own worktree and scratchpad. The trust gate stays as it is: untrusted authors' branches are still never built.
Acceptance criteria
- A scheduled run can build a trusted-author KSail pull request head inside its own worktree and run the binary from its scratchpad without a refusal.
- An untrusted author's branch is still refused.
Activity
devantler commentedon Oct 20, 2026
Second data point, 2026-10-20 04:30 UTC (scheduled Claude run): the refusal now comes one step earlier. Fetching the pull request's branch into the run's own isolated KSail checkout and checking out its head (
9b233ae7, authordevantler) was refused before any build started. So the gap covers checking out a trusted pull request head at all, not onlygo build. I did not retry. devantler-tech/ksail#7873 stays parked on this issue.chore: agent-claim #4366 nonce=7857196bbfde4a95a5e8a3533746831a
devantler commentedon Oct 20, 2026
Scoped Codex observation on 2026-10-20: fetching the trusted KSail PR #7873 branch and checking out its exact current head in a claimed, isolated worktree both succeeded. A bounded
go buildwas also permitted and entered dependency resolution; it was not refused by the approval layer.The build exited unsuccessfully with missing-module and missing-checksum diagnostics before producing a binary. Local disk space then fell below the required build floor, so further build and CLI trials stopped. This is an unresolved local validation prerequisite, not evidence that the feature works or that the PR is ready.
No permission rule or installed plugin was changed. This observation narrows the previously reported refusal to the affected runtime/run; it does not prove recovery for Claude, the negative untrusted-author control, or the required end-to-end CLI acceptance. Keep this issue and the PR's user-evaluation gate open until those checks are demonstrated.
devantler commentedon Oct 21, 2026
Another instance of the same family, 2026-10-21 ~19:00Z (scheduled run
agentic-engineer-2). The permission layer refused, as Untrusted Code Integration, a read-only step diagnosing a trusted Dependabot pull request: listing which files the failedgolangci-lintauto-fix wanted to change (from that job's saved CI log) next to the pull request's file list (devantler-tech/ksail#7915). Nothing was built, run or pushed.Effect: the run could not tell whether the lint failure belongs to the dependency bump or to
main. That PR stays unrepaired this run. In-rules path stays the one proposed above: a scoped rule that lets a scheduled run read CI output and diff metadata for trusted-author pull requests (Dependabot, Renovate,devantler).devantler commentedon Oct 22, 2026
New evidence, 2026-10-22 ~04:15Z (scheduled engineer run). I was trying to evaluate devantler-tech/.github#704 as a user before promoting it. That PR retires two disabled .NET
OrganizationRulesets. Its user-facing effect is what Crossplane accepts and reconciles. These two steps were refused:kubectl apply --dry-run=serverof the two rendered rulesets against the live CRD. It is read-only by design: server-side dry run persists nothing. It is the closest pre-merge proof of what the provider will accept.pr-body-contract.sh check), chained after a read of the PR body. The refusal gave no reason.Effect: the PR has a green review and CI at its head and its rendered output was inspected. Without a pre-merge acceptance proof, I left it as a draft rather than promote another lane's live org-policy change.
Proposed in-rules path: a permission rule that allows
kubectl apply --dry-run=serveronly, with the prod context and no other apply form, pluspr-body-contract.sh check. Both are read-only.devantler commentedon Oct 25, 2026
Same refusal again, 2026-10-25 ~07:10Z (scheduled engineer run). I tried to build devantler-tech/ksail#7964 at head
187af7a3and run its chat package tests, to try the new Copilot sign-in advice as a user before promoting it. The permission layer refused the build as code from an external source. The pull request is trusted (devantler), its CI is green and CodeRabbit is green at that head.Effect: devantler-tech/ksail#7964 stays a draft, because readiness condition 3 (tried as a user) is not met. Nothing else is blocked.