Application-agent PR review without prior setup: exact-ref comparison, tool bindings, and 10 verified cases

[pengfei-threemoonslab]

Current execution and evidence — PM review 2026-10-01

Primary track under #778; milestone 13 retains the Oct 14 evidence checkpoint. Latest measured baseline is the September 30 ledger in this discussion: 1 Q2 / 49 development cases, 9 PRs with rows, zero false-complete answers on main 6ced6f70. This is source-build development evidence, not a new released-build run or observed adoption.

1. Q2 measurement: commit the acceptance corpus, freeze a holdout, and report Q1/Q2 on every advisory release #908: preserve pins/runner/scoring and freeze the untouched holdout before the gated reader merges. Record its future-window scheduling constraint explicitly.
2. Application diff: resolve tools lists built by spread, concatenation, conditionals and comprehensions #909 list expressions + Application diff: resolve tool lists passed into agent-building functions and constructors #874 parameter/instance flow; keep their ownership separate. OpenAI Agents SDK: resolve bounded literal tool-list concatenation in static binding evidence #584's concatenation acceptance stays with Application diff: resolve tools lists built by spread, concatenation, conditionals and comprehensions #909.
3. Application diff: read tools bound as objects — MCP toolsets, agent-as-tool and hosted tools #910 object tools and built-ins, including Google ADK: preserve built-in load_memory identity and binding in capability diffs #866's load_memory regression.
4. Tool reach: name effects beyond HTTP — databases, subprocesses, files, cloud SDKs and messaging #913 observed effects + Application diff output: one finding per changed agent, the first unresolved hop, grouped uncertainty #914 concise reviewer findings.
5. Application diff: read agents derived with clone() as agents with their own bindings #911 clones and Application diff: identify agents whose name is not a literal (base-class template methods, computed names) #912 nonliteral identities, after the higher-yield shapes; their bounded identity work can proceed independently where documented.
6. GitHub Action: an application mode that posts the comparison as a PR comment #915 Action delivery can ship with current truthful text and adopt Application diff output: one finding per changed agent, the first unresolved hop, grouped uncertainty #914 later. Measure which agent frameworks real agent-changing PRs use, and decide the next reader (ADR) #916 measures framework mix without implementing a reader.

#580/#864/#865/#872/#875/#876 are closed delivered prerequisites. #655/#867 remain open for acceptance reconciliation: establish which residual clauses are unmet before rebuilding already-present comparison behavior. Checked prerequisite boxes below record issue completion only; they do not establish this epic's ten-case outcome.

Exit remains ten qualifying Q2 cases with separate development/holdout results on a released build. Any material false row or omission disqualifies its case. Preserve named partial coverage and static-only operation. No automatic semantic declarations, qualification claim or outreach authorization follows from this plan.

---

Selected execution — 2026-09-24

Days 1–5 rebaseline and design: PR #869. Fresh PyPI 1.1.0/contract 40 and main d9a6d0e/contract 41 produce the same limitations on the selected corpus: 27 PRs × two builds = 54 head scans, plus eight fresh exact-ref verifies on four representative PRs. Attest, Capstone and ScopeIQ have missing_manifest/zero top changes on both builds; Visulate refuses the mode-160000 Gitlink. The ledger includes pinned refs, scope/config identity, release wheel digest and dependency versions. This is zero qualifying cases, not a success-rate estimate, and does not close this epic. Selected milestone 13; #580/#655 design first, then #867 and bounded readers. See PR for the reviewed comparison/absence semantics.

---

The original engine baseline below is historical; PR #869 records the fresh released/main rebaseline. The end-to-end acceptance remains unchanged.

Outcome

Deliver useful, reproducible application-agent capability comparisons for real open-source PRs that have never committed an Agents Shipgate manifest. The report should tell a reviewer which agent's source-observed callable surface changed, show the evidence and coverage limits, and support a concrete review decision before project adoption.

This tracks the product direction requested on 2026-09-23: unconfigured-project comparison plus tool-binding coverage. Existing #655, #656, and #580 remain the owners of their implementation scope; this issue joins them to the newly reproduced reader and per-agent comparison work.

Evidence baseline

Engine: source commit 9df307ab11f8d17b9e342a8b451b29f96e62bb31, local version 1.0.0, contract 40.

The investigation screened 310 distinct public PRs and locally ran Agents Shipgate on 27 priority candidates. No candidate yet met the full ten-case selection standard. 310 is a screening count, not 310 local runs or 310 demonstrated product failures. These targeted observations are development evidence, not a representative success-rate estimate or the released-build value gate in #830.

Real application-agent change	Observed limitation	Work owner
attest#3 adds memory read/write tools to an existing agent	Exact-ref verify reports base_status=missing_manifest, empty top_changes; head reads only the three old local tools	#655, #864
capstone_project#3 adds SQL and memory tools	A manually supplied real base report enables nonempty SQL tool additions, but load_memory is unresolved	#655, #866
visulate-for-oracle#526 adds repository-memory tool factories	Head inventory is empty; factory variables are unresolved	#864, #865
scopeiq#2 introduces tools on two SDK agent objects	Five direct edges are already extracted, but an unresolved deployment root leaves root-scoped inventory empty	#867

The source changes above establish relevant scenarios. They are not claims that Agents Shipgate already delivered complete review value, nor claims of runtime vulnerabilities in those projects. Each reader issue records exact SHAs, commands, observations, and bounded acceptance.

Exact-ref comparison reproduction

For attest#3, after the scoped init --local-review in #864:

agents-shipgate verify --workspace subject   --config agents/attest_orchestrator/.agents-shipgate-local-review.yaml   --base 10e1f949d48b5349a180677c41c4ba30deab943d   --head cab295001ec01e7fca388acee67842c6f53b5150   --ci-mode advisory --format json

Observed base_status=missing_manifest, capability_review.top_changes=[]. Providing Git refs alone does not establish an application-tool comparison when the base has no manifest. #655 owns automatic comparison; #580 owns accurate added/relocated scope and absence semantics. The existing shipgate diff host-grant route does not cover this application source change.

Delivery sequence and dependencies

• shipgate diff v1 across Route A: manifest-free base synthesis; the capability delta is never unavailable on first adoption; verdict only as a footer #655 — manifest-free application tool-source base comparison, with reproducible evidence identity.
• Verifier: bind absent and relocated scoped base inputs without inventing an empty agent surface #580 — independently bound base/head scopes, including new or relocated agents. Missing manifest must not imply empty tools.
• Advisory application diff: compare observed tool bindings per agent when the deployment root is unresolved #867 — compare already-observed direct wiring per agent when deployment-root selection is unresolved.
• Google ADK: resolve repository-local imported functions and module-qualified tool bindings #864 — repository-local import and module-qualified function bindings.
• Google ADK: trace bounded FunctionTool factories into agent tool bindings #865 — bounded tool-factory return and registration chains; builds on import resolution.
• Google ADK: preserve built-in load_memory identity and binding in capability diffs #866 — exact ADK built-in identity and binding, initially load_memory.
• Apply Generated lock-state replaces the manifest as a prerequisite; insufficient_evidence leaves the headline #656's first-value separation: useful observed changes and explicit coverage limits must be available without filling semantic declarations. A whole lock-state redesign should not become a prerequisite for the bounded comparison increment.
• Run the end-to-end PR validation below and publish the reproducible evidence bundle.

Track additional obstacles explicitly during the rerun. Visulate's unrelated gitlink currently refuses verifier materialization (see #865); #430 and host fix #688 are relevant prior work. Do not count paired extraction as successful whole-PR verification while this remains unresolved. #584's SDK list-concatenation extension is adjacent work, not the cause of ScopeIQ's root-selection result.

End-to-end acceptance

• One documented local comparison flow accepts a repository/scope and exact base/head refs without committing a manifest or manually authoring a base report. Report both requested refs and the actual compared trees/merge base, engine identity, input scope, and reader coverage.
• Rows carry agent/tool identity, before/after, change kind/direction, source locations, and evidence basis. A reviewer can trace each conclusion to the specific PR. Stable output on identical inputs is verified.
• Known structural changes remain visible beside precise unresolved edges. Definitions without wiring cannot masquerade as agent capability; ambiguous deployment roots cannot erase already-observed per-agent wiring.
• Added/removed roots, moved files, changed bindings, unchanged tools, and unsupported dynamic paths have paired positive/negative regression cases. Incomplete reads cannot become “no changes” or a fabricated empty base.
• Preserve source evidence versus inference, declared authority versus observed wiring, and advisory comparison versus release authorization. Never auto-fill purpose, effects, authority, or reviewed agent_bindings merely to make a result pass.
• Validate 10 real application-agent PRs locally at pinned refs. For each, retain command/version, raw output, source-to-row reconciliation, coverage assessment, and the concrete reviewer decision the output enables. Verify the changed callable surface and direction; a material omission or false claim disqualifies the case.
• A standalone insufficient_evidence, a head-only scan, an empty diff, or coding-agent host grants do not satisfy the ten-case result. Useful structural comparison does not require a release pass, but must be accurate and complete for its stated changed scope.
• Publish attempted and qualifying counts honestly. The existing public PRs are regression candidates; do not assume they qualify after a parser repair, or generalize the result into market validation.

Related: #328 (reader limitations versus user configuration), #437 (bound-surface count semantics), #830 (separate released-build value gate). This tracker creates no outreach authorization and does not change existing release-control contracts.

[pengfei-threemoonslab]

Days 1–5 rebaseline and design: PR #869. Fresh PyPI 1.1.0/contract 40 and main d9a6d0e/contract 41 produce the same limitations on the selected corpus: 27 PRs × two builds = 54 head scans, plus eight fresh exact-ref verifies on four representative PRs. Attest, Capstone and ScopeIQ have missing_manifest/zero top changes on both builds; Visulate refuses the mode-160000 Gitlink. The ledger includes pinned refs, scope/config identity, release wheel digest and dependency versions. This is zero qualifying cases, not a success-rate estimate, and does not close this epic. Selected milestone 13; #580/#655 design first, then #867 and bounded readers. See PR for the reviewed comparison/absence semantics.

[pengfei-threemoonslab]

Implementation PR: #871 — diff --application compares SDK/ADK source-observed per-agent wiring at committed refs without init, a manifest, or authored declarations. This implementation does not depend on merging the optional planning PR #869.

Source-checked result on a fresh, unconfigured ScopeIQ #2 clone: exactly five added bindings (three on social_agent, python_exec and rag_query on synthesizer_agent), with both selected scopes completely read. PR #871 contains the exact refs, command and source links. Capstone #3 remains partial: two SQL additions are established, load_memory remains unresolved (#866), so it is not counted as complete.

This is a first SDK/ADK advisory-comparison increment, not completion of the ten-PR milestone. Legacy verifier-base synthesis, broader framework/import/factory coverage and released-build qualification remain open. No declarations or release permissions are inferred from the new result.

[pengfei-threemoonslab]

Owner decision 2026-09-25: order of work and acceptance corpus

Order. For these items, this supersedes the Days 1–5 sequence in #869. #580, #655 and #867 stay open.

1. Stop root-scope refusal on unrelated symlinks/gitlinks, and recognise from google.adk import Agent (in progress).
2. Key SDK identity on imports (LiveKit is not the OpenAI SDK); an unobserved agent is not a removal (fix(diff --application): key SDK identity on imports; unobserved agents are not removals #873).
3. Google ADK: resolve repository-local imported functions and module-qualified tool bindings #864 imported tools, for both the ADK and OpenAI SDK readers (in progress).
4. Application diff: resolve tool lists passed into agent-building functions and constructors #874 tool lists passed into agent-building functions. Application diff: derive the comparison scope from the change when --scope is omitted #875 (derive the scope from the change) runs in parallel once 1–3 land.
5. Google ADK: trace bounded FunctionTool factories into agent tool bindings #865 factories.
6. Application diff rows should name what a bound tool reaches: endpoint, action, credential, model-supplied arguments #872 effect evidence on each row (endpoint/action, credential name, model-supplied arguments).
7. Release.
8. Outreach.

Qualification. A case counts toward this issue's ten only at Q2. Each case is scored by hand against source at pinned refs.

• Q0 (relevant): the PR changes an SDK/ADK agent's tools, handoffs or sub-agents in application code.
• Q1 (correct and complete): every binding change on the affected agents appears, with no false rows, and compared is never reported while a binding is unobservable.
• Q2 (useful): Q1, the PR modifies an agent that exists at the base, and each changed row names what the tool reaches (Application diff rows should name what a bound tool reaches: endpoint, action, credential, model-supplied arguments #872) or names the unresolved hop.

Baseline at main a430e81a.

• 134 open PRs with SDK/ADK tool-wiring edits: 83 are real SDK/ADK changes. Q1: 1 (tensorflow#128063, a new directory, so it cannot reach Q2). Q2: 0.
• 50 real application PRs that modify an existing agent's wiring (merged or open, last 90 days): 3 of 42 root-scope runs produce any row. Q2: 0.
• Blockers across those 42 PRs:
  • dynamic tools expressions: 21 (Application diff: resolve tool lists passed into agent-building functions and constructors #874, Google ADK: trace bounded FunctionTool factories into agent tool bindings #865)
  • unresolved imported tools: 11 (Google ADK: resolve repository-local imported functions and module-qualified tool bindings #864)
  • a file using another framework in scope: 12
  • scope selection: the changed directory often holds no agent (Application diff: derive the comparison scope from the change when --scope is omitted #875)
• One false result, speechmatics-academy#142: LiveKit was read as the SDK, and four REMOVED rows appeared under compared. Addressed by fix(diff --application): key SDK identity on imports; unobserved agents are not removals #873.

Every corpus case is pinned to its merge base, head and scope. The corpus will be committed with the first rerun that moves a count.

[pengfei-threemoonslab]

Update to the order above. #876 is a release blocker, placed right after #873. It is a false complete answer: on kkmiecik-coder/CRM#5 the Wycena agent gains wyslij_obraz, and the result reads compared / no changes. The cause is that the SDK reader misses return Agent(...), and an agent in a test file makes the scope count as established. The 8 remaining root-scope runs of the 50-PR corpus have now completed. None produces a row: 3 exit 2 (two duplicate tool definitions, one gitlink), 4 are partial, and 1 is the silent compared above.

[pengfei-threemoonslab]

Q2 re-measure on main 6ced6f70, and the path to 10 cases (2026-09-30)

The 49 PRs that modify an existing SDK/ADK agent were re-pinned from merge-base to head and run with the derived scope (the default a new user gets). Every row was scored by hand against source.

	a430e81a (09-25)	6ced6f70 (09-30)
PRs with any row	3 / 42	9 / 49
Q2	0	1: jpka/attest#3, two memory tools added to an existing agent, with the unresolved hop named
False complete answers	1 (CRM#5)	0

Near misses:

• Merge KN branch with validator agent to dev Tiendat2703/MIS_TALENT#5–[codex] Harden post-v0.2 scanner edge cases #8: correct rows, but one agent's spread list [*FINANCE_TOOLS, …] is unread.
• Remove Textual UI and streamline CLI runtime xju2/hepagent#72: a removed wrapper agent is not shown.
• Harden O.R.I.O.N. security, release, and workspace workflows VidulaWickramasinghe/O.R.I.O.N#126: 176 uncertain rows.

Blockers across the 40 no-row PRs:

• dynamic tools expressions: 25;
• unresolved tools bound as objects: 13;
• unnamed agents: 11;
• clones: 9;
• builder parameters: 5.

The work that remains is coverage and usefulness. The comparison is now honest on every case.

Order of work (outreach excluded; it follows the exit criterion below):

1. Q2 measurement: commit the acceptance corpus, freeze a holdout, and report Q1/Q2 on every advisory release #908: commit the corpus, and freeze a holdout before any reader change below.
2. Application diff: resolve tools lists built by spread, concatenation, conditionals and comprehensions #909 dynamic tools expressions, together with Application diff: resolve tool lists passed into agent-building functions and constructors #874 (parameters, self.tools). This is the largest blocker, and Application diff: resolve tools lists built by spread, concatenation, conditionals and comprehensions #909 subsumes OpenAI Agents SDK: resolve bounded literal tool-list concatenation in static binding evidence #584.
3. Application diff: read tools bound as objects — MCP toolsets, agent-as-tool and hosted tools #910: tools bound as objects (MCP toolsets, agent-as-tool, hosted tools). These are the capabilities a reviewer most needs to approve.
4. Tool reach: name effects beyond HTTP — databases, subprocesses, files, cloud SDKs and messaging #913 effects beyond HTTP and Application diff output: one finding per changed agent, the first unresolved hop, grouped uncertainty #914 reviewer-first output. Together they turn correct rows into useful ones.
5. Application diff: read agents derived with clone() as agents with their own bindings #911 clones and Application diff: identify agents whose name is not a literal (base-class template methods, computed names) #912 agents without a literal name.
6. GitHub Action: an application mode that posts the comparison as a PR comment #915: GitHub Action application mode. It can ship with today's text, then adopt Application diff output: one finding per changed agent, the first unresolved hop, grouped uncertainty #914.
7. Measure which agent frameworks real agent-changing PRs use, and decide the next reader (ADR) #916: framework share, then an ADR on the next reader. This is research and runs in parallel.

Exit: 10 Q2 cases, scored on #908's development corpus and its holdout, on a released build. Each advisory release records both counts.