Triage external findings against existing evidence (2026-09-28) — no new feature justified #85

Disposition

ABSTAIN_FROM_NEW_FEATURE = YES

No new feature, module, layer, or independent duplicate investigation is justified by this review. The material findings map to active CodePro issues and roadmap gates. Record their relevance in those owners; preserve all project rules and require local evidence before any architecture change.

User requirement

Preserve all CodePro rules and demonstrate that every feature is needed. For each external finding, choose among:

• reuse/continue an existing owner;
• propose a new feature only with a falsifiable demonstrated local gap;
• abstain when the gap is not established.

Normative rule: BENCHMARK_SIGNAL != LOCAL_PROOF. External results do not qualify, accept, or promote a CodePro treatment.

Repository snapshot reviewed

• Repository: oigorbrito/codepro, default branch main.
• Current read-only review date: 2026-09-28.
• Architecture owner already exists: Architecture Confrontation v1 — benchmark external scaffolds against CodePro hot-path complexity #84.
• Existing cost/efficiency owners: Mini Efficiency v1 — optimize mini-swe-agent without changing executor semantics #31, Executor Cost-Capability v1 — reproduce new coding-agent efficiency results under CodePro controls #32, Harness Change Regularization v1 — test held-out transfer and cost-aware acceptance #37.
• Context, retrieval, benchmark, and decision owners: Context Memory v1 — compare searchable cross-window state against lossy compaction #33, Selective Retrieval v1 — test grounded context injection versus anchoring failure #34, Benchmark Expansion v1 — assess SWE-bench Science / Hard70 for CodePro evaluation #35, Benchmark Decision Protocol v1 — paired evidence and scaffold provenance #78.
• Telemetry contract gate: Phase 3 Telemetry Baseline v1 — verify normalized local execution evidence contract #82, with open implementation PR fix: make Phase 3 telemetry evidence non-overwriting #83.
• Roadmap Phases 3–17 separately defer/track telemetry, local-runtime plumbing, execution/verifier plumbing, scaffold comparison, cost metrics, ablation, held-out validation, and promotion.
• PRs ci: add Mini v2.4.6 runtime access gate #73/test: add Mini v2.4.6 provider-free verifier controls #74/qualify Mini v2.4.6 with Gemini API #77 remain open qualification work; nothing in this review authorizes changing their frozen treatments.
• AGENTS.md preserves Decision Basis records, minimal structural changes, execution-profile adoption, roadmap synchronization, state separation, and no silent fallback/scope expansion.

Material findings and CodePro disposition

1. Harness component ablations

Source/date: “An Empirical Study of Harness Design for Coding Agents,” arXiv:2609.20804, surfaced 2026-09-28. Search-index summary reports 176 settings over four models and SWE-Bench Verified / Terminal-Bench 2.1; planning’s accuracy/cost contribution depends on model and task/context.

Evidence class: paper/preprint summary; detailed primary-page retrieval failed in this review, so numerical details beyond the indexed abstract are not treated as verified.

Local comparison: #84 already requires mechanism-by-mechanism ablation and classifies planning/context/retrieval as optional or unproven until tested. #31/#32/#37 own efficiency and cost-aware decisions; #78 owns paired decisions.

Disposition: reuse #84 and existing experiment owners. No new planner, memory, router, retry controller, or architecture feature. Need local falsifiable treatment with same frozen tasks/model/runtime/verifier/budget and measure verified outcome, tokens, calls, retries, replans, wall time, provider cost/cost per verified resolution, held-out regression, and complexity. If no meaningful effect above noise or material cost is unjustified, bypass/remove or abstain. No local result established here.

2. Production-path / integration correctness

Source/date: SWE-Serve, arXiv:2609.26777, dated 2026-09-22. The authors report 53 SGLang production-inference tasks across six families, hidden tests and oracle/no-op controls. Among 19 tasks with serving E2E coverage, report 45.9% pass under the verifier versus 69.4% when E2E tests are excluded; removing E2E tests raises mean pass rate 23.4 percentage points across evaluated configurations.

Evidence class: author-reported research benchmark with released benchmark/verifiers; no CodePro reproduction. Domain is production inference engineering (SGLang, CPU/H100), so transfer to CodePro’s local task distribution is uncertain.

Local comparison: CodePro roadmap Phase 6 already tracks isolated workspace, exact starting revision, diff/command/test capture, independent verification, and persisted evidence. #84 expects verifier separation; #82 governs telemetry and explicitly forbids telemetry from inferring success.

Disposition: prioritize completion of existing Phase 6 verification gate and test relevant repository-level integration/end-to-end paths when the actual CodePro workload requires them. Do not add a second verifier subsystem or broaden all tasks to E2E without evidence. Hypothesis to test locally: a defined class of patches passes current local tests but fails declared integration/E2E acceptance checks. If a frozen representative set cannot demonstrate that gap, abstain from extra E2E feature scope.

3. Benchmark reliability / leakage

Source/date: SWE-Bench Pro Verified, arXiv:2609.08149, dated 2026-09-08. Authors identify possible gold-solution/evaluation leakage and task-quality defects; propose anti-hacking safeguards and minimal task refinement, and report lower scores for some systems.

Evidence class: author-reported benchmark audit/preprint; no CodePro reproduction.

Local comparison: #35 owns benchmark selection; #37 owns held-out transfer/regularization; #78 owns per-instance paired decision discipline; CodePro’s qualification/evidence contracts preserve frozen identities, independent verification, and non-promotion from benchmark signal.

Disposition: reuse these owners when selecting an external benchmark or qualifying its verifier. Do not adopt SWE-Bench Pro Verified by default or add anti-leakage machinery absent a CodePro benchmark threat model and concrete test. No local leakage defect demonstrated.

Feature necessity rule

A mechanism may remain mandatory only when local evidence shows:
GAIN > NOISE AND no material held-out regression AND cost/complexity justified AND reproducibility PASS.
Subset-only gains make it optional. No demonstrated gap means abstain. Existing governance, provenance, verification, and promotion controls remain preserved under their current contracts; this review does not weaken them.

Decision

NEW_FEATURE = NONE
DUPLICATE_ISSUE = NONE
EXTERNAL_SIGNAL = TRIAGE_ONLY
LOCAL_PROOF = NOT_ESTABLISHED
NEXT_ACTION = CONTINUE_EXISTING_OWNERS
RELEASE_OR_PROMOTION_AUTHORIZATION = NO

This is a triage/index record, not implementation authorization or a claim that any external result transfers locally.