Repository navigation
Prioritize causal regression findings in mypy_primer analysis - #11763
Conversation
Use GPT-5.6 Terra with the approved 100-credit budget. Require baseline, changed behavior, impact and pinned PR attribution; keep independently extracted regression warnings prominent even when model assessments are benign. Classification D: workflow harness/configuration; no checker, typeshed or type-precision expectation changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🔒 Automated review in progress — Bill Schnurr (@bschnurr) is auto-reviewing this PR. |
This comment has been minimized.
This comment has been minimized.
Classification D: harness issue. Use explicit Markdown fixture values instead of partial escaping helpers. Preserve exact report assertions and all checker type expectations; no typeshed change applies. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Result: Verification detailsVerification: Isolated verification observed failures whose relationship to this PR could not be determined: Targeted Prettier check; this review is not fully verified. Summary: The two targeted Jest suites passed: 105 analysis tests and 16 MCP tests, matching the PR's stated 121 focused tests. TypeScript compilation also passed. A targeted Prettier check failed on both changed Markdown files; whether this is introduced by the PR could not be established without baseline metadata. Coverage is strong, but the new fork-repository attribution acceptance path lacks a direct test. Test runs: 5 passed, 1 failed
❌
|
Bill Schnurr (bschnurr)
left a comment
There was a problem hiding this comment.
Approved via Review Center.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Classification D: harness/configuration. Avoid the pinned MCP tool's 10 KB per-call limit by collecting one project per submission. Preserve complete coverage, citation validation, historical reports, and the aggregate payload bound. Keep the 100-credit budget. No checker type precision or typeshed changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Classification D: harness/configuration. Increase the explicitly approved Sol analysis cap and update generated configuration, budget assertions, and documentation. Preserve model selection, report validation, and all checker type expectations; no typeshed changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Result: 🔴 Verification detailsVerification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified. Summary: Verification could not be completed because the execution environment was unavailable before test discovery or targeted execution. The PR adds substantial focused Jest coverage for regression-signal extraction, report ranking/rendering, exact-head citation validation, per-project submission assembly, publication completeness, malformed payload rejection, historical compatibility, and aggregate size limits. No test result is claimed as passed. Test runs: 2 not run
|
This comment has been minimized.
This comment has been minimized.
Classification D: harness coverage. Exercise exact-head citations from the fork and upstream repository, and reject mismatched commits, missing line anchors, and unrelated repository paths. Preserve complete report assertions and all existing type precision; no checker or typeshed changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Addressed the verification feedback in 8411be2.
These are local verification results; the later review runner's unavailable execution environment is a separate issue, not a failing test. The new feedback was in top-level verification summaries, and there are no unresolved inline review threads. |
|
Diff from mypy_primer, showing the effect of this PR on open source code: sympy (https://github.com/sympy/sympy)
- .../projects/sympy/sympy/solvers/bivariate.py:135:15 - error: Operator "-" not supported for "None" (reportOptionalOperand)
- .../projects/sympy/sympy/solvers/bivariate.py:139:17 - error: Operator "-" not supported for type "Basic | Unknown" (reportOperatorIssue)
- .../projects/sympy/sympy/solvers/bivariate.py:144:23 - error: Operator "-" not supported for "None" (reportOptionalOperand)
- .../projects/sympy/sympy/solvers/deutils.py:234:14 - error: Operator "not in" not supported for types "str" and "Unknown | int"
- Operator "not in" not supported for types "str" and "int" (reportOperatorIssue)
- .../projects/sympy/sympy/solvers/diophantine/diophantine.py:176:44 - error: Cannot access attribute "expand" for class "Basic"
- Attribute "expand" is unknown (reportAttributeAccessIssue)
- .../projects/sympy/sympy/solvers/diophantine/diophantine.py:423:38 - error: Argument of type "Unknown | Expr | Literal[0]" cannot be assigned to parameter "expr" of type "Expr" in function "make_args"
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:423:38 - error: Argument of type "int | Expr" cannot be assigned to parameter "expr" of type "Expr" in function "make_args"
- Type "Unknown | Expr | Literal[0]" is not assignable to type "Expr"
+ Type "int | Expr" is not assignable to type "Expr"
- "Literal[0]" is not assignable to "Expr" (reportArgumentType)
+ "int" is not assignable to "Expr" (reportArgumentType)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:504:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:505:19 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:506:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:569:42 - error: Operator "*" not supported for types "int" and "Unknown | Basic"
+ Operator "*" not supported for types "int" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:569:50 - error: Operator "*" not supported for types "Expr" and "Unknown | Basic"
+ Operator "*" not supported for types "Expr" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:725:42 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:735:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:736:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:737:19 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:814:47 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:841:26 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:842:22 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:36 - error: Operator "*" not supported for types "int" and "Unknown | Basic"
+ Operator "*" not supported for types "int" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:42 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:51 - error: Operator "*" not supported for types "int" and "Unknown | Basic"
+ Operator "*" not supported for types "int" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:57 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:843:66 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:845:53 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:854:18 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:856:22 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:865:22 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:865:36 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:867:27 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:868:27 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:869:27 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:870:27 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:871:27 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:872:27 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:876:24 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:877:24 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:878:24 - error: Operator "**" not supported for types "Unknown | Basic" and "Literal[2]"
+ Operator "**" not supported for types "Basic" and "Literal[2]" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:879:24 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:880:24 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:881:24 - error: Operator "*" not supported for types "Unknown | Basic" and "Unknown | Basic"
+ Operator "*" not supported for types "Basic" and "Basic" (reportOperatorIssue)
+ .../projects/sympy/sympy/solvers/diophantine/diophantine.py:888:37 - error: Operator "*" not supported for types "int" and "Unknown | None"
+ Operator "*" not supported for types "int" and "None" (reportOperatorIssue)
... (truncated 644 lines) ...
|
mypy_primer analysisAdvisory AI analysis of primer run 35003751736, attempt 1 Only SymPy changed. These differences are treated as non-blocking primer noise; no other project changed. Recorded changes: 251 added / 147 removed diagnostic headers; 181 added / 146 removed detail lines. Full evidence and limitations: download |
|
Result: ✅ Verification detailsVerification: Isolated verification observed failures that were not classified as caused by this PR: Dependency and focused-test discovery. Summary: Initial Jest discovery could not run because dependencies were absent; the frozen offline install then succeeded. The focused analysis suite passed 117/117 tests, and the MCP preflight suite passed 16/16 tests. These cover regression signals, report ranking/rendering, attribution validation, per-project assembly/publication, workflow configuration, and MCP safeguards. All relevant tests passed after setup, so confidence is high. Test runs: 3 passed, 1 failed
❌
|
Summary
Any/Unknownassertion results, other assertion failures, disappeared checking diagnostics, and increased gradual-type occurrences in changed details. These are leads, not confirmed regressions or proof of causation.likely-prattribution requires a line-pinned source citation at the analyzed head.Anyas proof that lost checking is harmless.Motivation
The first manual preview of #11601 classified 268 new pandas-stubs assertion failures and two scipy-stubs failures as high-confidence exposed typing issues, despite acknowledging that it had not inspected the affected overload declarations. The recorded results received bare
Anyinstead of expected container or shape-specialized types. This needed a regression investigation, not a diagnostic inventory or an assumption that intentional conformance changes cannot cause compatibility regressions.The new report contract separates the observed change, its consequences, and attribution. The deterministic warning layer prevents the same kind of benign model judgment from silently burying these signals. It is not an exhaustive detector and does not prove that every flagged difference is caused by the PR.
Validation
Unknownreturn details, and removed-check leads.4b991f5e4removes three partial escapers from test expectations in favor of explicit fixture values. Complete output assertions remain intact.Test policy classification: D — harness/configuration. No checker or typeshed behavior changed; test expectation changes cover the intentional report contract and presentation changes.
Live previews
The four comparison runs below used the historical #11601 primer run
31429223864, attempt 1, including all 11 projects.Successful 200-credit Sol preview
Run 35001550116, at commit
9ef49f680, completed successfully end to end and used 120.14492 analysis credits out of 200. Agent execution, report validation, threat detection, and the preview job all succeeded. Themypy-primer-analysis-reportartifact contains the actualcomment.mdandfull-report.md; no local recovery was needed.All 11 projects are included. The comment leads with pandas-stubs and scipy-stubs as high-confidence possible regressions, linking their bare-Any results to exact-head implementation and realistic PR test controls. Core and pip retain regression concerns but explicitly uncertain PR attribution. Exact downstream revisions and reproductions remain unverified, and baseline assertion acceptance is not an independent baseline reveal.
The publisher was intentionally skipped because this was a manual preview. No PR comment was posted.
Earlier runs and the submission fix
Terra run 34993340047 used 34.69602 analysis credits. Terra flagged the pandas-stubs and scipy-stubs erasures, but completion validation rejected Spark's
likely-prattribution because its citation used the wrong commit. Threat detection passed; preview and publication were skipped.Terra run 34994808196 used 29.07228 analysis credits. Spark's citation used the correct commit, but Core was classified as
possible-regressionwith no evidence, so validation failed again.Further investigation found a transport problem in that second run: the MCP tool rejected the initial report for exceeding its 10240-byte input limit. Terra compressed the report and dropped citations, including Core's only citation. The pinned compiler does not expose a per-custom-input limit override, so commit
9d15743ddchanges to per-project submissions rather than weakening that guard. Replaying the logged pre-compression payload through the unmodified pinned length validator preserves all 11 projects, each at most 2443 bytes. This fixes the transport problem, not unsupported causal claims: the original report still fails a separate Prefect citation check.The initial Sol preview 34998888439, at commit
9d15743dd, submitted all 11 projects, with the largest submission 3163 bytes. The complete normalized output passed the repository validator locally, and threat detection passed. Its implementation/test citations were more useful than the earlier reports, but it recorded 101.39692 credits against the former 100-credit cap, followed by HTTP 403 on the next provider request. The agent exited unsuccessfully after submission, so its GitHub preview job was skipped and the report was recovered locally. The separately approved increase to 200 in9ef49f680allowed the subsequent run above to finish normally. These were model-plus-submission changes, not a model-only comparison.