Skip to content

Add manual primer previews and simplify SymPy-only comments - #11760

Merged
Rich Chiodo (rchiodo) merged 1 commit into
mainfrom
rchiodo-mypy-primer-analysis
Sep 14, 2026
Merged

Rich Chiodo (rchiodo) merged 1 commit into
mainfrom
rchiodo-mypy-primer-analysis

Conversation

@rchiodo

Copy link
Copy Markdown
Collaborator

Summary

  • Render a short, deterministic non-blocking-noise notice when the canonical SymPy project is the only changed project. Keep the recorded counts and full-report link; retain the complete validated analysis in the artifact. Mixed-project runs keep normal per-project analysis.
  • Add workflow_dispatch inputs for a recorded primer run ID and attempt, allowing artifact-only previews for verified closed or merged PRs.
  • Render the proposed comment in the read-only preview job summary and save comment.md and full-report.md in the mypy-primer-analysis-report artifact.
  • Explicitly disable the privileged publisher for manual dispatch and reject preview manifests in publishAnalysis. Automatic runs still require a current open PR.

The SymPy-only notice is a reporting policy, not proof that a checker change is harmless. Report coverage, evidence validation, and the requirement to submit a valid report remain enforced.

Manual preview example

After this workflow is merged, preview the final recorded primer run for #11601 with:

gh workflow run mypy-primer-analysis.lock.yml --repo microsoft/pyright --ref main -f primer-run-id=31429223864 -f primer-run-attempt=1

This uses the original complete artifacts rather than the truncated raw PR comment. The recorded run must be successful, its attempt must match, all eight shards and the PR-number artifact must be available, and the PR must match the recorded repository, branch, and head commit. Closed-PR association is permitted only for previews; superseded heads and ambiguous associations are still rejected.

Manual previews retain the existing GPT-5.6 Luna model, 25-credit budget, read-only agent tools, native MCP preflight, report validation, and threat detection. They never post a PR comment. Each run has a distinct job-concurrency discriminator.

Validation

  • 101 focused analysis and MCP tests passed, including closed-PR preview association, publication rejection, exact comment rendering, generated dispatch/publisher gates, and execution of the generated collector and preview scripts.
  • Validated Apply overload materialization to nested Any #11601's actual retained artifacts through the new read-only preparation path: all 11 changed projects were retained, including projects omitted from its truncated PR comment.
  • Executed the production preview script with Node's native TypeScript support against the real historical manifest and explicitly synthetic report payloads; it rendered all 11 projects without inference or GitHub writes.
  • Confirmed that the manual-disabled publisher is the only generated job with PR/issue write permissions.
  • gh-aw v0.88.7 compilation and validation passed without warnings, repeat generation was byte-identical, action-resolution pins remained unchanged, and targeted lint/format/whitespace checks passed.

Test policy classification: D — harness/configuration. No checker behavior, typeshed, or type-precision expectations changed.

The actual AI preview for #11601 has not run yet; that remains a post-merge verification step.

Add artifact-only manual analysis for verified closed or merged PRs while retaining automatic publication guards. Render a compact non-blocking notice for canonical SymPy-only differences and retain the full report. Classification D: workflow harness/configuration; no typeshed or type-precision changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@bschnurr

Copy link
Copy Markdown
Member

🔒 Automated review in progress — Bill Schnurr (@bschnurr) is auto-reviewing this PR.

@github-actions

Copy link
Copy Markdown
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

sympy (https://github.com/sympy/sympy)
+   .../projects/sympy/sympy/solvers/ode/lie_group.py:619:61 - error: Operator "-" not supported for type "Unknown | Basic" (reportOperatorIssue)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1579:38 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1580:38 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1590:9 - error: No overloads for "update" match the provided arguments (reportCallIssue)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1590:12 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1590:19 - error: Argument of type "Unknown | dict[Unknown, Unknown] | None" cannot be assigned to parameter "m" of type "Iterable[tuple[str, Unknown]]" in function "update"
+     Type "Unknown | dict[Unknown, Unknown] | None" is not assignable to type "Iterable[tuple[str, Unknown]]"
+       "None" is incompatible with protocol "Iterable[tuple[str, Unknown]]"
+         "__iter__" is not present (reportArgumentType)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1597:43 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1597:64 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1603:9 - error: No overloads for "update" match the provided arguments (reportCallIssue)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1603:12 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1603:19 - error: Argument of type "Unknown | dict[Unknown, Unknown] | None" cannot be assigned to parameter "m" of type "Iterable[tuple[str, Unknown]]" in function "update"
+     Type "Unknown | dict[Unknown, Unknown] | None" is not assignable to type "Iterable[tuple[str, Unknown]]"
+       "None" is incompatible with protocol "Iterable[tuple[str, Unknown]]"
+         "__iter__" is not present (reportArgumentType)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1610:43 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1610:74 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1616:9 - error: No overloads for "update" match the provided arguments (reportCallIssue)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1616:12 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1616:19 - error: Argument of type "Unknown | dict[Unknown, Unknown] | None" cannot be assigned to parameter "m" of type "Iterable[tuple[str, Unknown]]" in function "update"
+     Type "Unknown | dict[Unknown, Unknown] | None" is not assignable to type "Iterable[tuple[str, Unknown]]"
+       "None" is incompatible with protocol "Iterable[tuple[str, Unknown]]"
+         "__iter__" is not present (reportArgumentType)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1623:49 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:1623:70 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3442:7 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3442:38 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3443:7 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3443:38 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3444:14 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3445:14 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3446:14 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3458:50 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3459:50 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3460:50 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3500:5 - error: No overloads for "update" match the provided arguments (reportCallIssue)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3500:7 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3500:14 - error: Argument of type "Unknown | dict[Unknown, Unknown] | None" cannot be assigned to parameter "m" of type "Iterable[tuple[str, Unknown]]" in function "update"
+     Type "Unknown | dict[Unknown, Unknown] | None" is not assignable to type "Iterable[tuple[str, Unknown]]"
+       "None" is incompatible with protocol "Iterable[tuple[str, Unknown]]"
+         "__iter__" is not present (reportArgumentType)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3501:18 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3501:43 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3502:9 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3502:16 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3502:24 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3502:31 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3503:9 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3503:15 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3503:23 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3503:30 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3504:7 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3504:46 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3504:54 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3505:9 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3505:19 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3505:29 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3506:10 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3507:10 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3508:10 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3559:5 - error: No overloads for "update" match the provided arguments (reportCallIssue)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3559:7 - error: "update" is not a known attribute of "None" (reportOptionalMemberAccess)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3559:14 - error: Argument of type "Unknown | dict[Unknown, Unknown] | None" cannot be assigned to parameter "m" of type "Iterable[tuple[str, Unknown]]" in function "update"
+     Type "Unknown | dict[Unknown, Unknown] | None" is not assignable to type "Iterable[tuple[str, Unknown]]"
+       "None" is incompatible with protocol "Iterable[tuple[str, Unknown]]"
+         "__iter__" is not present (reportArgumentType)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3560:18 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3560:43 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3561:9 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3561:16 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3561:24 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3561:31 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3562:9 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)
+   .../projects/sympy/sympy/solvers/ode/ode.py:3562:15 - error: Object of type "None" is not subscriptable (reportOptionalSubscript)

... (truncated 251 lines) ...

@github-actions

Copy link
Copy Markdown
Contributor

mypy_primer analysis

Advisory AI analysis of primer run 34908468059, attempt 1
for commit d7ef7e7cb8a030c9e955f5c991b58e0be0456666. This is not an approval or proof of correctness.

Added/removed counts are diagnostic headers; message rewrites can appear on both sides.
Detail lines are counted separately, without assuming a location or association with nearby headers.

Project Added Removed Detail lines + / - Assessment Confidence Explanation
sympy 165 49 78 / 33 Needs human review low SymPy gained 165 diagnostic headers (49 disappeared; net +116), concentrated in Optional/Unknown handling, but this PR changes only primer workflow/reporting code, so attribution to Pyright behavior is unresolved.
Evidence and limitations by project

sympy

The complete shard diff reports 38,682 -> 38,798 errors (+116), with 165 added and 49 removed diagnostic headers plus 78 added and 33 removed detail lines. Added headers are dominated by reportOptionalSubscript (+71), reportOptionalMemberAccess (+11), reportCallIssue (+7, with one removed), unspecified (+75, with 48 removed), reportAttributeAccessIssue (+17, with two removed), and reportOperatorIssue (+1). The largest concrete cluster is sympy/solvers/ode/ode.py:1579-1623 and 3442-3618: indexing values now inferred as None and calls to update are reported as both unknown-member/overload/argument errors, including Unknown | dict[Unknown, Unknown] | None not satisfying Iterable. This is a potentially meaningful precision/checking change, but the PR diff contains no analyzer, typeshed, or checker implementation changes. Other additions include attribute failures at riccati.py:702, recurr.py:424-449, and tensor/array/expressions/tests/test_array_expressions.py:247, 1036-1092; operator failures at vector/implicitregion.py:188-214 (notably int/MPZ and Expr/MPZ). Many lines are rewrites at existing locations rather than new headers: unions gain Add in stats and stochastic-process diagnostics, and detail-only lines change type unions without a location header. Removed examples include the riccati.py:275 zip/__init__ diagnostic and two pde.py:173 pop diagnostics, while riccati.py:869 and single.py:2652 are message/type-union rewrites. The exact PR changes the mypy-primer collector, report rendering, manual preview gates, documentation, and tests; it explicitly preserves full reports while shortening canonical SymPy-only comments. Therefore the primer delta cannot be safely classified as an ecosystem improvement or regression caused by this commit. It should be re-run or compared

Explanation truncated; see the full report artifact linked below.

Unresolved / coverage limits: The source run identifies head d7ef7e7 and base c2c824b, but available run metadata does not establish a separate synthetic merge/base checkout or SymPy revision used by primer. No reproduction was performed. The 75 unspecified added headers and detail-only rewrites cannot all be tied to a causal implementation change from this PR.

  • The PR commit changes five workflow/reporting, documentation, and test files; it does not change the analyzer, typeshed, or checker implementation.: d7ef7e7
  • The changed implementation adds preview provenance and validation while retaining the complete diagnostic manifest.:
    request: Request,
    repository: string,
    source: SourceRun,
    directory: string,
    preview = false
    ): Promise<Manifest | undefined> {
    const prText = readInput(join(directory, 'mypy_primer_diffs_pr_number', 'pr_number.txt')).trim();
    if (!/^[1-9]\d*$/.test(prText)) {
    throw new Error('Invalid PR number artifact');
    }
    const prNumber = positiveInteger(Number(prText));
    if (!(await isCurrentPullRequest(request, repository, source, prNumber, preview))) {
    return undefined;
    }
    const projects: ProjectDiff[] = [];
    for (let shard = 0; shard < shardCount; shard++) {
    const folder = join(directory, `mypy_primer_diffs_${shard}`);
    const filename = `diff_${shard}.txt`;
    if (lstatSync(folder).isSymbolicLink() || readdirSync(folder).join() !== filename) {
    throw new Error(`Unexpected contents in shard ${shard}`);
    }
    projects.push(...parseDiff(readInput(join(folder, filename)), shard));
    }
    if (projects.length > 100 || new Set(projects.map((project) => project.name)).size !== projects.length) {
  • The changed renderer shortens only the canonical SymPy-only comment and retains the full validated report; this is reporting policy, not checker behavior.:
    }
    const runUrl = `https://github.com/${manifest.repository}/actions/runs/${manifest.source.runId}`;
    const analysisUrl = `https://github.com/${manifest.repository}/actions/runs/${positiveInteger(analysisRunId)}`;
    const markerPrefix = manifest.preview ? '<!-- pyright-primer-analysis-preview:' : reportPrefix;
    const marker = `${markerPrefix}${manifest.source.headSha}:${manifest.source.runId}:${manifest.source.runAttempt} -->`;
    const provenance = [
    marker,
    manifest.preview ? '## mypy_primer analysis preview' : '## mypy_primer analysis',
    '',
    ...(manifest.preview ? ['**Preview only. No PR comment was posted.**', ''] : []),
    `Advisory AI analysis of [primer run ${manifest.source.runId}, attempt ${manifest.source.runAttempt}](${runUrl})`,
    `for commit \`${manifest.source.headSha}\`. This is not an approval or proof of correctness.`,
    '',
    ];
    const heading = [
    ...provenance,
    'Added/removed counts are diagnostic headers; message rewrites can appear on both sides.',
    'Detail lines are counted separately, without assuming a location or association with nearby headers.',
    '',
    ];
    const rows = [
    '| Project | Added | Removed | Detail lines + / - | Assessment | Confidence | Explanation |',
    '| --- | ---: | ---: | ---: | --- | --- | --- |',
    ];
    const details: string[] = [];
    const fullDetails: string[] = [];
    for (const project of manifest.projects) {
    const item = reports.find((candidate) => candidate.name === project.name)!;
    const assessment = assessments.get(text(item.assessment));
    const confidence = text(item.confidence);
    if (!assessment || !['low', 'medium', 'high'].includes(confidence)) {
    throw new Error(`Invalid assessment or confidence for ${project.name}`);
    }
    const summary = escapeMarkdown(text(item.summary, 240));
    const explanation = text(item.explanation, maxReportLength);
    const explanationPreview = Array.from(explanation).slice(0, 1800).join('');
    const unresolved = escapeMarkdown(text(item.unresolved, 800));
    const evidence = array(item.evidence).map((entry) => {
    const source = record(entry);
    const url = new URL(text(source.url, 1000));
    if (
    url.protocol !== 'https:' ||
    url.username ||
    url.password ||
    url.port ||
    !['github.com', 'typing.python.org'].includes(url.hostname)
    ) {
    throw new Error(`Unsupported evidence URL for ${project.name}`);
    }
    return `- ${escapeMarkdown(text(source.detail, 400))}: <${url.href}>`;
    });
    if (evidence.length > 5 || (!evidence.length && item.assessment !== 'needs-review')) {
    throw new Error(`Expected cited evidence for ${project.name}`);
    }
    rows.push(
    `| ${escapeMarkdown(project.name)} | ${project.added} | ${project.removed} | ` +
    `${project.detailLinesAdded} / ${project.detailLinesRemoved} | ` +
    `${assessment} | ${confidence} | ${summary} |`
    );
    const projectHeading = [`### ${escapeMarkdown(project.name)}`, ''];
    const projectEvidence = ['', `**Unresolved / coverage limits:** ${unresolved}`, '', ...evidence, ''];
    details.push(
    ...projectHeading,
    escapeMarkdown(explanationPreview),
    ...(explanationPreview.length < explanation.length
    ? ['', '**Explanation truncated; see the full report artifact linked below.**']
    : []),
    ...projectEvidence
    );
    fullDetails.push(...projectHeading, escapeMarkdown(explanation), ...projectEvidence);
    }
    const footer = [
    `**Full evidence and limitations:** download \`mypy-primer-analysis-report\` from [this analysis run](${analysisUrl}).`,
    'The original raw primer comment is unchanged. Treat uncertainty and possible regressions as requests for human review.',
    ];
    const compactBody = [...heading, ...rows, '', ...footer].join('\n');
    const expandedBody = [
    ...heading,
    ...rows,
    '',
    '<details>',
    '<summary>Evidence and limitations by project</summary>',
    '',
    ...details,
    '</details>',
    '',
    ...footer,
    ].join('\n');
    let body = expandedBody.length <= 60000 ? expandedBody : compactBody;
    const onlyProject = manifest.projects.length === 1 ? manifest.projects[0] : undefined;
    if (onlyProject?.name === 'sympy' && onlyProject.url === 'https://github.com/sympy/sympy') {
    body = [
    ...provenance,
    '**Only SymPy changed.** These differences are treated as non-blocking primer noise; no other project changed.',
    '',
    `Recorded changes: ${onlyProject.added} added / ${onlyProject.removed} removed diagnostic headers; ` +
    `${onlyProject.detailLinesAdded} added / ${onlyProject.detailLinesRemoved} removed detail lines.`,
    '',
    ...footer,
    ].join('\n');
    }
    if (body.length > 60000) {
    throw new Error('The report exceeds the comment limit; refusing to omit projects');
    }
  • Tests cover canonical SymPy-only rendering, mixed-project behavior, preview labeling, and refusal to publish previews, corroborating the workflow-only scope.:
    expect(result).toStrictEqual(expectedReport);
    });
    test('uses a short non-blocking notice only for canonical SymPy-only changes', () => {
    const f = fixture(
    sample
    .replace(/example/g, 'sympy')
    .replace('https://github.com/sympy/project', 'https://github.com/sympy/sympy')
    );
    f.report.projects[0].name = 'sympy';
    expect(renderReport(f.manifest, f.report, 100)).toStrictEqual({
    body: [
    ...expectedReportProvenance,
    '**Only SymPy changed.** These differences are treated as non-blocking primer noise; no other project changed.',
    '',
    'Recorded changes: 1 added / 1 removed diagnostic headers; 1 added / 1 removed detail lines.',
    '',
    ...expectedReportFooter,
    ].join('\n'),
    fullReport: expectedReport.fullReport.replace(/example/g, 'sympy'),
    });
    expect(() => renderReport(f.manifest, { projects: [] }, 100)).toThrow(
    'The analysis must cover every changed project exactly once'
    );
    f.report.projects[0].explanation = '';
    expect(() => renderReport(f.manifest, f.report, 100)).toThrow('Expected nonempty text');
    });
    test('keeps normal analysis for noncanonical SymPy and mixed-project changes', () => {
    const f = fixture();
    const sympy = { ...f.report.projects[0], name: 'sympy' };
    const manifest = { ...f.manifest, projects: [{ ...f.manifest.projects[0], name: 'sympy' }] };
    expect(renderReport(manifest, { projects: [sympy] }, 100)).toStrictEqual({
    body: expectedReport.body.replace(/example/g, 'sympy'),
    fullReport: expectedReport.fullReport.replace(/example/g, 'sympy'),
    });
    manifest.projects[0].url = 'https://github.com/sympy/sympy';
    manifest.projects.push(f.manifest.projects[0]);
    const rows = [expectedReportRow.replace('example', 'sympy'), expectedReportRow];
    const details = [
    ...expectedReportDetails.map((line) => line.replace('example', 'sympy')),
    ...expectedReportDetails,
    ];
    expect(renderReport(manifest, { projects: [sympy, ...f.report.projects] }, 100)).toStrictEqual({
    body: [
    ...expectedReportHeading,
    ...rows,
    '',
    '<details>',
    '<summary>Evidence and limitations by project</summary>',
    '',
    ...details,
    '</details>',
    '',
    ...expectedReportFooter,
    ].join('\n'),
    fullReport: [...expectedReportHeading, ...rows, '', ...details].join('\n'),
    });
    });
    test('labels preview reports and refuses to send them through the publisher', async () => {
    const f = fixture();
    const manifest = { ...f.manifest, preview: true };
    const asPreview = (content: string) =>
    content
    .replace('<!-- pyright-primer-analysis:', '<!-- pyright-primer-analysis-preview:')
    .replace(
    '## mypy_primer analysis\n\n',
    '## mypy_primer analysis preview\n\n**Preview only. No PR comment was posted.**\n\n'
    );
    expect(renderReport(manifest, f.report, 100)).toStrictEqual({
    body: asPreview(expectedReport.body),
    fullReport: asPreview(expectedReport.fullReport),
    });
    await expect(
    publishAnalysis(f.request, manifest, f.output(), 100, join(directory, 'report.md'), false)
    ).rejects.toThrow('Preview analyses cannot publish PR comments');
    expect(f.request.mock.calls).toStrictEqual([]);
    });
    test.each([1799, 1800, 1801, 2143])(
    'previews a %i-character explanation without losing the full report',
    (length) => {
  • The recorded primer source run is the successful run for head d7ef7e7; its metadata provides run provenance but not a causal checker change.: https://github.com/microsoft/pyright/actions/runs/34908468059

Full evidence and limitations: download mypy-primer-analysis-report from this analysis run.
The original raw primer comment is unchanged. Treat uncertainty and possible regressions as requests for human review.

@rchiodo
Rich Chiodo (rchiodo) merged commit 733418e into main Sep 14, 2026
20 checks passed
roles: all
needs: [collect]
if: needs.collect.outputs.has_changes == 'true'
concurrency:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning · Non-blocking recommendation

📍 .github/workflows/mypy-primer-analysis.md:22
[unverified] Using github.run_id unconditionally makes every automatic run's generated concurrency groups unique, removing the existing cross-run queue and allowing overlapping analysis agents. Preserve the previous discriminator for workflow_run events and use github.run_id only for manual previews; two overlapping automatic runs would confirm the runtime impact.

[verified]

`${onlyProject.detailLinesAdded} added / ${onlyProject.detailLinesRemoved} removed detail lines.`,
'',
...footer,
].join('\n');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning · Non-blocking recommendation

📍 build/ci/mypyPrimerAnalysis.ts:507
[unverified] This replacement labels every canonical SymPy-only result as “non-blocking primer noise,” including a validated likely-regression, so the visible comment can contradict the retained analysis. Preserve the assessment in the short notice or restrict this wording to non-regression assessments; add a likely-regression rendering test to confirm the behavior.

[verified]

@bschnurr

Copy link
Copy Markdown
Member

Result: ⚠️ partially-verified

Verification details

Verification: Isolated verification observed failures that were not classified as caused by this PR: Targeted ESLint check.

Summary: The targeted mypy-primer Jest suite passed all 85 tests, covering closed-PR previews, publication isolation, generated workflow scripts, and SymPy-only rendering. TypeScript type-checking and changed-file formatting also passed. A direct ESLint invocation did not run because it lacked the repository script&#x27;s legacy-config environment setting; this was unrelated to the PR. Verification is therefore partial, although behavioral test confidence is high.

Test runs: 3 passed, 1 failed

  • ❌ Failed | unrelated to this PR | Targeted ESLint check | pnpm exec eslint build/ci/mypyPrimerAnalysis.ts packages/pyright-internal/src/tests/mypyPrimerAnalysis.test.ts
  • ✅ Passed | mypy_primer analysis Jest suite | node ./node_modules/jest/bin/jest.js src/tests/mypyPrimerAnalysis.test.ts --runInBand
  • ✅ Passed | TypeScript type check | pnpm --dir packages/pyright-internal exec tsc --noEmit -p tsconfig.jest.json
  • ✅ Passed | Changed-file Prettier check | pnpm exec prettier -c .github/workflows/mypy-primer-analysis.lock.yml .github/workflows/mypy-primer-analysis.md build/ci/mypyPrimerAnalysis.ts docs/mypy-primer-analysis.md packages/pyright-internal/src/tests/mypyPrimerAnalysis.test.ts
❌ Targeted ESLint check diagnostic output
ESLint: 9.39.4
ESLint couldn't find an eslint.config.(js|mjs|cjs) file.
The repository's check:eslint script normally sets ESLINT_USE_FLAT_CONFIG=false.

@bschnurr Bill Schnurr (bschnurr) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@bschnurr Bill Schnurr (bschnurr) added the review-auto:approved Automated review: no blocking findings (approval posted). label Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants