Fix agentic triage comment submission and reject incomplete summaries

[priyankatiwari08]

Summary

Repair the safe-output submission failure behind #4704. In the recorded run for #4193, jq payload preparation was denied by the shell allowlist. The agent then submitted a diagnostic test message please ignore comment, consumed the one-comment quota, and could not publish its real triage summary.

• Allow jq and document separate, checked JSON payload preparation followed by exactly one safeoutputs add_comment . submission. Explicitly prohibit diagnostic writes, -/@- body experiments, and retries after a submission error.
• Add a trusted pre-publication validation step that checks the entire safe-output batch before any comments or labels are applied. Reject placeholder/unfinished comments, quoted or fenced templates, missing check rows or sections, multiple comments, label-only results, and mixed success/incomplete outcomes. Preserve explicit no-op and failure reporting.
• Keep the existing trigger conditions, one-comment quota, label allowlist, on-demand label behavior, and read-only agent permissions. Checkout only the trusted validator at the event SHA, with persisted credentials disabled.
• Regenerate the workflow lock with the existing gh-aw v0.88.2 compiler and add regression coverage to the lock-verification workflow.

Validation

• All 24 Node.js validator regression tests pass, covering complete summaries, diagnostic and unfinished content, duplicate rows/sections, generic/XML/comparison prose, error and incomplete batches, operation ordering, and mode/Environment-consistent label operations. This suite does not test jq payload encoding.
• The original Entra ID authentication broken under NativeAOT in v7.0 — Extensions.Azure reflection-based discovery is AOT-incompatible #4193 agent output is rejected both with its incomplete report and as a placeholder-only batch.
• gh aw compile succeeds; repeating compilation leaves the generated lock byte-for-byte unchanged.
• git diff --check passes.

Rollout and recovery

Related to #4704; intentionally do not close that tracker on merge. The previously reported completion count included workflow-attributed placeholders and is being re-audited. This PR fixes the workflow, not the existing backlog results.

After this reaches the default branch, verify a small representative pilot, retry only genuinely incomplete issues in completion-gated batches within the existing daily credit budget, and delete confirmed placeholder comments only after replacement summaries succeed. Keep #4704 open until recovery is verified. No backlog retries or comment deletions are included in this PR.

Checklist

• Tests added or updated
• Public API changes documented — N/A; workflow-only change
• Verified against customer repro (if applicable) — recorded workflow failure replayed; no product repro applies
• Ensure no breaking changes introduced — no public API changes; malformed triage output is intentionally rejected

[Copilot]

🟡 Changes recommended

The validator still accepts collector errors, embedded placeholders, and contradictory no-op/incomplete batches.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Repairs issue-triage safe-output submission and adds pre-publication validation.

Changes:

• Enables safe JSON payload creation with jq.
• Validates triage summaries before publishing comments or labels.
• Adds regression tests and regenerates the workflow lock.

File summaries

File	Description
.github/workflows/verify-aw-lock.yml	Runs validator tests during lock verification.
.github/workflows/issue-triage.md	Adds safe submission guidance and validation steps.
.github/workflows/issue-triage.lock.yml	Regenerates the executable workflow.
.github/scripts/validate-triage-output.cjs	Implements output validation.
.github/scripts/tests/validate-triage-output.test.cjs	Adds validator regression coverage.
.github/scripts/tests/README.md	Documents test execution and scope.

Review details

Suppressed comments (2)

.github/scripts/validate-triage-output.cjs:36

• The validator ignores the collector's top-level errors array. gh-aw records rejected/over-quota safe-output calls there while retaining any valid items, so a valid-looking comment plus a failed label or second submission is currently published as success. Reject any non-empty output.errors before processing items, and cover that mixed batch in the regression tests.

function validateTriageOutput(output) {
    if (!output || !Array.isArray(output.items) || output.items.length === 0 ||
        output.items.some(item => !item || typeof item.type !== 'string')) {
        throw new Error('Missing or malformed triage output: expected non-empty result items.');

.github/scripts/validate-triage-output.cjs:50

• A batch containing both noop and report_incomplete has no comment, enters this branch, and returns successfully. Those are contradictory completed/no-action and failed outcomes, so this still permits a mixed success/incomplete batch. Reject incomplete && noop before the no-comment return and add a regression test.

    if (comments.length === 0) {
        if (!incomplete && !noop) {
            throw new Error('Triage output has no summary or explicit no-op/incomplete result.');
        }
        if (output.items.some(item => ['add_labels', 'remove_labels'].includes(item.type))) {

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

The validator ignores ingestion errors, rejects valid generic syntax, and the workflow prompt still permits retries after submission errors.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 3

Open (3)

• Avoid rejecting valid multi-argument generic syntax · New
• Reject outputs containing errors alongside valid items · New
• Forbid all retries after submission attempts · New

Resolved since last review (1)

• This only rejects a value when the entire value is angle-bracketed, so an unfinished template value…

[Copilot]

Copilot review overview

🔵 Needs a closer look

Duplicate template rows can bypass whole-comment placeholder validation.

Review effort: Balanced
Findings: None

Resolved since last review (3)

• Forbid all retries after submission attempts
• Reject outputs containing errors alongside valid items
• Avoid rejecting valid multi-argument generic syntax

Previously missed (1)

In code that hasn't changed since last review

Validate placeholders across duplicate rows and all prose

.github/​scripts/​validate-triage-output.cjs:92

find validates only the first occurrence of each required row, and the placeholder scan is limited to that selected value. A completed row followed by | Issue type | <fill in> | is therefore accepted, so an unfinished template can still be published despite the PR's whole-comment validation goal. After validating the required rows and sections, scan all non-quoted/non-fenced prose for placeholders; add a duplicate-row regression case as well.

[paulmedynski]

Looks good, with one Copilot suggestion remaining.

[Copilot]

🔵 Needs a closer look

Placeholder detection still rejects valid code syntax and accepts bracket-free template instructions.

0 open findings

1 resolved since last review

• Avoid treating comparison operators as unfinished template placeholders

Previously missed (1)

In code that hasn't changed since last review

Placeholder matcher rejects valid generics and XML tags

.github/​scripts/​validate-triage-output.cjs:11

This field-name matcher also rejects legitimate C# generics and XML tags whenever their type/tag name is one of these words. For example, a completed Analysis containing Task<Result>, List<Item>, or an XML <summary> tag is classified as unfinished (the current validator throws for Task<Result>), so valid triage output can be blocked. Make placeholder detection context-aware or restrict it to the workflow's actual template forms, and add regressions for these common code/documentation forms.

This issue also appears on line 43 of the same file.

🧠 Review effort: Balanced

[paulmedynski]

Yet another Copilot suggestion.

[Copilot]

🔵 Needs a closer look

Leading whitespace is stripped before heading validation, allowing an indented code-block heading to pass.

0 open findings

Previously missed (1)

In code that hasn't changed since last review

Validate heading before trimming leading indentation

.github/​scripts/​validate-triage-output.cjs:95

Checking the heading against body.trim() removes leading indentation before validation. A body whose first line is ## 🔍 Triage Summary therefore passes this check (and the remaining checks when its rows/sections are unindented), even though Markdown renders that first line as a code block rather than the required heading. Validate the original string start instead, and add an indented-heading regression case.

🧠 Review effort: Balanced

[priyankatiwari08]

Validate heading before trimming leading indentation

Fixed in af74dd4. The heading is now checked without trimming leading whitespace. Added regressions for spaces and tabs; all 20 validator tests pass.

[Copilot]

🔵 Needs a closer look

The validator accepts label actions that contradict the summary and on-demand triage rules.

0 open findings

Previously missed (1)

In code that hasn't changed since last review

Validate label actions against summary and Environment state

.github/​scripts/​validate-triage-output.cjs:90

The gate only checks that label actions follow the comment; it does not verify that they match the summary. A missing/partial Environment with remove_labels, a complete environment with add_labels, and an on-demand summary with either label action are all accepted, even though the workflow requires the opposite behavior (lines 321-328). This can corrupt the Waiting for Author state that controls future activations. Validate the number and type of label operations against the heading variant and parsed Environment row, and add regressions for these contradictory batches.

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The validator still accepts a known placeholder and incomplete or mode-inconsistent label batches.

2 open findings

• Embedded placeholder diagnostic bypasses meaningful() validation · New
• Required workflow label operations are not enforced · New

🧠 Review effort: Balanced

[priyankatiwari08]

Addressed the current Copilot findings and remaining related gaps in 42d95c2:

• Reject test simple body in required rows and sections.
• Require Environment-consistent waiting-label operations for initial/follow-up triage; reject label mutations for on-demand triage.
• Reject duplicate required rows/sections and unfinished templates in additional prose, covering the older review finding.
• The generic/XML false-positive finding was already fixed in fd13931; its acceptance regressions remain intact.

All 24 validator tests pass. The new regression cases failed before the fixes. Both current inline threads have replies and are resolved. These checks validate the output batch; they do not make comment/label publication atomic.

[Copilot]

🟢 Approval recommended

The implementation and generated workflow are coherent; only the PR’s validation wording is stale.

1 open finding

• Update Validation text to match current test coverage · New

2 resolved since last review

• Required workflow label operations are not enforced
• Embedded placeholder diagnostic bypasses meaningful() validation

🧠 Review effort: Balanced