d4d-agent workflow: respond only to an explicit request with a dataset argument, never to a handle quoted in prose or code (#4108)

[realmarcin]

Fixes #4108

.github/workflows/d4d-agent.yml started its check-mention job on any issue, pull request or comment body that contained the assistant's handle (contains() does not see Markdown). Its detect step then took the rest of the line after the handle and a blank as the request. For an author in .github/ai-controllers.json, a body that merely quoted the handle before an input directory name could start a billed d4d api run and open a PR. Without a dataset name, it posted "I could not tell which dataset to generate". Run 36902377148 on #4093 reached the detect step; only the backtick after the quoted handle kept the regex from reading a request.

What changes

New src/github/assistant_request.py makes the decision (standard library only, Python 3.9 or later). A request is one line that:

• starts with the assistant's handle at column 0 (any ASCII letter case; a longer login is a different account);
• continues with spaces or tabs and the exact name of one directory under data/sheets_d4dassistant/inputs/, with nothing after it;
• stands alone: a blank line, or the start or end of the text, before it and after it;
• lies outside fenced code blocks and outside HTML blocks that span blank lines (comments, <pre>, <script>, <style>, <textarea>, processing instructions, declarations, CDATA);
• comes after no raw HTML that is left open. A line that may be raw HTML must close each comment and tag it opens. No line outside a fence may hold a start tag of <textarea>, <script>, <svg> or a similar element, a processing instruction, CDATA or --!>.

A quoted line, a list item, an indented code block or a table row either does not start with the handle or does not stand alone. A text whose request lines name different datasets holds no request. Anything else logs no request, with the reason for each line that starts with the handle (where requests conflict, one note names every line that asks), and nothing runs.

d4d-agent.yml:

• The job-level contains() check stays, documented as a cheap first gate only.
• The detect step now only authorizes the author and writes the text to $RUNNER_TEMP.
• A new Read the request step runs the script on the runner's own python3, only for allowed authors, and sets qualified-mention and dataset.
• The respond job takes --project from that dataset output and re-checks it against its own checkout, instead of searching the free text for a dataset name.
• The "could not tell which dataset" branch is removed, since nothing reaches it now. The no-bundle report is kept.
• Unchanged: allowed authors, triggers, the d4d api run command, the prompts, which file the run reads, and how manual dispatch picks a comment.

.github/D4D_ASSISTANT_README.md: the request section describes the line format. It says which file a run reads (the first .txt or .md under the directory, paths sorted), where the directory must exist for each kind of event, and which directory names a request can carry. The three free-text example requests are removed, since none of them is a request now.

Decisions the brief left open

• What counts as a declared project: a subdirectory of data/sheets_d4dassistant/inputs/. That is the set the old resolve step already matched; d4d api run --bundle accepts any name. Names must match [A-Za-z0-9][A-Za-z0-9_.-]* because they reach shell commands; a subdirectory named otherwise is logged as one a request cannot name. Matching is exact; the old grep ignored case.
• Nothing may follow the dataset name, on its line or on the next, so prose such as " CHORUS runs failed" is not a request.
• How "outside code spans" is enforced: by the stand-alone rule, not an inline parser. No code span, inline comment or quotation crosses a blank line. After one, a list item continues only on indented lines. A line with a blank line after it cannot be a heading's text or a table's header. The cost: a request line directly before or after other text, with no blank line between, is refused and logged.
• Fails closed where a line reader cannot place the end of a fence or HTML block without parsing lists or HTML: a fence or HTML block opened on a list marker, content to the left of an indented opener, an indented fence closed past three spaces, and a fence or HTML block opened in what may be an HTML block. It also fails closed after raw HTML that may be left open, as above. In those cases no later line is read as a request, and a later line that would ask for another dataset still cancels a request before it.
• No third-party Markdown parser: it would add a network install to a gate that runs on every event that passes the pre-filter. The raw-HTML check is a small reader of the HTML tokenizer's comment and tag rules.
• An unknown dataset gets no reply comment. It is logged as "no request" and nothing runs. The README says so.

Evidence

• Replay on real repository text (read-only queries): 40 issue/PR bodies and comments contain the handle, all by allowed authors. If they were posted today, the current detect regex and resolve grep would start a billed run on 8 of them and post the "could not tell" comment on 27. The new parser finds a request in none of them, before review round 1 and after it.
  • 30 of the 40 are messages addressed to the open-ended agent that D4D assistant is non-functional: the GitHub Action it depends on no longer exists #172 replaced: datasheet requests, follow-up instructions, tests and questions.
  • The other 10 only mention the handle: the bodies of Add D4D AI Assistant GitHub integration #55, D4D assistant is non-functional: the GitHub Action it depends on no longer exists #172, Replace the D4D assistant's dead action with the in-repo API path #173, GitHub D4D assistant defaults diverge from registered shared prompt condition #2088 and Skill #4021: false positives (d4d_assistant_edit.md gates though nothing loads it; every controller module counted as run as a script) #4093, and five status comments on Update documentation with a link to the Agent #36, Test D4D for YAML Link #76, add documentation for running different modes eg interactive vs batch #86, D4D for cm4ai #88 and D4D for cm4ai #91. The old logic would have posted the "could not tell" comment on 6 of those 10.
• Skill #4021: false positives (d4d_assistant_edit.md gates though nothing loads it; every controller module counted as run as a script) #4093 body (kept as a fixture, verbatim): no request. With the backticks removed, the old regex matches it, and the new parser still finds none.
• Rendering check (review round 1, a scratch harness that is not committed):
  • 180,000 random texts built from the line shapes in the round-1 findings were rendered with markdown-it-py 4.0 (its commonmark and gfm-like presets) and parsed with html5lib 1.1. Every request line the parser read appeared as a visible paragraph of its own.
  • Against the parser as it was before round 1, the same harness finds 2,696 of 30,000 texts where a request the parser reads is not shown as a paragraph of its own.
  • A second harness checked 120,000 random HTML fragments. Whenever the parser calls a fragment closed, html5lib shows the text after it. The one exception is <template>, which hides its content by what the element does, not by where its markup ends (see the proposed follow-ups below).
  • Neither harness is GitHub's own renderer (cmark-gfm and its sanitizer), and nothing was sent to GitHub's render API.
• Specificity audit (scan.py) on origin/main and on this branch: 167 violations in both. One surface is added: src/github/assistant_request.py, classed github_assistant / run_shaping, with no text spans. Rerun after review round 1: the same 167 violations and the same surfaces.

Tests

• tests/test_assistant_request.py (25 tests) covers:
  • the brief's fixtures and their near misses;
  • every HTML block kind, the stand-alone rule, extra words, indentation, two datasets, Unicode line separators, lines of other white space, non-ASCII case folds of the handle, and the fail-closed cases;
  • raw HTML left open, and raw HTML that closes;
  • the workflow wiring;
  • the step's own command, run as written with python -S.
• Mutation testing:
  • Before review round 1, the author's 21 mutants were all caught, but round 1 found 14 more that survived (The HTML-block test does not cover most of the HTML-block kinds the docstring lists #4425-No test covers the re.ASCII flag on the handle pattern #4428).
  • After round 1, 45 of 45 mutants are caught, those 14 included.
  • Two mutants survive that only remove a deliberate redundancy: the tokenizer still follows <? and --!>, which an earlier check already refuses.
• With tests/test_generation_specificity_skill.py and tests/test_audit_recall_leak.py: 215 passed. The parser and command-line tests also pass under Python 3.9.6.

Follow-ups filed

• d4d-agent.yml: editing a text that already holds a request runs generation again #4391: editing a text that already holds a request runs generation again.
• d4d-agent.yml reads its allow-list and request parser from the pull request it is evaluating #4392: the workflow reads its allow-list and request parser from the pull request it evaluates.
• .github/D4D_ASSISTANT_README.md still describes the open-ended assistant from before #172 #4393: .github/D4D_ASSISTANT_README.md still describes the open-ended assistant from before D4D assistant is non-functional: the GitHub Action it depends on no longer exists #172.
• d4d-agent.yml manual dispatch picks the newest comment containing the handle, not the newest request #4394: manual dispatch picks the newest comment containing the handle, not the newest request.
• d4d-agent workflow: generate from every document in the input directory, not only the first .txt or .md file #4450 (from review round 1): d4d-agent workflow: generate from every document in the input directory, not only the first .txt or .md file.
• d4d-agent workflow: reply when an allowed author's request line names a directory the run cannot see #4451 (from review round 1): d4d-agent workflow: reply when an allowed author's request line names a directory the run cannot see.
• assistant_request.py: a request line inside an element that hides its content is still read #4452 (from review round 1): assistant_request.py: a request line inside an element that hides its content is still read.

Review round 1

Twelve findings were filed as #4422-#4433. All twelve are fixed here.

• A request line inside an HTML comment opened in an HTML block, a block quote or a deep list item is still read as a request #4422, a request line hidden by raw HTML left open: fixed. A line that may be raw HTML must close each comment and tag it opens, by the HTML tokenizer's rules. An HTML block of kinds 1-5 must close them by its last line. No line outside a fence may hold a start tag of a raw-text or foreign-content element, a processing instruction, CDATA or --!>. The rendering check found one case beyond the issue: an inline <textarea> in an ordinary paragraph also swallows every later line.
• Once the reader loses track of a block, a later request for a different dataset no longer cancels an earlier request #4423, a conflict past the point where the reader loses track: fixed. Such a line is still not a request, but it now cancels a request for another dataset before it.
• The new README line says the datasheet is built from the documents in the input directory; the workflow reads only the first .txt/.md file #4424, the README said the run reads the documents in the directory: fixed. It now says the run reads one file, and how that file is chosen.
• The HTML-block test does not cover most of the HTML-block kinds the docstring lists #4425, The fenced-block test cannot fail for the closing-fence rule or the tilde info-string rule #4426, No test covers the blank-line definition: widening it to any whitespace turns a handle inside a code span into a request #4427, No test covers the re.ASCII flag on the handle pattern #4428, tests that could not fail: fixed. Every mutant these issues name is now caught.
• 'Quoted lines, list items, table rows ... never start with the handle' is false; the blank-line rule is what excludes them #4429, the claim that quoted lines, list items and table rows never start with the handle: fixed in the docstring and in this description. A request line now also needs a blank line, or the end of the text, after it. That keeps out the one-column table header the issue found.
• The log note 'does not start a paragraph' is false after a heading, a closing fence or a one-line HTML comment, and a test asserts it #4430, a log note that was false after a heading, a fence or a comment: fixed. The notes now state the rule. The setext heading the issue found is refused by the same rule.
• The docstring says every line starting with the handle gets a reason, but a repeated conflicting line gets none #4431, a repeated conflicting line went unnamed: fixed. The conflict note names every line that asks.
• PR body: 'All were free-text requests written for the open-ended agent that #172 replaced', but the 40 replayed texts include non-requests #4432, a false sentence about the replayed texts: fixed in the Evidence section.
• README says to add the input directory "first" but not that issue and PR-conversation requests read the default branch, nor the directory-name rule; such requests now run nothing and post nothing #4433, the README did not say where the directory must be or which names a request can carry: fixed. The log now reports a directory whose name breaks the rule as such, not as a missing directory.

Behavior change in round 1: a request line followed directly by other text was read before; it is now refused and logged.

Proposed follow-ups, not filed from this PR:

• having the run read every document in the directory rather than one file;
• replying when an allowed author's request names a directory the run cannot see;
• content hidden by what an element does rather than by where its markup ends (<template>, a hidden attribute, <select>, the fallback content of <video> or <canvas>). This needs GitHub's sanitizer checked first.

🤖 Generated with Claude Code