Skip to content

fix(issue-quality): require actionable reproduction detail #981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Wibias merged 2 commits into from
Aug 4, 2026
Merged

fix(issue-quality): require actionable reproduction detail #981

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8675d4a
fix(issue-quality): require actionable reproduction detail
Wibias Aug 4, 2026
3c547be
fix(issue-quality): reject generic request-action phrases
Wibias Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 86 additions & 3 deletions .github/scripts/issue-quality.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,12 +458,94 @@ function isTooTerseFeatureSection(text) {
}

/**
* Bug Reproduction needs steps or concrete signals. A title-like phrase with
* no commands, paths, digits, or product keywords is not actionable.
* Bug Reproduction needs concrete signals that let a maintainer reproduce the
* failure. Product keywords alone (e.g. "choose model deepseek" or "send a
* message in the codex plugin") are not actionable: the report must name a
* command, an error, a file/config path, or an exact observed output.
*/
// Commands and exact technical actions, e.g. "ocx start", "run bun",
// "send a streaming request", "curl https://...".
const REPRO_COMMAND_RE = new RegExp([
"\\b(?:run|start|stop|restart|install|launch|execute|reproduce|trigger|invoke)\\s+(?:(?:the|an|a)\\s+)?(?:ocx|bun|npm|pnpm|yarn|curl|node|codex|proxy|server|dashboard|plugin)\\b",
"\\b(?:ocx|bun|npm|pnpm|yarn|curl|node|codex)\\s+(?:start|run|stop|restart|install|config|--[a-z-]+)\\b",
"\\b(?:send|issue|make|post)\\s+(?:a|an|any)\\s+(?:streaming|api|http|json|completion|chat|config|auth|embedding|post|graphql|grpc)\\s+(?:request|call|command|prompt|query)\\b",
"\\b(?:send|issue|make|post)\\s+(?:a|an|any)\\s+(?:api|curl|endpoint|url)\\b",
"\\b(?:send|issue|make|post)\\s+(?:a|an|any)\\s+[\\w.-]+\\s+request\\s+to\\s+(?:the\\s+)?(?:endpoint|url|api|server|proxy|\\S+/\\S+)\\b",
"\\b(?:pip|npm|bun)\\s+install\\b",
"\\b(?:curl|wget)\\s+[^\\s]+",
].join("|"), "i");

// Error, exception, and failure tokens, plus status codes in status context
// (bare 3-digit numbers can be ports or version numbers).
const REPRO_FAILURE_RE = new RegExp([
"\\b(?:segfault|sigsegv|panic|abort|exception|traceback|stack\\s*trace|timeout|timed\\s*out|refused|reset|denied|failed?|error|crash|hang|hangs?|stuck|spinning|empty\\s*response)\\b",

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require context around generic failure words

Treating any occurrence of failed or error as actionable lets the exact low-detail reports this gate is meant to reject pass validation. With otherwise complete form metadata, a Reproduction containing only It failed. or There is an error. now makes validateIssue return valid, whereas the parent implementation rejected both as too vague. Require an accompanying command, error output/code, or sufficient descriptive context instead of allowing these generic words to bypass the word threshold by themselves.

Useful? React with 👍 / 👎.

"\\b(?:status\\s*(?:code\\s*)?|code\\s*|http\\s*)(?:is|of|:)?\\s*[1-5]\\d\\d\\b",
].join("|"), "i");

// File, config, and log paths such as ~/.codex/config.toml or C:\\logs\\ocx.log.
const REPRO_PATH_RE = new RegExp([
"~?/[\\w.@-]+(?:/[\\w.@-]+)+",
"[A-Za-z]:\\\\(?:[\\w.@-]+\\\\)+[\\w.@-]+",
"~?/[\\w.@-]+/[\\w.@-]+\\.(?:json|yaml|yml|toml|conf|log|env|txt|ts|js|tsx|jsx|sh|ps1|py)",
"[\\w.@-]+\\.(?:json|yaml|yml|toml|conf|log|env)\\b",
].join("|"));
const ACTIONABLE_REPRO_RE = new RegExp(
[REPRO_COMMAND_RE.source, REPRO_FAILURE_RE.source, REPRO_PATH_RE.source].join("|"),
"i",
);

// Sigil-only fences with no body content are never actionable.
const EMPTY_FENCE_RE = /^[ \t]{0,3}(?:```+|~~~+)\s*\n\s*\n[ \t]{0,3}(?:```+|~~~+)\s*$/;

/**
* True when a bug Reproduction names commands, error tokens, file/config
* paths, or exact technical actions. Product/model mentions without any of
* those signals (e.g. #977) are treated as unactionable. Fenced blocks only
* count as actionable when their body contains non-whitespace content.
*/
function hasActionableReproductionDetail(text) {
const c = clean(text);
if (!c) return false;
if (ACTIONABLE_REPRO_RE.test(c)) return true;
// Fenced blocks: only count when the body has non-whitespace content.
if (/```|~~~/.test(c)) {
const parts = stripFencedActionableContent(c);
if (parts) return true;
}
return false;
}

/**
* Walk the text looking for a fenced code block whose body contains
* non-whitespace content. Returns the non-empty body or null.
*/
function stripFencedActionableContent(text) {
const fenceRe = /^[ \t]{0,3}(`{3,}|~{3,})/;
const lines = text.split("\n");
let i = 0;
while (i < lines.length) {
const m = lines[i].match(fenceRe);
if (!m) { i++; continue; }
const marker = m[1];
const markerLen = marker.length;
// Find the closing fence on a later line.
let j = i + 1;
const endRe = new RegExp(
`^[ \\t]{0,3}${marker[0] === "`" ? "`" : "~"}{${markerLen},}[ \\t]*$`,
);
while (j < lines.length && !endRe.test(lines[j])) j++;
if (j > i + 1) {
const body = lines.slice(i + 1, j).join("\n");
if (body.trim()) return body;
}
i = j + 1;
}
return null;
}

function isTooTerseBugReproduction(text) {
if (isEmpty(text) || isPlaceholder(text)) return false;
if (hasConcreteDetail(text)) return false;
if (hasActionableReproductionDetail(text)) return false;

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Accept concise steps outside the verb whitelist

This replacement rejects actionable reproductions whenever they contain fewer than 12 words but use ordinary verbs or observed-output wording outside these English regexes. For example, Open dashboard. Select DeepSeek. Submit prompt. Observe blank result. previously passed via hasConcreteDetail, but now validateIssue marks it vague because none of open, select, submit, observe, or blank result is recognized. Since the issue workflow can auto-close invalid reports, broaden the actionable-step/output detection (or recognize structured step sequences) rather than requiring this fixed vocabulary.

Useful? React with 👍 / 👎.

return countWords(text) < 12;
}

Expand Down Expand Up @@ -950,6 +1032,7 @@ module.exports = {
isUnusableVersion,
countWords,
hasConcreteDetail,
hasActionableReproductionDetail,
labelForKind,
KIND_TO_LABEL,
hasSubstantialStructuredContent,
Expand Down
55 changes: 55 additions & 0 deletions .github/scripts/issue-quality.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ const {
isUnusableVersion,
countWords,
hasConcreteDetail,
hasActionableReproductionDetail,
rejectsWorkflowDispatchPullRequest,
rejectsWorkflowDispatchNonDefaultBranch,
} = require("./issue-quality.cjs");
Expand Down Expand Up @@ -462,6 +463,25 @@ describe("validateIssue - feature", () => {
assert.ok(vagueResult.reasons.some((r) => r.includes("too vague")));
});

it("treats only commands, errors, paths, or exact actions as actionable reproduction detail", () => {
assert.equal(hasActionableReproductionDetail("1. choose model deepseek\n2. send a message in codex plugin"), false);
assert.equal(hasActionableReproductionDetail("I want to work with deepseek in VSCode, but it dont reply"), false);
assert.equal(hasActionableReproductionDetail("1. ocx start --port 10100\n2. Send a request"), true);
assert.equal(hasActionableReproductionDetail("Run ocx start and send any streaming request."), true);
assert.equal(hasActionableReproductionDetail("ocx start on Raspberry Pi 4, send any streaming request."), true);
assert.equal(hasActionableReproductionDetail("send a request"), false);
assert.equal(hasActionableReproductionDetail("make a call"), false);
assert.equal(hasActionableReproductionDetail("post a command"), false);
assert.equal(hasActionableReproductionDetail("make an API call"), true);
assert.equal(hasActionableReproductionDetail("send an HTTP request"), true);
assert.equal(hasActionableReproductionDetail("send a request to /v1/responses"), true);
Comment on lines +472 to +477

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add isolated tests for the new regex branches.

The assertion at Line 477 uses "send a request to /v1/responses". The new pattern at .github/scripts/issue-quality.cjs Line 473 requires a token before request, such as "send a GET request to /v1/responses". The current assertion can pass through another actionable-detail matcher.

Add direct cases for the Line 472 and Line 473 alternatives. This prevents false coverage of the new patterns.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/scripts/issue-quality.test.cjs around lines 472 - 477, Add isolated
assertions in the hasActionableReproductionDetail tests for each new regex
alternative in issue-quality.cjs, including inputs that explicitly contain the
required token before “request” and the corresponding expected true results.
Replace or supplement the ambiguous “send a request to /v1/responses” case so
coverage cannot come from another matcher.

assert.equal(hasActionableReproductionDetail("The proxy returns HTTP 502 after the first streaming chunk."), true);
assert.equal(hasActionableReproductionDetail("Paste ~/.codex/config.toml, then restart the proxy."), true);
assert.equal(hasActionableReproductionDetail("```\n\n```"), false);
assert.equal(hasActionableReproductionDetail("~~~\n\n~~~"), false);
assert.equal(hasActionableReproductionDetail("```\nSIGSEGV at 0x0000\n```"), true);
});

it("rejects fenced placeholder-only examples", () => {
const fencedPlaceholders = [
"```\nN/A\n```",
Expand Down Expand Up @@ -864,6 +884,41 @@ describe("validateIssue - bug", () => {
assert.ok(result.reasons.some((r) => /Reproduction/i.test(r) && /vague/i.test(r)));
});

it("rejects a #977-shaped bug with product keywords but no actionable reproduction", () => {
const body = [
"### Client or integration",
"Other",
"### Area",
"Proxy and routing",
"### Summary",
"I want to work with deepseek in VSCode, but it dont reply,just thinking",
"### Reproduction",
"1.choose model deepseek",
"2.send a message in codex plugin",
"### Version",
"2.10.0",
"### Operating system",
"Ubuntu 24.04",
"### Provider and model",
"deepseek",
].join("\n");
const result = validateIssue({
title: "Dont work in VSCode Codex plugin",
body,
labels: ["bug", "proxy"],
});
assert.equal(result.kind, "bug");
assert.equal(result.valid, false);
assert.ok(
result.reasons.some((r) => /Reproduction/i.test(r) && /vague/i.test(r)),
`Expected a vague Reproduction reason, got: ${result.reasons.join("; ")}`,
);
assert.ok(
result.guidance.some((g) => /commands|steps/i.test(g)),
`Expected reproduction guidance, got: ${result.guidance.join("; ")}`,
);
});

it("rejects unknown Operating system stand-ins on the new bug form", () => {
const body = [
"### Client or integration",
Expand Down
Loading