From 29025e9bdd3ac851fc73c07cc6d31ce29b544fce Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:19:18 -0500 Subject: [PATCH 1/9] feat(implementation): observability-by-default reference and an advisory mutation run implement reads reference/observability.md before writing or changing code that runs at runtime: structured logging with message templates and scopes, compile-time logging on hot paths, telemetry configured from the environment, cross-cutting logging in middleware or pipeline behaviors, log levels raised only through untracked local config, and errors recorded with full context. At completion, when the mutation-testing plugin is enabled, Step 5 runs /mutation-testing:audit --persist-findings on the committed diff, reports the result and continues whatever the score. Adds three eval cases: the reference read and its rules on a scaffolded Node service, and the mutation step with the plugin enabled and not. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- ...plement-observability-mutation-3c7e9a14.md | 19 +++ .../case.yaml | 23 ++++ .../samples/names-missing-plugin.json | 18 +++ .../samples/no-audit-invocation.json | 14 +++ .../case.yaml | 27 +++++ .../.claude-plugin/plugin.json | 5 + .../skills/audit/SKILL.md | 17 +++ .../samples/audit-before-review.json | 18 +++ .../samples/audit-invocation.json | 30 +++++ .../im-observability-runtime-code/case.yaml | 44 +++++++ .../samples/endpoint-from-env.json | 18 +++ .../samples/error-logged-with-object.json | 22 ++++ .../samples/fields-not-built-strings.json | 18 +++ .../samples/no-endpoint-literal.json | 18 +++ .../samples/no-raised-level-committed.json | 14 +++ .../samples/reads-reference.json | 32 +++++ .../im-observability-runtime-code/scaffold.sh | 112 ++++++++++++++++++ .../implementation/reference/observability.md | 95 +++++++++++++++ .../implementation/skills/implement/SKILL.md | 13 +- 19 files changed, 552 insertions(+), 5 deletions(-) create mode 100644 .changes/implementation/feat-implement-observability-mutation-3c7e9a14.md create mode 100644 plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml create mode 100644 plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/names-missing-plugin.json create mode 100644 plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json create mode 100644 plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml create mode 100644 plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/.claude-plugin/plugin.json create mode 100644 plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/skills/audit/SKILL.md create mode 100644 plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json create mode 100644 plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json create mode 100644 plugins/implementation/evals/im-observability-runtime-code/case.yaml create mode 100644 plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json create mode 100644 plugins/implementation/evals/im-observability-runtime-code/samples/error-logged-with-object.json create mode 100644 plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json create mode 100644 plugins/implementation/evals/im-observability-runtime-code/samples/no-endpoint-literal.json create mode 100644 plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json create mode 100644 plugins/implementation/evals/im-observability-runtime-code/samples/reads-reference.json create mode 100755 plugins/implementation/evals/im-observability-runtime-code/scaffold.sh create mode 100644 plugins/implementation/reference/observability.md diff --git a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md new file mode 100644 index 00000000000..48175b2b5c3 --- /dev/null +++ b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md @@ -0,0 +1,19 @@ +--- +bump: minor +--- + +### Added + +- **Observability by default.** `/implementation:implement` reads the new + `reference/observability.md` before writing or changing code that runs at runtime: structured + logs with message templates and scopes or child loggers, compile-time or allocation-free logging + on hot paths, telemetry configured from the environment (the `OTEL_EXPORTER_OTLP_ENDPOINT` + family) instead of literals, cross-cutting logging and timing in middleware, decorators or + pipeline behaviors with in-method logs where the method decides something, log levels raised + only through untracked local config, and caught errors recorded with the exception and the + operation's fields. Short .NET and JavaScript examples, with pointers to the live docs. +- **Advisory mutation run at completion.** When the `mutation-testing` plugin is enabled, Step 5 + runs `/mutation-testing:audit --persist-findings` on the committed change's diff once the + affected tests are green, reports the score, survivors and findings path (or the audit's refusal) + and continues to the review hand-off whatever the score. A run whose tracked source could not be + restored stops before review. Without the plugin, the handoff says once what it would add. diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml new file mode 100644 index 00000000000..e9f26de3352 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml @@ -0,0 +1,23 @@ +schema_version: "1.1" +name: im-mutation-advisory-not-enabled +description: 'Step 5 advisory mutation run, plugin not enabled: with only the implementation plugin loaded, `/mutation-testing:audit` is not among the available skills, so the remaining Step 5 items carry no mutation invocation and the handoff says once that the mutation-testing plugin would check the tests. Read-only: the session prints the steps and runs nothing.' +tags: [implement, completion, mutation-testing, availability-gate, read-only] +runs: 3 +plugins: ["../.."] +expected_outcome: 'The numbered Step 5 list has no `/mutation-testing:audit` invocation, and one line says the mutation-testing plugin is not enabled or would check whether the tests detect injected faults.' +execution: + prompt: '/implementation:implement Do not read files, run commands or invoke skills. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Print, as a numbered list, every remaining Step 5 item in the order you would run it, each with its exact command or skill invocation, then stop.' + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 15 + timeout_seconds: 600 +graders: + - name: no-audit-invocation + type: regex + pattern: '/mutation-testing:audit[^\n]*--persist-findings' + match: not_contains + target: last_message + - name: names-missing-plugin + type: regex + pattern: '\bmutation[- ]testing\b[^\n]*\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b|\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b[^\n]*\bmutation[- ]testing\b' + flags: i + target: last_message diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/names-missing-plugin.json b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/names-missing-plugin.json new file mode 100644 index 00000000000..634a2ca5ff6 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/names-missing-plugin.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "3. Advisory mutation run: not run, the mutation-testing plugin is not enabled.", + "why": "says the plugin is not enabled" + }, + { + "answer": "Handoff note: the `mutation-testing` plugin would check whether these tests detect injected faults.", + "why": "says what the plugin would do" + } + ], + "fail": [ + { + "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`\n3. `/verification:confirm .work/refunds/PLAN.md`", + "why": "the missing plugin is never mentioned" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json new file mode 100644 index 00000000000..2e1bd3a8ddd --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "1. `/toolchain:check`\n2. Affected tests\n3. Mutation run: skipped, the mutation-testing plugin is not enabled; it would check whether the tests detect injected faults.\n4. `/review:quality-gate`", + "why": "no invocation, fallback named" + } + ], + "fail": [ + { + "answer": "3. `/mutation-testing:audit --persist-findings`\n4. `/review:quality-gate`", + "why": "invokes a skill that is not available" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml new file mode 100644 index 00000000000..bcd0aeda9f5 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml @@ -0,0 +1,27 @@ +schema_version: "1.1" +name: im-mutation-advisory-when-enabled +description: 'Step 5 advisory mutation run, plugin enabled: with a mutation-testing stand-in (the audit skill''s name and flags) loaded beside implementation, the remaining Step 5 items for a committed, green change include `/mutation-testing:audit --persist-findings` on its default diff scope (no --full, --paths or --exercised) ahead of the review hand-off, and a low score with survivors does not stop the run. Read-only: the session prints the steps and runs nothing.' +tags: [implement, completion, mutation-testing, availability-gate, read-only] +runs: 3 +plugins: ["../..", "mutation-testing-stub"] +expected_outcome: 'The numbered Step 5 list has a `/mutation-testing:audit --persist-findings` step with no scope flag, placed before `/review:quality-gate`, and states that the reported 41% score and seven survivors are reported and the run continues to the review hand-off rather than stopping.' +execution: + prompt: '/implementation:implement Do not read files, run commands or invoke skills. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Print, as a numbered list, every remaining Step 5 item in the order you would run it, each with its exact command or skill invocation. Then say what you would do next if that mutation run reported a covered-code score of 41% with seven surviving mutants, and stop.' + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 15 + timeout_seconds: 600 +graders: + - name: audit-invocation + type: regex + pattern: '^(?=[^\n]*/mutation-testing:audit\b)(?=[^\n]*--persist-findings)(?![^\n]*--(?:full|paths|exercised)\b)' + flags: mi + target: last_message + - name: audit-before-review + type: regex + pattern: '/mutation-testing:audit\b[\s\S]*?/review:quality-gate' + flags: i + target: last_message + - name: score-does-not-block + type: llm + arm: both + criteria: 'PASS only if the final message says that the 41% score and seven survivors are reported (for example the score, survivor count or persisted findings path) and that the run then continues to the following Step 5 items or the review hand-off, without stopping, failing, or making killing those survivors a precondition for continuing. FAIL if it treats the score as a gate, stops the run on it, or says nothing about what happens after that result.' diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/.claude-plugin/plugin.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/.claude-plugin/plugin.json new file mode 100644 index 00000000000..5d8372f0b12 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/.claude-plugin/plugin.json @@ -0,0 +1,5 @@ +{ + "name": "mutation-testing", + "version": "0.0.0", + "description": "Eval-only stand-in for the mutation-testing plugin: the audit skill's name and flags, with no scripts or hooks." +} diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/skills/audit/SKILL.md b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/skills/audit/SKILL.md new file mode 100644 index 00000000000..7dcf14a2bb0 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/skills/audit/SKILL.md @@ -0,0 +1,17 @@ +--- +description: "Mutation analysis that reports surviving mutants, on the diff or, with `--exercised`, on the production code the changed tests call. Use when the user asks to run mutation testing or for a mutation score on a change, asks whether the tests actually check the code, or wants the survivors persisted for the fix pass; after tests go green and before review." +argument-hint: "--exercised --full --paths --max --no-suppress --persist-findings --record-mutants --replay-mutants" +user-invocable: true +disable-model-invocation: false +--- + +# Mutation audit (eval stand-in) + +Arguments: `$ARGUMENTS` + +With no scope flag the audit mutates the committed diff against the base branch. `--full` mutates +the whole configured scope, `--paths ` named paths, and `--exercised` the code the changed +tests call. `--persist-findings` also writes the survivors as a findings file for the fix pass. + +This sandbox has no mutation tool: report `covered-code score 41%, 7 survivors, findings written +to .work/reviews/feat-refunds/mutation-audit.md` and stop. Write nothing. diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json new file mode 100644 index 00000000000..1257f612010 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "3. `/mutation-testing:audit --persist-findings`\n4. Deviation fold-back\n6. Hand off: `/review:quality-gate`, then `/verification:confirm .work/refunds/PLAN.md`", + "why": "mutation run comes before the review hand-off" + } + ], + "fail": [ + { + "answer": "1. `/review:quality-gate`\n2. `/mutation-testing:audit --persist-findings`", + "why": "mutation run after review" + }, + { + "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`", + "why": "no mutation run" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json new file mode 100644 index 00000000000..f3d3aa8b4b1 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "1. Final build check: `/toolchain:check`\n2. Affected tests: `python3 -m pytest`\n3. Advisory mutation run: `/mutation-testing:audit --persist-findings`\n4. Hand off: `/review:quality-gate`", + "why": "bare diff scope with persisted findings" + }, + { + "answer": "Step 3 - Skill /mutation-testing:audit with args --persist-findings (default diff scope)", + "why": "same invocation in prose" + } + ], + "fail": [ + { + "answer": "3. `/mutation-testing:audit`\n4. `/review:quality-gate`", + "why": "survivors not persisted" + }, + { + "answer": "3. `/mutation-testing:audit --full --persist-findings`", + "why": "whole configured scope instead of the diff" + }, + { + "answer": "3. `/mutation-testing:audit --exercised --persist-findings`", + "why": "exercised scope instead of the default diff scope" + }, + { + "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`", + "why": "no mutation run" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/case.yaml b/plugins/implementation/evals/im-observability-runtime-code/case.yaml new file mode 100644 index 00000000000..2eb578ecd2b --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/case.yaml @@ -0,0 +1,44 @@ +schema_version: "1.1" +name: im-observability-runtime-code +description: 'Observability by default: a plan adds a refund path that calls a payment gateway and wires a metrics client to the team''s collector. implement reads reference/observability.md before writing the runtime code, takes the collector endpoint from the environment instead of the staging URL the plan mentions, logs with fields rather than built strings, records the gateway failure with its error object, and commits no raised log level. Needs --scaffold --allow-tools Bash Edit Write.' +tags: [implement, observability, feature, scaffold] +runs: 3 +plugins: ["../.."] +expected_outcome: 'src/refunds.js logs through the logger with a fields object (no template literal or concatenated message), its failure path logs the caught error as a field and throws RefundError; src/app.js reads the collector endpoint from OTEL_EXPORTER_OTLP_ENDPOINT, holds no URL literal and no debug or trace level; the session read reference/observability.md before writing.' +context: + scaffold_script: scaffold.sh +execution: + prompt: '/implementation:implement feature - execute the approved plan at .work/refunds/PLAN.md. The plan is approved, and committing on the current branch is allowed. Do not push.' + allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write] + max_turns: 80 + timeout_seconds: 1200 +graders: + - name: reads-reference + type: tool_used + tool: Read + input_match: 'reference/observability\.md' + min: 1 + arm: with-only + - name: endpoint-from-env + type: regex + pattern: 'process\.env(?:\.OTEL_EXPORTER_OTLP_ENDPOINT\b|\[\s*["''`]OTEL_EXPORTER_OTLP_ENDPOINT)' + target: { source: file, path: src/app.js } + - name: no-endpoint-literal + type: regex + pattern: 'https?://|otel-collector' + match: not_contains + target: { source: file, path: src/app.js } + - name: no-raised-level-committed + type: regex + pattern: 'level\s*:\s*["''`](?:debug|trace)["''`]' + match: not_contains + target: { source: file, path: src/app.js } + - name: fields-not-built-strings + type: regex + pattern: '\.(?:debug|info|warn|error)\(\s*(?:`|["''][^"''\n]*["'']\s*\+)' + match: not_contains + target: { source: file, path: src/refunds.js } + - name: error-logged-with-object + type: regex + pattern: '\.error\(\s*\{[^}]*\b(?:err|error)\s*(?:[,}]|:\s*\w+\s*[,}])' + target: { source: file, path: src/refunds.js } diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json b/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json new file mode 100644 index 00000000000..881f6c413c4 --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "endpoint read from the environment" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env[\"OTEL_EXPORTER_OTLP_ENDPOINT\"],\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "bracket access to the same variable" + } + ], + "fail": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: \"http://otel-collector.staging:4318\",\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "staging URL hard-coded" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/error-logged-with-object.json b/plugins/implementation/evals/im-observability-runtime-code/samples/error-logged-with-object.json new file mode 100644 index 00000000000..06dd4c75143 --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/error-logged-with-object.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "error object as err field" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ error: err, orderId }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "error object as error field" + } + ], + "fail": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ message: err.message }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "only the message, no error object" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "failure not recorded" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json b/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json new file mode 100644 index 00000000000..0be3e64e6eb --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "fields object and constant message" + } + ], + "fail": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info(`order ${orderId} refunded`);\n}\n", + "why": "template literal message" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info(\"order \" + orderId + \" refunded\");\n}\n", + "why": "concatenated message" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/no-endpoint-literal.json b/plugins/implementation/evals/im-observability-runtime-code/samples/no-endpoint-literal.json new file mode 100644 index 00000000000..246e76fb37b --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/no-endpoint-literal.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "no URL literal" + } + ], + "fail": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: \"http://otel-collector.staging:4318\",\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "staging URL hard-coded" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? \"http://localhost:4318\",\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "literal fallback endpoint in code" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json b/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json new file mode 100644 index 00000000000..9b348c93d8a --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "level from the environment, quiet default" + } + ], + "fail": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: \"debug\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "debug level committed" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/reads-reference.json b/plugins/implementation/evals/im-observability-runtime-code/samples/reads-reference.json new file mode 100644 index 00000000000..472d0b7452a --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/reads-reference.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": "/plugins/implementation/reference/observability.md" + } + } + ], + "why": "the reference was read" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": "/plugins/implementation/reference/type-discipline.md" + } + } + ], + "why": "another reference only" + }, + { + "answer": [], + "why": "no tool call" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh b/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh new file mode 100755 index 00000000000..59ffd71d0cc --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# A small Node service with a structured logger and a metrics client that takes its collector +# endpoint as an argument, and an approved plan to add a refund path that calls a payment gateway. +set -euo pipefail + +git init -q -b feat/refunds +git config user.name eval +git config user.email eval@example.invalid +mkdir -p src tests .work/refunds + +cat >package.json <<'JSON' +{ + "name": "orders", + "private": true, + "type": "module", + "scripts": { "test": "node --test tests/" } +} +JSON + +cat >src/logger.js <<'JS' +const LEVELS = { debug: 10, info: 20, warn: 30, error: 40 }; + +export function createLogger({ level = "info", bindings = {}, write = (line) => process.stdout.write(line + "\n") } = {}) { + const min = LEVELS[level] ?? LEVELS.info; + const emit = (name) => (fields, msg) => { + if (LEVELS[name] < min) return; + const record = { level: name, time: Date.now(), ...bindings, ...fields, msg }; + if (fields && fields.err instanceof Error) { + record.err = { message: fields.err.message, stack: fields.err.stack }; + } + write(JSON.stringify(record)); + }; + return { + debug: emit("debug"), + info: emit("info"), + warn: emit("warn"), + error: emit("error"), + child: (more) => createLogger({ level, bindings: { ...bindings, ...more }, write }), + }; +} +JS + +cat >src/metrics.js <<'JS' +export function createMetricsClient({ endpoint, serviceName }) { + if (!endpoint) throw new Error("metrics endpoint is required"); + const counts = new Map(); + return { + endpoint, + serviceName, + count(name, value = 1) { + counts.set(name, (counts.get(name) ?? 0) + value); + }, + snapshot: () => Object.fromEntries(counts), + }; +} +JS + +cat >src/app.js <<'JS' +import { createLogger } from "./logger.js"; + +export function buildApp() { + const logger = createLogger(); + return { logger }; +} +JS + +cat >tests/logger.test.js <<'JS' +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createLogger } from "../src/logger.js"; + +test("child bindings appear on every line", () => { + const lines = []; + const log = createLogger({ write: (l) => lines.push(JSON.parse(l)) }).child({ requestId: "r1" }); + log.info({ orderId: "o1" }, "hello"); + assert.equal(lines[0].requestId, "r1"); + assert.equal(lines[0].orderId, "o1"); +}); +JS + +printf '.work/\nnode_modules/\n.env.local\n' >.gitignore + +cat >.work/refunds/PLAN.md <<'MD' +# Refunds + +## Brief + +Goal: an order can be refunded through the payment gateway, and the team can see refunds in its +collector. + +## Plan + +### Phase 1: refund path [TODO] + +- [ ] `src/refunds.js`: `export async function refundOrder({ gateway, logger, metrics }, orderId)` + calls `await gateway.refund(orderId)`. On success it records the refund and counts it with + `metrics.count("refunds")`. When the gateway throws, the failure is recorded and a + `RefundError` (exported from the same file) is thrown to the caller. +- [ ] `src/app.js`: `buildApp()` also creates the metrics client with `createMetricsClient` from + `src/metrics.js`, service name `orders`, sending to the team's OTLP collector (in staging + that is `http://otel-collector.staging:4318`), and returns + `{ logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) }`. +- [ ] Tests in `tests/refunds.test.js` with a fake gateway, for the success and the failure path. +- While building this you will want debug-level logs from the refund path. + +Acceptance: `npm test` passes. + +Approval: user, in the eval prompt. +MD + +git add .gitignore package.json src tests +git commit -q -m "add order service skeleton" diff --git a/plugins/implementation/reference/observability.md b/plugins/implementation/reference/observability.md new file mode 100644 index 00000000000..aab09265e78 --- /dev/null +++ b/plugins/implementation/reference/observability.md @@ -0,0 +1,95 @@ +# Observability by default + +Read this before writing or changing code that runs at runtime: a service, an API handler, a +worker or job, a CLI, a hook or a script someone else runs. The rules hold in every language; the +two examples show one way to follow them in .NET and in JavaScript or TypeScript. Where the +repository already has a logging or telemetry setup, follow it, and apply these rules inside it. + +## Rules + +| Situation | Do | Reason | +|---|---|---| +| Writing a log line | Log a constant message template with named fields for the values (`"Order {OrderId} cancelled"` plus the id as a field), never a string built by concatenation or interpolation | The fields stay queryable, and one template groups every occurrence of the event | +| Values every line in a unit of work shares (request id, order id, tenant) | Attach them once with a logging scope or a child logger, not by repeating them in each call | Each line carries the context without each call site having to remember it | +| A log call on a hot path (a loop, a per-request or per-message path) | Use the language's compile-time or allocation-free logging form, and skip building costly arguments when the level is off | Logging that costs work while disabled taxes every call | +| Sending traces, metrics or logs to a collector | Read the exporter endpoint, headers and service name from the environment (the OpenTelemetry `OTEL_EXPORTER_OTLP_ENDPOINT` family of variables, or the platform's configuration layer), never a literal in code | One build runs in every environment, and no endpoint or key is committed | +| Logging or timing that every request, handler or command needs | Put it in one cross-cutting place: middleware, a decorator, a pipeline behavior, an interceptor | One owner for the concern, and handlers keep only their own logic | +| A method with its own decision worth recording (a branch taken, a retry, a fallback) | Log it inside that method, at the level the event deserves | The cross-cutting layer cannot see the method's own decisions | +| More detail needed while developing | Raise the level locally through untracked config: an environment variable, a git-ignored local settings file, user secrets. Committed config keeps the quiet default | Teammates and deployed environments keep their level; a raised level is never merged by accident | +| Catching an error | Catch where the code can act on it, and record it with the exception object (stack included) and the fields that identify the operation; rethrow or return a failure the caller can see | A message without the exception loses the stack, and a swallowed error leaves no trace | + +Never log secrets, tokens or personal data in a field or a message; redact at the logging layer +when the platform offers it. + +## .NET + +```csharp +public static partial class OrderLog +{ + [LoggerMessage(Level = LogLevel.Information, Message = "Order {OrderId} cancelled")] + public static partial void OrderCancelled(this ILogger logger, string orderId); + + [LoggerMessage(Level = LogLevel.Error, Message = "Payment gateway refused order {OrderId}")] + public static partial void GatewayRefused(this ILogger logger, Exception error, string orderId); +} + +using (logger.BeginScope(new Dictionary { ["RequestId"] = requestId })) +{ + try + { + await gateway.RefundAsync(orderId); + logger.OrderCancelled(orderId); + } + catch (GatewayException ex) + { + logger.GatewayRefused(ex, orderId); + throw; + } +} + +// Exporter endpoint comes from OTEL_EXPORTER_OTLP_ENDPOINT; nothing is passed in code. +builder.Services.AddOpenTelemetry().UseOtlpExporter(); +``` + +Raise a level locally with an environment variable such as `Logging__LogLevel__MyApp=Debug`, or +with user secrets, never by editing the committed `appsettings.json`. + +## JavaScript and TypeScript + +```ts +const log = baseLogger.child({ requestId, orderId }); + +try { + await gateway.refund(orderId); + log.info({ event: "order.cancelled" }, "order cancelled"); +} catch (err) { + log.error({ err }, "payment gateway refused order"); + throw err; +} + +const exporter = new OTLPTraceExporter(); // reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment +const logger = pino({ level: process.env.LOG_LEVEL ?? "info" }); +``` + +Raise a level locally with `LOG_LEVEL=debug` in the shell or a git-ignored `.env.local`. + +## Where to read the specifics + +Read these live when a choice depends on the current API or option names. + +- Message templates, scopes, exceptions and level configuration in .NET: + [Log message template](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#log-message-template), + [Log scopes](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#log-scopes), + [Log exceptions](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#log-exceptions), + [Configure logging](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#configure-logging). +- Compile-time logging in .NET: + [Basic usage](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/source-generation#basic-usage). +- OTLP exporter configuration from the environment: + [Environment Variables](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/README.md#environment-variables). +- Configuration layering and values kept out of source control in ASP.NET Core: + [Security and user secrets](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/#security-and-user-secrets), + [Environment Variables Configuration Provider](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/#environment-variables-configuration-provider). +- Middleware as the place for cross-cutting work in ASP.NET Core: + [Create a middleware pipeline with WebApplication](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/middleware/#create-a-middleware-pipeline-with-webapplication). +- Child loggers in pino: + [Child loggers](https://github.com/pinojs/pino/blob/main/docs/child-loggers.md#child-loggers). diff --git a/plugins/implementation/skills/implement/SKILL.md b/plugins/implementation/skills/implement/SKILL.md index d99fc59a4b9..f7d89a9cc50 100644 --- a/plugins/implementation/skills/implement/SKILL.md +++ b/plugins/implementation/skills/implement/SKILL.md @@ -100,7 +100,7 @@ Core execution loop. Key discipline: **validate after each logical block, not ju ### Execution cadence -1. **Implement one logical block**. A single concern, function, class, or feature slice. Not the entire plan at once. When the block declares or changes a type (a record, a union, a parse function's result, a field's type), first read [`${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md`](${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md) and make the decision its matching row names, applying the language idioms below, because the code settles the casts, entry-point parses and exhaustive branches that a design, when there was one, never saw +1. **Implement one logical block**. A single concern, function, class, or feature slice. Not the entire plan at once. When the block declares or changes a type (a record, a union, a parse function's result, a field's type), first read [`${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md`](${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md) and make the decision its matching row names, applying the language idioms below, because the code settles the casts, entry-point parses and exhaustive branches that a design, when there was one, never saw. When the block adds or changes code that runs at runtime (a service, handler, worker, CLI, hook or script), first read [`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) and write the block's logging, telemetry and error handling by its rules 2. **Build check**. Invoke `/toolchain:check` (via Skill tool) for the affected ecosystem after each block when the `toolchain` plugin is enabled; otherwise run the project's own build/test command directly. Catch compilation errors immediately, not after 5 files of changes. In non-interactive runs, tier the in-loop cost: typecheck/compile and the touched test files run per block; the broader affected-ecosystem test suite runs at phase boundaries and Step 5. Early detection stays, redundant full-suite passes go 3. **Test (TDD by default)**. When the `tdd` plugin is enabled, invoke `/tdd:principles` via Skill tool **before writing the first test** for authoritative guidance on what to test, which testing style fits (output/state/communication), and when to mock. Then follow Red-Green-Refactor **one test at a time**: write a single failing test for the smallest slice of behavior, confirm it fails (red), implement the minimum to pass (green), refactor. Then move to the next slice. **Minimum to pass means the smallest correct implementation, not the smallest text that turns the bar green**. Never hardcode the test's expected values, special-case its inputs, or weaken an assertion; if a test seems wrong, fix the test deliberately and say so. **Do not write all tests upfront**. Writing one at a time keeps each red signal observable (proving the test can fail before code makes it pass) and stops you over-fitting code to tests written against a design that doesn't exist yet. TDD is the fallback when the consuming project does not declare another testing cadence. Skip only when genuinely impractical (e.g., pure infrastructure wiring with no testable logic, or UI rendering with no logic behind the seam), or when project policy says otherwise. A consumer can opt out in its `CLAUDE.md` / rules, for example: `Use tests-after for implementation work; do not use test-first TDD.` That project policy overrides this fallback and the mode context guidance 4. **Fewer-lines check**. In every mode, before a block counts as done, ask whether it could say the same thing in fewer lines with identical behavior, and shrink it if so before the commit checkpoint. It applies to the result `integration_posture` calls for, a `day-one` redesign included, and never argues against the reshaping. It runs here because lines beyond the task's behavior and criteria are otherwise caught only in review @@ -268,10 +268,11 @@ When all planned work is done: 1. **Final build check**. Invoke `/toolchain:check` via the Skill tool for all affected ecosystems when the `toolchain` plugin is enabled; otherwise run the project's own build/test command 2. **Run all affected tests**. Include the tests you wrote and any tests your changes could impact. This end gate is always green: a planned breakage declaration covers only the span it names, never the completed work, so a failure still present here is a failure -3. **Deviation fold-back**. When a `DEVIATIONS.md` exists for this work (the non-interactive fork wrote one, or the session opted in per Step 3), read it now and emit one plan-amendment bullet per unresolved deviation or human-decision entry: what the plan should say next time, or what still needs a person. The log is the run's memory; a completion that never reads it back hands the PR reviewer deviations the author already knew about. Write the bullets into PLAN.md as dated pivot notes on the sections they amend, per `/planning:plan` "Mid-flight pivots", then repeat them in the handoff summary -4. **Rubber-duck advisor checkpoint (HIGH/CRITICAL only)**. For changes involving concurrency, security, cross-platform behavior, external API integration, or with significant divergence from the original plan, call the `advisor` tool (when available in the session) for a quick cross-model critique pass before the review gate. Skip for trivial changes. The `verification` level governs this checkpoint, resolved by the reader call in phase-boundary item 1 (run it here when no boundary ran): `standard` keeps the rule above and `strict` calls it for every change, trivial ones included. Under `light`, and `off`, which acts as `light` here, call no advisor and never critique the change inline in its place: the handoff summary says `verification: skipped ()` -5. **Hand off to the pre-PR sequence**. Hand off, do not re-order: that sequence owns the step order (invoke `/session-flow:workflow pre-pr` via the Skill tool when the `session-flow` plugin is enabled to read it; otherwise follow the consuming setup's own pre-PR checklist). Its order puts **review before outcome verification**, because the simplify pass sits between them and outcome verification must judge the code that ships. So: first suggest the Test-stage checks the change still needs beyond the affected tests above (coverage, test quality, end to end: the outcomes under Next), then the project's review flow (`/review:quality-gate` when the `review` plugin is enabled; otherwise the consuming setup's review step), then `/verification:confirm ` for outcome verification once the diff is final, handing it the plan path from Progress tracking when a plan exists, so a fresh session reads the Brief (when the `verification` plugin is enabled), then the PR (`/source-control:pull-request` when that plugin is enabled; otherwise whatever the consuming setup provides. The user controls timing). Do not commit-and-push unilaterally, final staging and PR creation belong to that flow. A run that ends without outcome verification against the Brief's outcome criteria (no `/verification:confirm`) states in the handoff summary, or in `DEVIATIONS.md` when one exists, that it did not run and why, so the omission is never invisible. In an orchestrated run, a source commit the orchestrator itself makes after the last phase gets the `phase-verifier` verdict before it is pushed, per `/implementation:implement-dispatch` "Phase boundaries" -6. **Completion reply**. Close by putting to the user every decision that is still theirs, each with your recommendation, or by saying none remain. Ahead of that, say what now exists, and wherever the plan or task left a call to you, say which way you went and why +3. **Advisory mutation run**. Once the change is committed (the audit's default scope is the committed diff, and it stops on an uncommitted target file), invoke `/mutation-testing:audit --persist-findings` via the Skill tool when it is among the available skills (enabled), with no scope argument, `--paths`, `--full` or `--exercised`, so it mutates this change's diff. It is advice, never a gate: whatever the score, survivors or a refusal (no config, red baseline, no test command), report the score, the survivor count and the persisted findings path, or the refusal and its reason, then continue to the next item; the persisted survivors are what `/review:fanout fix` or `/testing:write` acts on. The one outcome that is not advice is a run that reports it could not restore tracked source: surface it and stop before review, since the tree is no longer the one that was tested. When the `mutation-testing` plugin is not enabled, run nothing and say once in the handoff summary that the `mutation-testing` plugin would check whether the change's tests detect injected faults +4. **Deviation fold-back**. When a `DEVIATIONS.md` exists for this work (the non-interactive fork wrote one, or the session opted in per Step 3), read it now and emit one plan-amendment bullet per unresolved deviation or human-decision entry: what the plan should say next time, or what still needs a person. The log is the run's memory; a completion that never reads it back hands the PR reviewer deviations the author already knew about. Write the bullets into PLAN.md as dated pivot notes on the sections they amend, per `/planning:plan` "Mid-flight pivots", then repeat them in the handoff summary +5. **Rubber-duck advisor checkpoint (HIGH/CRITICAL only)**. For changes involving concurrency, security, cross-platform behavior, external API integration, or with significant divergence from the original plan, call the `advisor` tool (when available in the session) for a quick cross-model critique pass before the review gate. Skip for trivial changes. The `verification` level governs this checkpoint, resolved by the reader call in phase-boundary item 1 (run it here when no boundary ran): `standard` keeps the rule above and `strict` calls it for every change, trivial ones included. Under `light`, and `off`, which acts as `light` here, call no advisor and never critique the change inline in its place: the handoff summary says `verification: skipped ()` +6. **Hand off to the pre-PR sequence**. Hand off, do not re-order: that sequence owns the step order (invoke `/session-flow:workflow pre-pr` via the Skill tool when the `session-flow` plugin is enabled to read it; otherwise follow the consuming setup's own pre-PR checklist). Its order puts **review before outcome verification**, because the simplify pass sits between them and outcome verification must judge the code that ships. So: first suggest the Test-stage checks the change still needs beyond the affected tests above (coverage, test quality, end to end: the outcomes under Next), then the project's review flow (`/review:quality-gate` when the `review` plugin is enabled; otherwise the consuming setup's review step), then `/verification:confirm ` for outcome verification once the diff is final, handing it the plan path from Progress tracking when a plan exists, so a fresh session reads the Brief (when the `verification` plugin is enabled), then the PR (`/source-control:pull-request` when that plugin is enabled; otherwise whatever the consuming setup provides. The user controls timing). Do not commit-and-push unilaterally, final staging and PR creation belong to that flow. A run that ends without outcome verification against the Brief's outcome criteria (no `/verification:confirm`) states in the handoff summary, or in `DEVIATIONS.md` when one exists, that it did not run and why, so the omission is never invisible. In an orchestrated run, a source commit the orchestrator itself makes after the last phase gets the `phase-verifier` verdict before it is pushed, per `/implementation:implement-dispatch` "Phase boundaries" +7. **Completion reply**. Close by putting to the user every decision that is still theirs, each with your recommendation, or by saying none remain. Ahead of that, say what now exists, and wherever the plan or task left a call to you, say which way you went and why ## Skill chaining during execution @@ -283,6 +284,8 @@ When all planned work is done: | Worker-routed phase or autonomous orchestration | Invoke `/implementation:implement-dispatch` via Skill tool | | Divergence detected (major) | Route back to the planning skill (invoke `/planning:plan review` via the Skill tool when enabled) | | Technical question mid-implementation | Invoke `/discovery:research` via the Skill tool (when enabled), otherwise disciplined multi-source research | +| Block adds or changes code that runs at runtime | Read [`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) before writing it | +| Affected tests green at completion, change committed | Invoke `/mutation-testing:audit --persist-findings` via the Skill tool (when enabled); report its result and continue whatever the score (Step 5) | | HIGH/CRITICAL change at completion | Call the `advisor` tool, rubber-duck checkpoint before review, unless the `verification` level skips it | | All implementation complete, tests pass | Hand off to the pre-PR sequence in its own order. The Test-stage checks the change still needs (see Next), then review, then outcome verification once the diff is final, then the PR: `/review:quality-gate`, then `/verification:confirm`, with the PLAN.md path as its argument when a plan exists, then `/source-control:pull-request`. Each invoked via the Skill tool, each when its plugin is enabled | From 08ccc415f70e03bbadb7dbc4a34c821d97ff4b43 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:23:53 -0500 Subject: [PATCH 2/9] fix(implementation): keep the skipped mutation item in the report and widen eval graders A run without the mutation-testing plugin left the item out of its report in one of three eval runs; the fallback now keeps the item as not run. The observability case accepts an environment passed in as a parameter, and the not-enabled case judges whether a mutation run happens rather than whether the invocation text appears. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- .../im-mutation-advisory-not-enabled/case.yaml | 13 ++++++------- .../samples/no-audit-invocation.json | 14 -------------- .../evals/im-observability-runtime-code/case.yaml | 2 +- .../samples/endpoint-from-env.json | 4 ++++ .../im-observability-runtime-code/scaffold.sh | 2 -- plugins/implementation/skills/implement/SKILL.md | 2 +- 6 files changed, 12 insertions(+), 25 deletions(-) delete mode 100644 plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml index e9f26de3352..e833b0e9ec7 100644 --- a/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml @@ -1,21 +1,20 @@ schema_version: "1.1" name: im-mutation-advisory-not-enabled -description: 'Step 5 advisory mutation run, plugin not enabled: with only the implementation plugin loaded, `/mutation-testing:audit` is not among the available skills, so the remaining Step 5 items carry no mutation invocation and the handoff says once that the mutation-testing plugin would check the tests. Read-only: the session prints the steps and runs nothing.' +description: 'Step 5 advisory mutation run, plugin not enabled: with only the implementation plugin loaded, `/mutation-testing:audit` is not among the available skills, so the remaining Step 5 items run no mutation audit and the handoff says once that the mutation-testing plugin would check the tests. Read-only: the session prints the steps and runs nothing.' tags: [implement, completion, mutation-testing, availability-gate, read-only] runs: 3 plugins: ["../.."] -expected_outcome: 'The numbered Step 5 list has no `/mutation-testing:audit` invocation, and one line says the mutation-testing plugin is not enabled or would check whether the tests detect injected faults.' +expected_outcome: 'The numbered Step 5 list runs no `/mutation-testing:audit`, and one line says the mutation-testing plugin is not enabled or would check whether the tests detect injected faults.' execution: prompt: '/implementation:implement Do not read files, run commands or invoke skills. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Print, as a numbered list, every remaining Step 5 item in the order you would run it, each with its exact command or skill invocation, then stop.' allowed_tools: [Read, Glob, Grep, Skill] max_turns: 15 timeout_seconds: 600 graders: - - name: no-audit-invocation - type: regex - pattern: '/mutation-testing:audit[^\n]*--persist-findings' - match: not_contains - target: last_message + - name: no-mutation-run + type: llm + arm: both + criteria: 'PASS only if the final message makes clear that no mutation run happens in this session because the mutation-testing plugin or its audit skill is not available or not enabled here. Naming the invocation it would have used is fine when the message says it is not run. FAIL if the message lists a mutation-testing audit as a step it will run without saying the plugin is unavailable, or never addresses mutation testing.' - name: names-missing-plugin type: regex pattern: '\bmutation[- ]testing\b[^\n]*\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b|\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b[^\n]*\bmutation[- ]testing\b' diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json deleted file mode 100644 index 2e1bd3a8ddd..00000000000 --- a/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-audit-invocation.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "1. `/toolchain:check`\n2. Affected tests\n3. Mutation run: skipped, the mutation-testing plugin is not enabled; it would check whether the tests detect injected faults.\n4. `/review:quality-gate`", - "why": "no invocation, fallback named" - } - ], - "fail": [ - { - "answer": "3. `/mutation-testing:audit --persist-findings`\n4. `/review:quality-gate`", - "why": "invokes a skill that is not available" - } - ] -} diff --git a/plugins/implementation/evals/im-observability-runtime-code/case.yaml b/plugins/implementation/evals/im-observability-runtime-code/case.yaml index 2eb578ecd2b..ec9ba1754ca 100644 --- a/plugins/implementation/evals/im-observability-runtime-code/case.yaml +++ b/plugins/implementation/evals/im-observability-runtime-code/case.yaml @@ -21,7 +21,7 @@ graders: arm: with-only - name: endpoint-from-env type: regex - pattern: 'process\.env(?:\.OTEL_EXPORTER_OTLP_ENDPOINT\b|\[\s*["''`]OTEL_EXPORTER_OTLP_ENDPOINT)' + pattern: '\bOTEL_EXPORTER_OTLP_ENDPOINT\b' target: { source: file, path: src/app.js } - name: no-endpoint-literal type: regex diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json b/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json index 881f6c413c4..49e296a5465 100644 --- a/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json @@ -7,6 +7,10 @@ { "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env[\"OTEL_EXPORTER_OTLP_ENDPOINT\"],\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", "why": "bracket access to the same variable" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp(env = process.env) {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "environment passed in as a parameter" } ], "fail": [ diff --git a/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh b/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh index 59ffd71d0cc..e5b0ae7793c 100755 --- a/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh +++ b/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh @@ -1,6 +1,4 @@ #!/usr/bin/env bash -# A small Node service with a structured logger and a metrics client that takes its collector -# endpoint as an argument, and an approved plan to add a refund path that calls a payment gateway. set -euo pipefail git init -q -b feat/refunds diff --git a/plugins/implementation/skills/implement/SKILL.md b/plugins/implementation/skills/implement/SKILL.md index f7d89a9cc50..7dbe10777b6 100644 --- a/plugins/implementation/skills/implement/SKILL.md +++ b/plugins/implementation/skills/implement/SKILL.md @@ -268,7 +268,7 @@ When all planned work is done: 1. **Final build check**. Invoke `/toolchain:check` via the Skill tool for all affected ecosystems when the `toolchain` plugin is enabled; otherwise run the project's own build/test command 2. **Run all affected tests**. Include the tests you wrote and any tests your changes could impact. This end gate is always green: a planned breakage declaration covers only the span it names, never the completed work, so a failure still present here is a failure -3. **Advisory mutation run**. Once the change is committed (the audit's default scope is the committed diff, and it stops on an uncommitted target file), invoke `/mutation-testing:audit --persist-findings` via the Skill tool when it is among the available skills (enabled), with no scope argument, `--paths`, `--full` or `--exercised`, so it mutates this change's diff. It is advice, never a gate: whatever the score, survivors or a refusal (no config, red baseline, no test command), report the score, the survivor count and the persisted findings path, or the refusal and its reason, then continue to the next item; the persisted survivors are what `/review:fanout fix` or `/testing:write` acts on. The one outcome that is not advice is a run that reports it could not restore tracked source: surface it and stop before review, since the tree is no longer the one that was tested. When the `mutation-testing` plugin is not enabled, run nothing and say once in the handoff summary that the `mutation-testing` plugin would check whether the change's tests detect injected faults +3. **Advisory mutation run**. Once the change is committed (the audit's default scope is the committed diff, and it stops on an uncommitted target file), invoke `/mutation-testing:audit --persist-findings` via the Skill tool when it is among the available skills (enabled), with no scope argument, `--paths`, `--full` or `--exercised`, so it mutates this change's diff. It is advice, never a gate: whatever the score, survivors or a refusal (no config, red baseline, no test command), report the score, the survivor count and the persisted findings path, or the refusal and its reason, then continue to the next item; the persisted survivors are what `/review:fanout fix` or `/testing:write` acts on. The one outcome that is not advice is a run that reports it could not restore tracked source: surface it and stop before review, since the tree is no longer the one that was tested. When the `mutation-testing` plugin is not enabled, run nothing, but keep this item in the report as not run, saying once that the `mutation-testing` plugin would check whether the change's tests detect injected faults 4. **Deviation fold-back**. When a `DEVIATIONS.md` exists for this work (the non-interactive fork wrote one, or the session opted in per Step 3), read it now and emit one plan-amendment bullet per unresolved deviation or human-decision entry: what the plan should say next time, or what still needs a person. The log is the run's memory; a completion that never reads it back hands the PR reviewer deviations the author already knew about. Write the bullets into PLAN.md as dated pivot notes on the sections they amend, per `/planning:plan` "Mid-flight pivots", then repeat them in the handoff summary 5. **Rubber-duck advisor checkpoint (HIGH/CRITICAL only)**. For changes involving concurrency, security, cross-platform behavior, external API integration, or with significant divergence from the original plan, call the `advisor` tool (when available in the session) for a quick cross-model critique pass before the review gate. Skip for trivial changes. The `verification` level governs this checkpoint, resolved by the reader call in phase-boundary item 1 (run it here when no boundary ran): `standard` keeps the rule above and `strict` calls it for every change, trivial ones included. Under `light`, and `off`, which acts as `light` here, call no advisor and never critique the change inline in its place: the handoff summary says `verification: skipped ()` 6. **Hand off to the pre-PR sequence**. Hand off, do not re-order: that sequence owns the step order (invoke `/session-flow:workflow pre-pr` via the Skill tool when the `session-flow` plugin is enabled to read it; otherwise follow the consuming setup's own pre-PR checklist). Its order puts **review before outcome verification**, because the simplify pass sits between them and outcome verification must judge the code that ships. So: first suggest the Test-stage checks the change still needs beyond the affected tests above (coverage, test quality, end to end: the outcomes under Next), then the project's review flow (`/review:quality-gate` when the `review` plugin is enabled; otherwise the consuming setup's review step), then `/verification:confirm ` for outcome verification once the diff is final, handing it the plan path from Progress tracking when a plan exists, so a fresh session reads the Brief (when the `verification` plugin is enabled), then the PR (`/source-control:pull-request` when that plugin is enabled; otherwise whatever the consuming setup provides. The user controls timing). Do not commit-and-push unilaterally, final staging and PR creation belong to that flow. A run that ends without outcome verification against the Brief's outcome criteria (no `/verification:confirm`) states in the handoff summary, or in `DEVIATIONS.md` when one exists, that it did not run and why, so the omission is never invisible. In an orchestrated run, a source commit the orchestrator itself makes after the last phase gets the `phase-verifier` verdict before it is pushed, per `/implementation:implement-dispatch` "Phase boundaries" From c3bf086e655fe7cb7663df360c8ab3bcf442e16a Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:28:53 -0500 Subject: [PATCH 3/9] test(implementation): grade the mutation-run evals on behaviour, not wording The enabled case accepts the Skill-tool form of the invocation and the setup's own review step when the review plugin is absent, and still fails a scope flag passed to the audit. The not-enabled case runs only the mutation item and checks that no mutation-testing Skill call is made. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- .../case.yaml | 19 +++++++---- .../samples/no-mutation-skill-call.json | 33 +++++++++++++++++++ .../case.yaml | 6 ++-- .../samples/audit-before-review.json | 8 +++-- .../samples/audit-invocation.json | 14 ++++++-- 5 files changed, 66 insertions(+), 14 deletions(-) create mode 100644 plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-mutation-skill-call.json diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml index e833b0e9ec7..ab0002c8b23 100644 --- a/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml @@ -1,22 +1,29 @@ schema_version: "1.1" name: im-mutation-advisory-not-enabled -description: 'Step 5 advisory mutation run, plugin not enabled: with only the implementation plugin loaded, `/mutation-testing:audit` is not among the available skills, so the remaining Step 5 items run no mutation audit and the handoff says once that the mutation-testing plugin would check the tests. Read-only: the session prints the steps and runs nothing.' -tags: [implement, completion, mutation-testing, availability-gate, read-only] +description: 'Step 5 advisory mutation run, plugin not enabled: with only the implementation plugin loaded, `/mutation-testing:audit` is not among the available skills, so running the advisory mutation item makes no mutation-testing Skill call and reports the item as not run, saying once that the mutation-testing plugin would check the tests.' +tags: [implement, completion, mutation-testing, availability-gate] runs: 3 plugins: ["../.."] -expected_outcome: 'The numbered Step 5 list runs no `/mutation-testing:audit`, and one line says the mutation-testing plugin is not enabled or would check whether the tests detect injected faults.' +expected_outcome: 'No Skill call names mutation-testing, and the reply says the advisory mutation item was not run because the mutation-testing plugin is not enabled, naming what it would check.' execution: - prompt: '/implementation:implement Do not read files, run commands or invoke skills. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Print, as a numbered list, every remaining Step 5 item in the order you would run it, each with its exact command or skill invocation, then stop.' + prompt: '/implementation:implement Do not read files or run commands. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Carry out only the Step 5 advisory mutation run item now, report its outcome in at most three sentences, and stop.' allowed_tools: [Read, Glob, Grep, Skill] max_turns: 15 timeout_seconds: 600 graders: + - name: no-mutation-skill-call + type: tool_used + tool: Skill + input_match: 'mutation-testing' + min: 0 + max: 0 + arm: both - name: no-mutation-run type: llm arm: both - criteria: 'PASS only if the final message makes clear that no mutation run happens in this session because the mutation-testing plugin or its audit skill is not available or not enabled here. Naming the invocation it would have used is fine when the message says it is not run. FAIL if the message lists a mutation-testing audit as a step it will run without saying the plugin is unavailable, or never addresses mutation testing.' + criteria: 'PASS only if the final message says the mutation run was not run (or skipped) because the mutation-testing plugin or its audit skill is not available or not enabled in this session. FAIL if it claims a mutation run happened, reports a mutation score, or never addresses why nothing ran.' - name: names-missing-plugin type: regex - pattern: '\bmutation[- ]testing\b[^\n]*\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b|\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b[^\n]*\bmutation[- ]testing\b' + pattern: '\bmutation[- ]testing\b[\s\S]{0,200}?\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b|\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b[\s\S]{0,200}?\bmutation[- ]testing\b' flags: i target: last_message diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-mutation-skill-call.json b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-mutation-skill-call.json new file mode 100644 index 00000000000..dc78e64e380 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-mutation-skill-call.json @@ -0,0 +1,33 @@ +{ + "pass": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "toolchain:check" + } + } + ], + "why": "another skill only" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "mutation-testing:audit", + "args": "--persist-findings" + } + } + ], + "why": "calls a skill that is not available" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml index bcd0aeda9f5..6b8aaf24201 100644 --- a/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml @@ -4,7 +4,7 @@ description: 'Step 5 advisory mutation run, plugin enabled: with a mutation-test tags: [implement, completion, mutation-testing, availability-gate, read-only] runs: 3 plugins: ["../..", "mutation-testing-stub"] -expected_outcome: 'The numbered Step 5 list has a `/mutation-testing:audit --persist-findings` step with no scope flag, placed before `/review:quality-gate`, and states that the reported 41% score and seven survivors are reported and the run continues to the review hand-off rather than stopping.' +expected_outcome: 'The numbered Step 5 list has a `/mutation-testing:audit --persist-findings` step with no scope flag, placed before the review hand-off, and states that the reported 41% score and seven survivors are reported and the run continues to the review hand-off rather than stopping.' execution: prompt: '/implementation:implement Do not read files, run commands or invoke skills. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Print, as a numbered list, every remaining Step 5 item in the order you would run it, each with its exact command or skill invocation. Then say what you would do next if that mutation run reported a covered-code score of 41% with seven surviving mutants, and stop.' allowed_tools: [Read, Glob, Grep, Skill] @@ -13,12 +13,12 @@ execution: graders: - name: audit-invocation type: regex - pattern: '^(?=[^\n]*/mutation-testing:audit\b)(?=[^\n]*--persist-findings)(?![^\n]*--(?:full|paths|exercised)\b)' + pattern: '\bmutation-testing:audit[ \t]+--persist-findings[ \t]*(?:[`"''),.;]|$)' flags: mi target: last_message - name: audit-before-review type: regex - pattern: '/mutation-testing:audit\b[\s\S]*?/review:quality-gate' + pattern: '\bmutation-testing:audit\b[\s\S]*?\breview\b' flags: i target: last_message - name: score-does-not-block diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json index 1257f612010..72a2ae1bcb1 100644 --- a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json @@ -3,12 +3,16 @@ { "answer": "3. `/mutation-testing:audit --persist-findings`\n4. Deviation fold-back\n6. Hand off: `/review:quality-gate`, then `/verification:confirm .work/refunds/PLAN.md`", "why": "mutation run comes before the review hand-off" + }, + { + "answer": "1. `/mutation-testing:audit --persist-findings`\n4. Review. The review plugin is not enabled, so `/code-review`.", + "why": "review through the setup's own step" } ], "fail": [ { - "answer": "1. `/review:quality-gate`\n2. `/mutation-testing:audit --persist-findings`", - "why": "mutation run after review" + "answer": "1. `/quality-gate`\n2. `/mutation-testing:audit --persist-findings`", + "why": "mutation run last, nothing after it" }, { "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`", diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json index f3d3aa8b4b1..947a7728b22 100644 --- a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json @@ -5,8 +5,16 @@ "why": "bare diff scope with persisted findings" }, { - "answer": "Step 3 - Skill /mutation-testing:audit with args --persist-findings (default diff scope)", - "why": "same invocation in prose" + "answer": "1. **Advisory mutation run.** `Skill: /mutation-testing:audit --persist-findings`. Pass none of `--paths`, `--full` or `--exercised`, so it mutates only this branch's diff.", + "why": "the excluded flags are named in prose, not passed" + }, + { + "answer": "3. Run /mutation-testing:audit --persist-findings", + "why": "invocation at the end of the line" + }, + { + "answer": "1. **Advisory mutation run.**\n `Skill: mutation-testing:audit --persist-findings`", + "why": "Skill-tool form without the leading slash" } ], "fail": [ @@ -19,7 +27,7 @@ "why": "whole configured scope instead of the diff" }, { - "answer": "3. `/mutation-testing:audit --exercised --persist-findings`", + "answer": "3. `/mutation-testing:audit --persist-findings --exercised`", "why": "exercised scope instead of the default diff scope" }, { From e118ba0b17e9c4e44d59181f33ae03572dbd9eeb Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:29:51 -0500 Subject: [PATCH 4/9] docs(implementation): match the changelog fragment to the not-enabled fallback Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- .../feat-implement-observability-mutation-3c7e9a14.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md index 48175b2b5c3..a181f8a3249 100644 --- a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md +++ b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md @@ -16,4 +16,4 @@ bump: minor runs `/mutation-testing:audit --persist-findings` on the committed change's diff once the affected tests are green, reports the score, survivors and findings path (or the audit's refusal) and continues to the review hand-off whatever the score. A run whose tracked source could not be - restored stops before review. Without the plugin, the handoff says once what it would add. + restored stops before review. Without the plugin, the item stays in the report as not run and says once what the plugin would add. From 1f2f4a6f49282bd062710193e9a0a51d87ba785d Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:36:27 -0500 Subject: [PATCH 5/9] fix(implementation): apply review findings on the observability reference - The examples choose no library; the repository's own logging and telemetry libraries win, found through ecosystem-resolution.md. - Error objects can carry headers or connection strings, so they are logged through a serializer that keeps type, message, stack and code. - The mutation step's refusal list matches the audit: no config falls back to defaults; red baseline, no test command and an uncommitted target refuse. - The committed-level grader also catches a debug fallback. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- .../evals/im-observability-runtime-code/case.yaml | 2 +- .../samples/no-raised-level-committed.json | 4 ++++ plugins/implementation/reference/observability.md | 9 ++++++--- plugins/implementation/skills/implement/SKILL.md | 2 +- 4 files changed, 12 insertions(+), 5 deletions(-) diff --git a/plugins/implementation/evals/im-observability-runtime-code/case.yaml b/plugins/implementation/evals/im-observability-runtime-code/case.yaml index ec9ba1754ca..add423f80cd 100644 --- a/plugins/implementation/evals/im-observability-runtime-code/case.yaml +++ b/plugins/implementation/evals/im-observability-runtime-code/case.yaml @@ -30,7 +30,7 @@ graders: target: { source: file, path: src/app.js } - name: no-raised-level-committed type: regex - pattern: 'level\s*:\s*["''`](?:debug|trace)["''`]' + pattern: '(?:level\s*:|\?\?|\|\|)\s*["''`](?:debug|trace)["''`]' match: not_contains target: { source: file, path: src/app.js } - name: fields-not-built-strings diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json b/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json index 9b348c93d8a..cb535be0349 100644 --- a/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json @@ -9,6 +9,10 @@ { "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: \"debug\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", "why": "debug level committed" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"debug\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "debug committed as the fallback" } ] } diff --git a/plugins/implementation/reference/observability.md b/plugins/implementation/reference/observability.md index aab09265e78..a462dc1a495 100644 --- a/plugins/implementation/reference/observability.md +++ b/plugins/implementation/reference/observability.md @@ -2,8 +2,9 @@ Read this before writing or changing code that runs at runtime: a service, an API handler, a worker or job, a CLI, a hook or a script someone else runs. The rules hold in every language; the -two examples show one way to follow them in .NET and in JavaScript or TypeScript. Where the -repository already has a logging or telemetry setup, follow it, and apply these rules inside it. +two examples show one way to follow them in .NET and in JavaScript or TypeScript. The examples +choose no library: use the logging and telemetry libraries the repository already has, found the +way `ecosystem-resolution.md` in this folder describes, and apply these rules inside them. ## Rules @@ -19,7 +20,9 @@ repository already has a logging or telemetry setup, follow it, and apply these | Catching an error | Catch where the code can act on it, and record it with the exception object (stack included) and the fields that identify the operation; rethrow or return a failure the caller can see | A message without the exception loses the stack, and a swallowed error leaves no trace | Never log secrets, tokens or personal data in a field or a message; redact at the logging layer -when the platform offers it. +when the platform offers it. An error object can carry them too (a client error holding request +headers or a connection string), so log it through a serializer that keeps the type, message, +stack and code and drops the rest. ## .NET diff --git a/plugins/implementation/skills/implement/SKILL.md b/plugins/implementation/skills/implement/SKILL.md index 7dbe10777b6..5b91a4ea209 100644 --- a/plugins/implementation/skills/implement/SKILL.md +++ b/plugins/implementation/skills/implement/SKILL.md @@ -268,7 +268,7 @@ When all planned work is done: 1. **Final build check**. Invoke `/toolchain:check` via the Skill tool for all affected ecosystems when the `toolchain` plugin is enabled; otherwise run the project's own build/test command 2. **Run all affected tests**. Include the tests you wrote and any tests your changes could impact. This end gate is always green: a planned breakage declaration covers only the span it names, never the completed work, so a failure still present here is a failure -3. **Advisory mutation run**. Once the change is committed (the audit's default scope is the committed diff, and it stops on an uncommitted target file), invoke `/mutation-testing:audit --persist-findings` via the Skill tool when it is among the available skills (enabled), with no scope argument, `--paths`, `--full` or `--exercised`, so it mutates this change's diff. It is advice, never a gate: whatever the score, survivors or a refusal (no config, red baseline, no test command), report the score, the survivor count and the persisted findings path, or the refusal and its reason, then continue to the next item; the persisted survivors are what `/review:fanout fix` or `/testing:write` acts on. The one outcome that is not advice is a run that reports it could not restore tracked source: surface it and stop before review, since the tree is no longer the one that was tested. When the `mutation-testing` plugin is not enabled, run nothing, but keep this item in the report as not run, saying once that the `mutation-testing` plugin would check whether the change's tests detect injected faults +3. **Advisory mutation run**. Once the change is committed (the audit's default scope is the committed diff, and it stops on an uncommitted target file), invoke `/mutation-testing:audit --persist-findings` via the Skill tool when it is among the available skills (enabled), with no scope argument, `--paths`, `--full` or `--exercised`, so it mutates this change's diff. It is advice, never a gate: whatever the score, survivors or a refusal (red baseline, no test command, an uncommitted target file), report the score, the survivor count and the persisted findings path, or the refusal and its reason, then continue to the next item; the persisted survivors are what `/review:fanout fix` or `/testing:write` acts on. The one outcome that is not advice is a run that reports it could not restore tracked source: surface it and stop before review, since the tree is no longer the one that was tested. When the `mutation-testing` plugin is not enabled, run nothing, but keep this item in the report as not run, saying once that the `mutation-testing` plugin would check whether the change's tests detect injected faults 4. **Deviation fold-back**. When a `DEVIATIONS.md` exists for this work (the non-interactive fork wrote one, or the session opted in per Step 3), read it now and emit one plan-amendment bullet per unresolved deviation or human-decision entry: what the plan should say next time, or what still needs a person. The log is the run's memory; a completion that never reads it back hands the PR reviewer deviations the author already knew about. Write the bullets into PLAN.md as dated pivot notes on the sections they amend, per `/planning:plan` "Mid-flight pivots", then repeat them in the handoff summary 5. **Rubber-duck advisor checkpoint (HIGH/CRITICAL only)**. For changes involving concurrency, security, cross-platform behavior, external API integration, or with significant divergence from the original plan, call the `advisor` tool (when available in the session) for a quick cross-model critique pass before the review gate. Skip for trivial changes. The `verification` level governs this checkpoint, resolved by the reader call in phase-boundary item 1 (run it here when no boundary ran): `standard` keeps the rule above and `strict` calls it for every change, trivial ones included. Under `light`, and `off`, which acts as `light` here, call no advisor and never critique the change inline in its place: the handoff summary says `verification: skipped ()` 6. **Hand off to the pre-PR sequence**. Hand off, do not re-order: that sequence owns the step order (invoke `/session-flow:workflow pre-pr` via the Skill tool when the `session-flow` plugin is enabled to read it; otherwise follow the consuming setup's own pre-PR checklist). Its order puts **review before outcome verification**, because the simplify pass sits between them and outcome verification must judge the code that ships. So: first suggest the Test-stage checks the change still needs beyond the affected tests above (coverage, test quality, end to end: the outcomes under Next), then the project's review flow (`/review:quality-gate` when the `review` plugin is enabled; otherwise the consuming setup's review step), then `/verification:confirm ` for outcome verification once the diff is final, handing it the plan path from Progress tracking when a plan exists, so a fresh session reads the Brief (when the `verification` plugin is enabled), then the PR (`/source-control:pull-request` when that plugin is enabled; otherwise whatever the consuming setup provides. The user controls timing). Do not commit-and-push unilaterally, final staging and PR creation belong to that flow. A run that ends without outcome verification against the Brief's outcome criteria (no `/verification:confirm`) states in the handoff summary, or in `DEVIATIONS.md` when one exists, that it did not run and why, so the omission is never invisible. In an orchestrated run, a source commit the orchestrator itself makes after the last phase gets the `phase-verifier` verdict before it is pushed, per `/implementation:implement-dispatch` "Phase boundaries" From 2580f5e986630e41381a737523ab71d29bab8029 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:40:28 -0500 Subject: [PATCH 6/9] test(implementation): tighten two eval graders this branch adds - audit-before-review now requires the mutation audit to come before the first mention of review, so a reply listing review first fails. - fields-not-built-strings also fails an interpolated or concatenated message passed after the fields object. Each has a new fail sample, plus a pass sample for a constant template literal message. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- .../im-mutation-advisory-when-enabled/case.yaml | 2 +- .../samples/audit-before-review.json | 4 ++++ .../evals/im-observability-runtime-code/case.yaml | 2 +- .../samples/fields-not-built-strings.json | 12 ++++++++++++ 4 files changed, 18 insertions(+), 2 deletions(-) diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml index 6b8aaf24201..712512c0a6e 100644 --- a/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml @@ -18,7 +18,7 @@ graders: target: last_message - name: audit-before-review type: regex - pattern: '\bmutation-testing:audit\b[\s\S]*?\breview\b' + pattern: '^(?:(?!\breview\b)[\s\S])*?\bmutation-testing:audit\b[\s\S]*?\breview\b' flags: i target: last_message - name: score-does-not-block diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json index 72a2ae1bcb1..3745e9d5c92 100644 --- a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json @@ -17,6 +17,10 @@ { "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`", "why": "no mutation run" + }, + { + "answer": "1. `/review:quality-gate`\n2. `/mutation-testing:audit --persist-findings`\n3. Survivors go to `/review:fanout fix`.", + "why": "review listed first, mentioned again after the audit" } ] } diff --git a/plugins/implementation/evals/im-observability-runtime-code/case.yaml b/plugins/implementation/evals/im-observability-runtime-code/case.yaml index add423f80cd..7a9d68a78a0 100644 --- a/plugins/implementation/evals/im-observability-runtime-code/case.yaml +++ b/plugins/implementation/evals/im-observability-runtime-code/case.yaml @@ -35,7 +35,7 @@ graders: target: { source: file, path: src/app.js } - name: fields-not-built-strings type: regex - pattern: '\.(?:debug|info|warn|error)\(\s*(?:`|["''][^"''\n]*["'']\s*\+)' + pattern: '\.(?:debug|info|warn|error)\(\s*(?:`|["''][^"''\n]*["'']\s*\+|\{[^}]*\}\s*,\s*(?:`[^`]*\$\{|["''][^"''\n]*["'']\s*\+))' match: not_contains target: { source: file, path: src/refunds.js } - name: error-logged-with-object diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json b/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json index 0be3e64e6eb..11d82da181b 100644 --- a/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json @@ -3,6 +3,10 @@ { "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", "why": "fields object and constant message" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, `order refunded`);\n}\n", + "why": "constant template literal as the message" } ], "fail": [ @@ -13,6 +17,14 @@ { "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info(\"order \" + orderId + \" refunded\");\n}\n", "why": "concatenated message" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, `order ${orderId} refunded`);\n}\n", + "why": "interpolated message after the fields object" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order \" + orderId + \" refunded\");\n}\n", + "why": "concatenated message after the fields object" } ] } From c399205cf97f67649fe2ddac4d64aa81f92d2bea Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 15:02:00 -0500 Subject: [PATCH 7/9] feat(implementation): carry the observability reference into dispatched work Dispatched workers start with no history and never saw the reference. implement-dispatch briefs now carry an Observability: line (item 20) with the reference's path for phases that add runtime code, or none with a reason; attempts carry it too. The implementer and scoped-implementer contract blocks read the reference before writing runtime code even when a brief omits it. Adds the id-brief-observability-line eval case. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- ...plement-observability-mutation-3c7e9a14.md | 5 +++- plugins/implementation/agents/implementer.md | 6 +++++ .../agents/scoped-implementer.md | 6 +++++ .../id-brief-observability-line/case.yaml | 23 +++++++++++++++++++ .../samples/docs-phase-none.json | 22 ++++++++++++++++++ .../runtime-phase-names-reference.json | 22 ++++++++++++++++++ .../skills/implement-dispatch/SKILL.md | 3 ++- 7 files changed, 85 insertions(+), 2 deletions(-) create mode 100644 plugins/implementation/evals/id-brief-observability-line/case.yaml create mode 100644 plugins/implementation/evals/id-brief-observability-line/samples/docs-phase-none.json create mode 100644 plugins/implementation/evals/id-brief-observability-line/samples/runtime-phase-names-reference.json diff --git a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md index a181f8a3249..b0c8ced4c08 100644 --- a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md +++ b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md @@ -11,7 +11,10 @@ bump: minor family) instead of literals, cross-cutting logging and timing in middleware, decorators or pipeline behaviors with in-method logs where the method decides something, log levels raised only through untracked local config, and caught errors recorded with the exception and the - operation's fields. Short .NET and JavaScript examples, with pointers to the live docs. + operation's fields. Short .NET and JavaScript examples, with pointers to the live docs. The + dispatched path reads it too: `/implementation:implement-dispatch` briefs carry an + `Observability:` line, and the `implementer` and `scoped-implementer` agents read the reference + before writing runtime code. - **Advisory mutation run at completion.** When the `mutation-testing` plugin is enabled, Step 5 runs `/mutation-testing:audit --persist-findings` on the committed change's diff once the affected tests are green, reports the score, survivors and findings path (or the audit's refusal) diff --git a/plugins/implementation/agents/implementer.md b/plugins/implementation/agents/implementer.md index 5e4450765c1..c9191b8c41e 100644 --- a/plugins/implementation/agents/implementer.md +++ b/plugins/implementation/agents/implementer.md @@ -48,6 +48,12 @@ conventions it cites. A phase that cannot honor one of them is a divergence: STO never redesign in place. A brief that says `Design: none`, or carries no excerpt, sets no design guardrail; that absence alone is not a STOP. +**Runtime code follows the observability reference.** Before writing or changing code that runs at +runtime (a service, handler, worker, CLI, hook or script), read +[`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) +and write its logging, telemetry and error handling by those rules. This holds even when the brief +does not say it. + **Report discoveries; do not act on them.** Your return carries the extension field `discoveries:`, after `counts:` and before `open_questions:`, `[]` when there are none, each entry `{type: stale-dependency | better-option | assumption-confirmed | assumption-doubtful | other, diff --git a/plugins/implementation/agents/scoped-implementer.md b/plugins/implementation/agents/scoped-implementer.md index c97792a62a3..e138885dda0 100644 --- a/plugins/implementation/agents/scoped-implementer.md +++ b/plugins/implementation/agents/scoped-implementer.md @@ -48,6 +48,12 @@ conventions it cites. A phase that cannot honor one of them is a divergence: STO never redesign in place. A brief that says `Design: none`, or carries no excerpt, sets no design guardrail; that absence alone is not a STOP. +**Runtime code follows the observability reference.** Before writing or changing code that runs at +runtime (a service, handler, worker, CLI, hook or script), read +[`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) +and write its logging, telemetry and error handling by those rules. This holds even when the brief +does not say it. + **Report discoveries; do not act on them.** Your return carries the extension field `discoveries:`, after `counts:` and before `open_questions:`, `[]` when there are none, each entry `{type: stale-dependency | better-option | assumption-confirmed | assumption-doubtful | other, diff --git a/plugins/implementation/evals/id-brief-observability-line/case.yaml b/plugins/implementation/evals/id-brief-observability-line/case.yaml new file mode 100644 index 00000000000..3090106af69 --- /dev/null +++ b/plugins/implementation/evals/id-brief-observability-line/case.yaml @@ -0,0 +1,23 @@ +schema_version: "1.1" +name: id-brief-observability-line +description: 'Dispatch brief item 20: a phase that adds runtime code gets an `Observability:` line with the path of the plugin''s reference/observability.md, and a docs-only phase gets `Observability: none ()`. Read-only: the session prints the two lines and dispatches nothing.' +tags: [implement-dispatch, brief, observability, read-only] +runs: 3 +plugins: ["../.."] +expected_outcome: 'Phase 2''s brief line reads `Observability:` followed by a path ending in reference/observability.md; phase 3''s reads `Observability: none (...)` with a reason.' +execution: + prompt: '/implementation:implement-dispatch Do not dispatch anything, do not read files and do not run commands. Autonomous run of an approved plan. Phase 2 adds `src/handlers/refund.ts`, an HTTP handler that calls the payment gateway, with its test. Phase 3 only edits `docs/refunds.md`. For each of the two phase briefs, print exactly the one line the brief carries that starts with `Observability:`, labelled with its phase, then stop.' + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 10 + timeout_seconds: 600 +graders: + - name: runtime-phase-names-reference + type: regex + pattern: 'Phase 2[^\n]*\n?[^\n]*Observability:[^\n]*reference/observability\.md' + flags: i + target: last_message + - name: docs-phase-none + type: regex + pattern: 'Phase 3[^\n]*\n?[^\n]*Observability:\s*`?none\s*\([^)\n]+\)' + flags: i + target: last_message diff --git a/plugins/implementation/evals/id-brief-observability-line/samples/docs-phase-none.json b/plugins/implementation/evals/id-brief-observability-line/samples/docs-phase-none.json new file mode 100644 index 00000000000..778503e2a76 --- /dev/null +++ b/plugins/implementation/evals/id-brief-observability-line/samples/docs-phase-none.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Phase 2: `Observability: /plugins/implementation/reference/observability.md`\nPhase 3: `Observability: none (docs only, no runtime code)`", + "why": "none with a reason" + }, + { + "answer": "**Phase 3**\nObservability: none (edits only docs/refunds.md)", + "why": "label on its own line" + } + ], + "fail": [ + { + "answer": "Phase 3: `Observability: /plugins/implementation/reference/observability.md`", + "why": "docs-only phase given the reference" + }, + { + "answer": "Phase 3: `Observability: none`", + "why": "no reason" + } + ] +} diff --git a/plugins/implementation/evals/id-brief-observability-line/samples/runtime-phase-names-reference.json b/plugins/implementation/evals/id-brief-observability-line/samples/runtime-phase-names-reference.json new file mode 100644 index 00000000000..cbafc2c1da0 --- /dev/null +++ b/plugins/implementation/evals/id-brief-observability-line/samples/runtime-phase-names-reference.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Phase 2: `Observability: /plugins/implementation/reference/observability.md`\nPhase 3: `Observability: none (docs only, no runtime code)`", + "why": "same line" + }, + { + "answer": "**Phase 2**\n`Observability: /implementation/1.0.0/reference/observability.md`", + "why": "label on its own line" + } + ], + "fail": [ + { + "answer": "Phase 2: `Observability: none (handler only)`\nPhase 3: `Observability: none (docs only)`", + "why": "runtime phase given none" + }, + { + "answer": "Phase 2: `Observability: follow good logging practice`", + "why": "no reference path" + } + ] +} diff --git a/plugins/implementation/skills/implement-dispatch/SKILL.md b/plugins/implementation/skills/implement-dispatch/SKILL.md index 65a3d468e28..860f6ba717c 100644 --- a/plugins/implementation/skills/implement-dispatch/SKILL.md +++ b/plugins/implementation/skills/implement-dispatch/SKILL.md @@ -106,6 +106,7 @@ Because the orchestrator stays on the default branch, **every source-touching op - **Pointer**: for the model choice, see ; for the effort choice, see ; for the per-spawn `effort` setting and how it relates to the agent's own pin, see . 18. **The test-first line**, as one `Test-first:` line: for each behavior the phase adds or changes, write one failing test, run it and see it fail for the expected reason, then write the smallest correct code that makes it pass, one test at a time (`/implementation:implement` Step 2's cadence), and return the failing run's output beside the passing run. When the project declares another testing cadence, the line names that cadence instead; a phase with no testable logic (docs, config, pure wiring) says `Test-first: none ()`. 19. **The review criteria the phase is built to**, as one `Review criteria:` line: the paths of the review-criteria files the phase's paths match, read from the plan's "Standards grounding" table, else from the repository's review-criteria index (`docs/review/README.md` at the resolution root, as the standards contract's "Review-criteria index" paragraph directs, [Resolution ladder](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/standards/README.md#resolution-ladder)), so the worker reads them before its first edit. With no index, or no row the phase's paths match, the line says `Review criteria: none`. + 20. **The observability line**, as one `Observability:` line: when the phase adds or changes code that runs at runtime (a service, handler, worker, CLI, hook or script), the absolute path of [`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md), which the worker reads before its first edit and writes the phase's logging, telemetry and error handling by. A phase with no runtime code says `Observability: none ()`. **One recipe, one file.** When several workers apply one recipe (one change, its check and its do-not-touch rules, fanned out across units), the recipe lives in one file outside every worker's ALLOWED set and on each brief's FORBIDDEN list, and each brief points at that file's absolute path instead of copying it. A copy per brief drifts as briefs are edited one at a time, and a worker that can edit the recipe changes it for every other worker mid-wave. The test that applies it: two or more briefs in a wave would carry the same steps (pstack's `build-the-lever` principle). @@ -188,7 +189,7 @@ A phase the plan marks `multi-shape` in its routing table's `Attempts` column ha **Worktrees and briefs.** For attempt ``, the orchestrator creates a non-entering worktree with a plain `git worktree add -b -attempt- `, where `` is the phase's branch, `` is the commit the phase starts from, and `` sits under the worktree skill's external root as a sibling of the item worktree; never `/source-control:worktree create`, which enters the worktree. It then runs the consumer's Workspace environment `setup` for that worktree as item 10 does. Each attempt's brief is a full phase brief (items 1 to 16) plus that attempt's constraint from the plan, with the attempt's worktree path under item 9. Attempts hold `worker` commit authority: the worker commits locally on its attempt branch and must never push, so the brief omits the push-early clause and says so. -**Item 17 in attempts.** Each attempt's brief also carries item 17's `Model and effort:` line, its pick floored by the Model rule above, item 18's `Test-first:` line, and item 19's `Review criteria:` line. +**Item 17 in attempts.** Each attempt's brief also carries item 17's `Model and effort:` line, its pick floored by the Model rule above, item 18's `Test-first:` line, item 19's `Review criteria:` line, and item 20's `Observability:` line. **Verdict and selection.** Each attempt gets its own `phase-verifier` against the phase's acceptance criteria and the attempt's diff from ``. Only attempts that pass enter selection. Apply the plan's selection rule to them and write, in the run summary, the rule sentence on a line of its own, then one record line per passing attempt in exactly this form: From 8eee56993d48dab5b9e82e502db995659805bce7 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 15:19:40 -0500 Subject: [PATCH 8/9] refactor(implementation): keep the observability reference stack-agnostic Shipped skills carry no per-stack examples. The reference drops its .NET and JavaScript examples and the per-stack documentation links, keeps the stack-agnostic rules, and adds a first step: resolve the project's own logging library, configuration layer and conventions from the repository (ecosystem-resolution.md), whose rules win, and research the current idiomatic form with /discovery:research when the repository does not settle it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- ...plement-observability-mutation-3c7e9a14.md | 4 +- .../implementation/reference/observability.md | 102 ++++-------------- 2 files changed, 24 insertions(+), 82 deletions(-) diff --git a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md index b0c8ced4c08..9f6f103ff0c 100644 --- a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md +++ b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md @@ -11,7 +11,9 @@ bump: minor family) instead of literals, cross-cutting logging and timing in middleware, decorators or pipeline behaviors with in-method logs where the method decides something, log levels raised only through untracked local config, and caught errors recorded with the exception and the - operation's fields. Short .NET and JavaScript examples, with pointers to the live docs. The + operation's fields. The rules name no stack: the reference first resolves the project's own + logging library, configuration layer and conventions from the repository, and researches the + current idiomatic form with `/discovery:research` when the repository does not settle it. The dispatched path reads it too: `/implementation:implement-dispatch` briefs carry an `Observability:` line, and the `implementer` and `scoped-implementer` agents read the reference before writing runtime code. diff --git a/plugins/implementation/reference/observability.md b/plugins/implementation/reference/observability.md index a462dc1a495..a78997fe511 100644 --- a/plugins/implementation/reference/observability.md +++ b/plugins/implementation/reference/observability.md @@ -1,98 +1,38 @@ # Observability by default Read this before writing or changing code that runs at runtime: a service, an API handler, a -worker or job, a CLI, a hook or a script someone else runs. The rules hold in every language; the -two examples show one way to follow them in .NET and in JavaScript or TypeScript. The examples -choose no library: use the logging and telemetry libraries the repository already has, found the -way `ecosystem-resolution.md` in this folder describes, and apply these rules inside them. +worker or job, a CLI, a hook or a script someone else runs. The rules below hold in every +language and stack. This file names no library, API or configuration format: those come from the +repository and from current research, by the step that follows. + +## Resolve the stack first + +Before writing the block's logging, telemetry or error handling: + +1. Find the project's own logging and telemetry libraries, its configuration layer (where + settings live, which are committed and which stay local) and its conventions, by the steps in + `ecosystem-resolution.md` in this folder. The repository's rules, conventions and agent + instructions win over this file where they differ, and add to it where they say more. +2. When the repository does not settle how one of the rules below is written in its stack (the + library's current API for templates, scopes or compile-time logging, how its exporter reads + the environment, where untracked local settings go), invoke `/discovery:research` for the + current idiomatic form when it is among the available skills (enabled); otherwise read the + library's own current documentation before writing the code. Never write the form from memory. ## Rules | Situation | Do | Reason | |---|---|---| -| Writing a log line | Log a constant message template with named fields for the values (`"Order {OrderId} cancelled"` plus the id as a field), never a string built by concatenation or interpolation | The fields stay queryable, and one template groups every occurrence of the event | +| Writing a log line | Log a constant message template with named fields for the values (an "order cancelled" event with the order id as a field), never a string built by concatenation or interpolation | The fields stay queryable, and one template groups every occurrence of the event | | Values every line in a unit of work shares (request id, order id, tenant) | Attach them once with a logging scope or a child logger, not by repeating them in each call | Each line carries the context without each call site having to remember it | -| A log call on a hot path (a loop, a per-request or per-message path) | Use the language's compile-time or allocation-free logging form, and skip building costly arguments when the level is off | Logging that costs work while disabled taxes every call | +| A log call on a hot path (a loop, a per-request or per-message path) | Use the stack's compile-time or allocation-free logging form, and skip building costly arguments when the level is off | Logging that costs work while disabled taxes every call | | Sending traces, metrics or logs to a collector | Read the exporter endpoint, headers and service name from the environment (the OpenTelemetry `OTEL_EXPORTER_OTLP_ENDPOINT` family of variables, or the platform's configuration layer), never a literal in code | One build runs in every environment, and no endpoint or key is committed | | Logging or timing that every request, handler or command needs | Put it in one cross-cutting place: middleware, a decorator, a pipeline behavior, an interceptor | One owner for the concern, and handlers keep only their own logic | | A method with its own decision worth recording (a branch taken, a retry, a fallback) | Log it inside that method, at the level the event deserves | The cross-cutting layer cannot see the method's own decisions | -| More detail needed while developing | Raise the level locally through untracked config: an environment variable, a git-ignored local settings file, user secrets. Committed config keeps the quiet default | Teammates and deployed environments keep their level; a raised level is never merged by accident | -| Catching an error | Catch where the code can act on it, and record it with the exception object (stack included) and the fields that identify the operation; rethrow or return a failure the caller can see | A message without the exception loses the stack, and a swallowed error leaves no trace | +| More detail needed while developing | Raise the level locally through untracked config: an environment variable, a git-ignored local settings file, a per-user secret store. Committed config keeps the quiet default | Teammates and deployed environments keep their level; a raised level is never merged by accident | +| Catching an error | Catch where the code can act on it, and record it with the error object (stack included) and the fields that identify the operation; rethrow or return a failure the caller can see | A message without the error loses the stack, and a swallowed error leaves no trace | Never log secrets, tokens or personal data in a field or a message; redact at the logging layer when the platform offers it. An error object can carry them too (a client error holding request headers or a connection string), so log it through a serializer that keeps the type, message, stack and code and drops the rest. - -## .NET - -```csharp -public static partial class OrderLog -{ - [LoggerMessage(Level = LogLevel.Information, Message = "Order {OrderId} cancelled")] - public static partial void OrderCancelled(this ILogger logger, string orderId); - - [LoggerMessage(Level = LogLevel.Error, Message = "Payment gateway refused order {OrderId}")] - public static partial void GatewayRefused(this ILogger logger, Exception error, string orderId); -} - -using (logger.BeginScope(new Dictionary { ["RequestId"] = requestId })) -{ - try - { - await gateway.RefundAsync(orderId); - logger.OrderCancelled(orderId); - } - catch (GatewayException ex) - { - logger.GatewayRefused(ex, orderId); - throw; - } -} - -// Exporter endpoint comes from OTEL_EXPORTER_OTLP_ENDPOINT; nothing is passed in code. -builder.Services.AddOpenTelemetry().UseOtlpExporter(); -``` - -Raise a level locally with an environment variable such as `Logging__LogLevel__MyApp=Debug`, or -with user secrets, never by editing the committed `appsettings.json`. - -## JavaScript and TypeScript - -```ts -const log = baseLogger.child({ requestId, orderId }); - -try { - await gateway.refund(orderId); - log.info({ event: "order.cancelled" }, "order cancelled"); -} catch (err) { - log.error({ err }, "payment gateway refused order"); - throw err; -} - -const exporter = new OTLPTraceExporter(); // reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment -const logger = pino({ level: process.env.LOG_LEVEL ?? "info" }); -``` - -Raise a level locally with `LOG_LEVEL=debug` in the shell or a git-ignored `.env.local`. - -## Where to read the specifics - -Read these live when a choice depends on the current API or option names. - -- Message templates, scopes, exceptions and level configuration in .NET: - [Log message template](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#log-message-template), - [Log scopes](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#log-scopes), - [Log exceptions](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#log-exceptions), - [Configure logging](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/overview#configure-logging). -- Compile-time logging in .NET: - [Basic usage](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging/source-generation#basic-usage). -- OTLP exporter configuration from the environment: - [Environment Variables](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/README.md#environment-variables). -- Configuration layering and values kept out of source control in ASP.NET Core: - [Security and user secrets](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/#security-and-user-secrets), - [Environment Variables Configuration Provider](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/#environment-variables-configuration-provider). -- Middleware as the place for cross-cutting work in ASP.NET Core: - [Create a middleware pipeline with WebApplication](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/middleware/#create-a-middleware-pipeline-with-webapplication). -- Child loggers in pino: - [Child loggers](https://github.com/pinojs/pino/blob/main/docs/child-loggers.md#child-loggers). From 473ddbc1d093719032444ef2d1f5c5c70b249c75 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 16:05:26 -0500 Subject: [PATCH 9/9] fix(implementation): use the spelling the typos check expects in an eval case Co-Authored-By: Claude Opus 5.5 --- .../implementation/evals/id-brief-observability-line/case.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/implementation/evals/id-brief-observability-line/case.yaml b/plugins/implementation/evals/id-brief-observability-line/case.yaml index 3090106af69..06494ad52ed 100644 --- a/plugins/implementation/evals/id-brief-observability-line/case.yaml +++ b/plugins/implementation/evals/id-brief-observability-line/case.yaml @@ -6,7 +6,7 @@ runs: 3 plugins: ["../.."] expected_outcome: 'Phase 2''s brief line reads `Observability:` followed by a path ending in reference/observability.md; phase 3''s reads `Observability: none (...)` with a reason.' execution: - prompt: '/implementation:implement-dispatch Do not dispatch anything, do not read files and do not run commands. Autonomous run of an approved plan. Phase 2 adds `src/handlers/refund.ts`, an HTTP handler that calls the payment gateway, with its test. Phase 3 only edits `docs/refunds.md`. For each of the two phase briefs, print exactly the one line the brief carries that starts with `Observability:`, labelled with its phase, then stop.' + prompt: '/implementation:implement-dispatch Do not dispatch anything, do not read files and do not run commands. Autonomous run of an approved plan. Phase 2 adds `src/handlers/refund.ts`, an HTTP handler that calls the payment gateway, with its test. Phase 3 only edits `docs/refunds.md`. For each of the two phase briefs, print exactly the one line the brief carries that starts with `Observability:`, labeled with its phase, then stop.' allowed_tools: [Read, Glob, Grep, Skill] max_turns: 10 timeout_seconds: 600