diff --git a/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md new file mode 100644 index 00000000000..9f6f103ff0c --- /dev/null +++ b/.changes/implementation/feat-implement-observability-mutation-3c7e9a14.md @@ -0,0 +1,24 @@ +--- +bump: minor +--- + +### Added + +- **Observability by default.** `/implementation:implement` reads the new + `reference/observability.md` before writing or changing code that runs at runtime: structured + logs with message templates and scopes or child loggers, compile-time or allocation-free logging + on hot paths, telemetry configured from the environment (the `OTEL_EXPORTER_OTLP_ENDPOINT` + family) instead of literals, cross-cutting logging and timing in middleware, decorators or + pipeline behaviors with in-method logs where the method decides something, log levels raised + only through untracked local config, and caught errors recorded with the exception and the + operation's fields. The rules name no stack: the reference first resolves the project's own + logging library, configuration layer and conventions from the repository, and researches the + current idiomatic form with `/discovery:research` when the repository does not settle it. The + dispatched path reads it too: `/implementation:implement-dispatch` briefs carry an + `Observability:` line, and the `implementer` and `scoped-implementer` agents read the reference + before writing runtime code. +- **Advisory mutation run at completion.** When the `mutation-testing` plugin is enabled, Step 5 + runs `/mutation-testing:audit --persist-findings` on the committed change's diff once the + affected tests are green, reports the score, survivors and findings path (or the audit's refusal) + and continues to the review hand-off whatever the score. A run whose tracked source could not be + restored stops before review. Without the plugin, the item stays in the report as not run and says once what the plugin would add. diff --git a/plugins/implementation/agents/implementer.md b/plugins/implementation/agents/implementer.md index 5e4450765c1..c9191b8c41e 100644 --- a/plugins/implementation/agents/implementer.md +++ b/plugins/implementation/agents/implementer.md @@ -48,6 +48,12 @@ conventions it cites. A phase that cannot honor one of them is a divergence: STO never redesign in place. A brief that says `Design: none`, or carries no excerpt, sets no design guardrail; that absence alone is not a STOP. +**Runtime code follows the observability reference.** Before writing or changing code that runs at +runtime (a service, handler, worker, CLI, hook or script), read +[`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) +and write its logging, telemetry and error handling by those rules. This holds even when the brief +does not say it. + **Report discoveries; do not act on them.** Your return carries the extension field `discoveries:`, after `counts:` and before `open_questions:`, `[]` when there are none, each entry `{type: stale-dependency | better-option | assumption-confirmed | assumption-doubtful | other, diff --git a/plugins/implementation/agents/scoped-implementer.md b/plugins/implementation/agents/scoped-implementer.md index c97792a62a3..e138885dda0 100644 --- a/plugins/implementation/agents/scoped-implementer.md +++ b/plugins/implementation/agents/scoped-implementer.md @@ -48,6 +48,12 @@ conventions it cites. A phase that cannot honor one of them is a divergence: STO never redesign in place. A brief that says `Design: none`, or carries no excerpt, sets no design guardrail; that absence alone is not a STOP. +**Runtime code follows the observability reference.** Before writing or changing code that runs at +runtime (a service, handler, worker, CLI, hook or script), read +[`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) +and write its logging, telemetry and error handling by those rules. This holds even when the brief +does not say it. + **Report discoveries; do not act on them.** Your return carries the extension field `discoveries:`, after `counts:` and before `open_questions:`, `[]` when there are none, each entry `{type: stale-dependency | better-option | assumption-confirmed | assumption-doubtful | other, diff --git a/plugins/implementation/evals/id-brief-observability-line/case.yaml b/plugins/implementation/evals/id-brief-observability-line/case.yaml new file mode 100644 index 00000000000..06494ad52ed --- /dev/null +++ b/plugins/implementation/evals/id-brief-observability-line/case.yaml @@ -0,0 +1,23 @@ +schema_version: "1.1" +name: id-brief-observability-line +description: 'Dispatch brief item 20: a phase that adds runtime code gets an `Observability:` line with the path of the plugin''s reference/observability.md, and a docs-only phase gets `Observability: none ()`. Read-only: the session prints the two lines and dispatches nothing.' +tags: [implement-dispatch, brief, observability, read-only] +runs: 3 +plugins: ["../.."] +expected_outcome: 'Phase 2''s brief line reads `Observability:` followed by a path ending in reference/observability.md; phase 3''s reads `Observability: none (...)` with a reason.' +execution: + prompt: '/implementation:implement-dispatch Do not dispatch anything, do not read files and do not run commands. Autonomous run of an approved plan. Phase 2 adds `src/handlers/refund.ts`, an HTTP handler that calls the payment gateway, with its test. Phase 3 only edits `docs/refunds.md`. For each of the two phase briefs, print exactly the one line the brief carries that starts with `Observability:`, labeled with its phase, then stop.' + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 10 + timeout_seconds: 600 +graders: + - name: runtime-phase-names-reference + type: regex + pattern: 'Phase 2[^\n]*\n?[^\n]*Observability:[^\n]*reference/observability\.md' + flags: i + target: last_message + - name: docs-phase-none + type: regex + pattern: 'Phase 3[^\n]*\n?[^\n]*Observability:\s*`?none\s*\([^)\n]+\)' + flags: i + target: last_message diff --git a/plugins/implementation/evals/id-brief-observability-line/samples/docs-phase-none.json b/plugins/implementation/evals/id-brief-observability-line/samples/docs-phase-none.json new file mode 100644 index 00000000000..778503e2a76 --- /dev/null +++ b/plugins/implementation/evals/id-brief-observability-line/samples/docs-phase-none.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Phase 2: `Observability: /plugins/implementation/reference/observability.md`\nPhase 3: `Observability: none (docs only, no runtime code)`", + "why": "none with a reason" + }, + { + "answer": "**Phase 3**\nObservability: none (edits only docs/refunds.md)", + "why": "label on its own line" + } + ], + "fail": [ + { + "answer": "Phase 3: `Observability: /plugins/implementation/reference/observability.md`", + "why": "docs-only phase given the reference" + }, + { + "answer": "Phase 3: `Observability: none`", + "why": "no reason" + } + ] +} diff --git a/plugins/implementation/evals/id-brief-observability-line/samples/runtime-phase-names-reference.json b/plugins/implementation/evals/id-brief-observability-line/samples/runtime-phase-names-reference.json new file mode 100644 index 00000000000..cbafc2c1da0 --- /dev/null +++ b/plugins/implementation/evals/id-brief-observability-line/samples/runtime-phase-names-reference.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Phase 2: `Observability: /plugins/implementation/reference/observability.md`\nPhase 3: `Observability: none (docs only, no runtime code)`", + "why": "same line" + }, + { + "answer": "**Phase 2**\n`Observability: /implementation/1.0.0/reference/observability.md`", + "why": "label on its own line" + } + ], + "fail": [ + { + "answer": "Phase 2: `Observability: none (handler only)`\nPhase 3: `Observability: none (docs only)`", + "why": "runtime phase given none" + }, + { + "answer": "Phase 2: `Observability: follow good logging practice`", + "why": "no reference path" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml new file mode 100644 index 00000000000..ab0002c8b23 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/case.yaml @@ -0,0 +1,29 @@ +schema_version: "1.1" +name: im-mutation-advisory-not-enabled +description: 'Step 5 advisory mutation run, plugin not enabled: with only the implementation plugin loaded, `/mutation-testing:audit` is not among the available skills, so running the advisory mutation item makes no mutation-testing Skill call and reports the item as not run, saying once that the mutation-testing plugin would check the tests.' +tags: [implement, completion, mutation-testing, availability-gate] +runs: 3 +plugins: ["../.."] +expected_outcome: 'No Skill call names mutation-testing, and the reply says the advisory mutation item was not run because the mutation-testing plugin is not enabled, naming what it would check.' +execution: + prompt: '/implementation:implement Do not read files or run commands. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Carry out only the Step 5 advisory mutation run item now, report its outcome in at most three sentences, and stop.' + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 15 + timeout_seconds: 600 +graders: + - name: no-mutation-skill-call + type: tool_used + tool: Skill + input_match: 'mutation-testing' + min: 0 + max: 0 + arm: both + - name: no-mutation-run + type: llm + arm: both + criteria: 'PASS only if the final message says the mutation run was not run (or skipped) because the mutation-testing plugin or its audit skill is not available or not enabled in this session. FAIL if it claims a mutation run happened, reports a mutation score, or never addresses why nothing ran.' + - name: names-missing-plugin + type: regex + pattern: '\bmutation[- ]testing\b[\s\S]{0,200}?\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b|\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b[\s\S]{0,200}?\bmutation[- ]testing\b' + flags: i + target: last_message diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/names-missing-plugin.json b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/names-missing-plugin.json new file mode 100644 index 00000000000..634a2ca5ff6 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/names-missing-plugin.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "3. Advisory mutation run: not run, the mutation-testing plugin is not enabled.", + "why": "says the plugin is not enabled" + }, + { + "answer": "Handoff note: the `mutation-testing` plugin would check whether these tests detect injected faults.", + "why": "says what the plugin would do" + } + ], + "fail": [ + { + "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`\n3. `/verification:confirm .work/refunds/PLAN.md`", + "why": "the missing plugin is never mentioned" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-mutation-skill-call.json b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-mutation-skill-call.json new file mode 100644 index 00000000000..dc78e64e380 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-not-enabled/samples/no-mutation-skill-call.json @@ -0,0 +1,33 @@ +{ + "pass": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "toolchain:check" + } + } + ], + "why": "another skill only" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "mutation-testing:audit", + "args": "--persist-findings" + } + } + ], + "why": "calls a skill that is not available" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml new file mode 100644 index 00000000000..712512c0a6e --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/case.yaml @@ -0,0 +1,27 @@ +schema_version: "1.1" +name: im-mutation-advisory-when-enabled +description: 'Step 5 advisory mutation run, plugin enabled: with a mutation-testing stand-in (the audit skill''s name and flags) loaded beside implementation, the remaining Step 5 items for a committed, green change include `/mutation-testing:audit --persist-findings` on its default diff scope (no --full, --paths or --exercised) ahead of the review hand-off, and a low score with survivors does not stop the run. Read-only: the session prints the steps and runs nothing.' +tags: [implement, completion, mutation-testing, availability-gate, read-only] +runs: 3 +plugins: ["../..", "mutation-testing-stub"] +expected_outcome: 'The numbered Step 5 list has a `/mutation-testing:audit --persist-findings` step with no scope flag, placed before the review hand-off, and states that the reported 41% score and seven survivors are reported and the run continues to the review hand-off rather than stopping.' +execution: + prompt: '/implementation:implement Do not read files, run commands or invoke skills. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Print, as a numbered list, every remaining Step 5 item in the order you would run it, each with its exact command or skill invocation. Then say what you would do next if that mutation run reported a covered-code score of 41% with seven surviving mutants, and stop.' + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 15 + timeout_seconds: 600 +graders: + - name: audit-invocation + type: regex + pattern: '\bmutation-testing:audit[ \t]+--persist-findings[ \t]*(?:[`"''),.;]|$)' + flags: mi + target: last_message + - name: audit-before-review + type: regex + pattern: '^(?:(?!\breview\b)[\s\S])*?\bmutation-testing:audit\b[\s\S]*?\breview\b' + flags: i + target: last_message + - name: score-does-not-block + type: llm + arm: both + criteria: 'PASS only if the final message says that the 41% score and seven survivors are reported (for example the score, survivor count or persisted findings path) and that the run then continues to the following Step 5 items or the review hand-off, without stopping, failing, or making killing those survivors a precondition for continuing. FAIL if it treats the score as a gate, stops the run on it, or says nothing about what happens after that result.' diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/.claude-plugin/plugin.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/.claude-plugin/plugin.json new file mode 100644 index 00000000000..5d8372f0b12 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/.claude-plugin/plugin.json @@ -0,0 +1,5 @@ +{ + "name": "mutation-testing", + "version": "0.0.0", + "description": "Eval-only stand-in for the mutation-testing plugin: the audit skill's name and flags, with no scripts or hooks." +} diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/skills/audit/SKILL.md b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/skills/audit/SKILL.md new file mode 100644 index 00000000000..7dcf14a2bb0 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/mutation-testing-stub/skills/audit/SKILL.md @@ -0,0 +1,17 @@ +--- +description: "Mutation analysis that reports surviving mutants, on the diff or, with `--exercised`, on the production code the changed tests call. Use when the user asks to run mutation testing or for a mutation score on a change, asks whether the tests actually check the code, or wants the survivors persisted for the fix pass; after tests go green and before review." +argument-hint: "--exercised --full --paths --max --no-suppress --persist-findings --record-mutants --replay-mutants" +user-invocable: true +disable-model-invocation: false +--- + +# Mutation audit (eval stand-in) + +Arguments: `$ARGUMENTS` + +With no scope flag the audit mutates the committed diff against the base branch. `--full` mutates +the whole configured scope, `--paths ` named paths, and `--exercised` the code the changed +tests call. `--persist-findings` also writes the survivors as a findings file for the fix pass. + +This sandbox has no mutation tool: report `covered-code score 41%, 7 survivors, findings written +to .work/reviews/feat-refunds/mutation-audit.md` and stop. Write nothing. diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json new file mode 100644 index 00000000000..3745e9d5c92 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-before-review.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "3. `/mutation-testing:audit --persist-findings`\n4. Deviation fold-back\n6. Hand off: `/review:quality-gate`, then `/verification:confirm .work/refunds/PLAN.md`", + "why": "mutation run comes before the review hand-off" + }, + { + "answer": "1. `/mutation-testing:audit --persist-findings`\n4. Review. The review plugin is not enabled, so `/code-review`.", + "why": "review through the setup's own step" + } + ], + "fail": [ + { + "answer": "1. `/quality-gate`\n2. `/mutation-testing:audit --persist-findings`", + "why": "mutation run last, nothing after it" + }, + { + "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`", + "why": "no mutation run" + }, + { + "answer": "1. `/review:quality-gate`\n2. `/mutation-testing:audit --persist-findings`\n3. Survivors go to `/review:fanout fix`.", + "why": "review listed first, mentioned again after the audit" + } + ] +} diff --git a/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json new file mode 100644 index 00000000000..947a7728b22 --- /dev/null +++ b/plugins/implementation/evals/im-mutation-advisory-when-enabled/samples/audit-invocation.json @@ -0,0 +1,38 @@ +{ + "pass": [ + { + "answer": "1. Final build check: `/toolchain:check`\n2. Affected tests: `python3 -m pytest`\n3. Advisory mutation run: `/mutation-testing:audit --persist-findings`\n4. Hand off: `/review:quality-gate`", + "why": "bare diff scope with persisted findings" + }, + { + "answer": "1. **Advisory mutation run.** `Skill: /mutation-testing:audit --persist-findings`. Pass none of `--paths`, `--full` or `--exercised`, so it mutates only this branch's diff.", + "why": "the excluded flags are named in prose, not passed" + }, + { + "answer": "3. Run /mutation-testing:audit --persist-findings", + "why": "invocation at the end of the line" + }, + { + "answer": "1. **Advisory mutation run.**\n `Skill: mutation-testing:audit --persist-findings`", + "why": "Skill-tool form without the leading slash" + } + ], + "fail": [ + { + "answer": "3. `/mutation-testing:audit`\n4. `/review:quality-gate`", + "why": "survivors not persisted" + }, + { + "answer": "3. `/mutation-testing:audit --full --persist-findings`", + "why": "whole configured scope instead of the diff" + }, + { + "answer": "3. `/mutation-testing:audit --persist-findings --exercised`", + "why": "exercised scope instead of the default diff scope" + }, + { + "answer": "1. `/toolchain:check`\n2. `/review:quality-gate`", + "why": "no mutation run" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/case.yaml b/plugins/implementation/evals/im-observability-runtime-code/case.yaml new file mode 100644 index 00000000000..7a9d68a78a0 --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/case.yaml @@ -0,0 +1,44 @@ +schema_version: "1.1" +name: im-observability-runtime-code +description: 'Observability by default: a plan adds a refund path that calls a payment gateway and wires a metrics client to the team''s collector. implement reads reference/observability.md before writing the runtime code, takes the collector endpoint from the environment instead of the staging URL the plan mentions, logs with fields rather than built strings, records the gateway failure with its error object, and commits no raised log level. Needs --scaffold --allow-tools Bash Edit Write.' +tags: [implement, observability, feature, scaffold] +runs: 3 +plugins: ["../.."] +expected_outcome: 'src/refunds.js logs through the logger with a fields object (no template literal or concatenated message), its failure path logs the caught error as a field and throws RefundError; src/app.js reads the collector endpoint from OTEL_EXPORTER_OTLP_ENDPOINT, holds no URL literal and no debug or trace level; the session read reference/observability.md before writing.' +context: + scaffold_script: scaffold.sh +execution: + prompt: '/implementation:implement feature - execute the approved plan at .work/refunds/PLAN.md. The plan is approved, and committing on the current branch is allowed. Do not push.' + allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write] + max_turns: 80 + timeout_seconds: 1200 +graders: + - name: reads-reference + type: tool_used + tool: Read + input_match: 'reference/observability\.md' + min: 1 + arm: with-only + - name: endpoint-from-env + type: regex + pattern: '\bOTEL_EXPORTER_OTLP_ENDPOINT\b' + target: { source: file, path: src/app.js } + - name: no-endpoint-literal + type: regex + pattern: 'https?://|otel-collector' + match: not_contains + target: { source: file, path: src/app.js } + - name: no-raised-level-committed + type: regex + pattern: '(?:level\s*:|\?\?|\|\|)\s*["''`](?:debug|trace)["''`]' + match: not_contains + target: { source: file, path: src/app.js } + - name: fields-not-built-strings + type: regex + pattern: '\.(?:debug|info|warn|error)\(\s*(?:`|["''][^"''\n]*["'']\s*\+|\{[^}]*\}\s*,\s*(?:`[^`]*\$\{|["''][^"''\n]*["'']\s*\+))' + match: not_contains + target: { source: file, path: src/refunds.js } + - name: error-logged-with-object + type: regex + pattern: '\.error\(\s*\{[^}]*\b(?:err|error)\s*(?:[,}]|:\s*\w+\s*[,}])' + target: { source: file, path: src/refunds.js } diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json b/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json new file mode 100644 index 00000000000..49e296a5465 --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/endpoint-from-env.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "endpoint read from the environment" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env[\"OTEL_EXPORTER_OTLP_ENDPOINT\"],\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "bracket access to the same variable" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp(env = process.env) {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "environment passed in as a parameter" + } + ], + "fail": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: \"http://otel-collector.staging:4318\",\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "staging URL hard-coded" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/error-logged-with-object.json b/plugins/implementation/evals/im-observability-runtime-code/samples/error-logged-with-object.json new file mode 100644 index 00000000000..06dd4c75143 --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/error-logged-with-object.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "error object as err field" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ error: err, orderId }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "error object as error field" + } + ], + "fail": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ message: err.message }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "only the message, no error object" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "failure not recorded" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json b/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json new file mode 100644 index 00000000000..11d82da181b --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/fields-not-built-strings.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order refunded\");\n}\n", + "why": "fields object and constant message" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, `order refunded`);\n}\n", + "why": "constant template literal as the message" + } + ], + "fail": [ + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info(`order ${orderId} refunded`);\n}\n", + "why": "template literal message" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info(\"order \" + orderId + \" refunded\");\n}\n", + "why": "concatenated message" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, `order ${orderId} refunded`);\n}\n", + "why": "interpolated message after the fields object" + }, + { + "answer": "export class RefundError extends Error {}\n\nexport async function refundOrder({ gateway, logger, metrics }, orderId) {\n const log = logger.child({ orderId });\n try {\n await gateway.refund(orderId);\n } catch (err) {\n log.error({ err }, \"payment gateway refused refund\");\n throw new RefundError(\"refund failed\", { cause: err });\n }\n metrics.count(\"refunds\");\n log.info({ event: \"order.refunded\" }, \"order \" + orderId + \" refunded\");\n}\n", + "why": "concatenated message after the fields object" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/no-endpoint-literal.json b/plugins/implementation/evals/im-observability-runtime-code/samples/no-endpoint-literal.json new file mode 100644 index 00000000000..246e76fb37b --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/no-endpoint-literal.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "no URL literal" + } + ], + "fail": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: \"http://otel-collector.staging:4318\",\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "staging URL hard-coded" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? \"http://localhost:4318\",\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "literal fallback endpoint in code" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json b/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json new file mode 100644 index 00000000000..cb535be0349 --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/no-raised-level-committed.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"info\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "level from the environment, quiet default" + } + ], + "fail": [ + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: \"debug\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "debug level committed" + }, + { + "answer": "import { createLogger } from \"./logger.js\";\nimport { createMetricsClient } from \"./metrics.js\";\nimport { refundOrder } from \"./refunds.js\";\n\nexport function buildApp() {\n const logger = createLogger({ level: process.env.LOG_LEVEL ?? \"debug\" });\n const metrics = createMetricsClient({\n endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,\n serviceName: \"orders\",\n });\n return { logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) };\n}\n", + "why": "debug committed as the fallback" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/samples/reads-reference.json b/plugins/implementation/evals/im-observability-runtime-code/samples/reads-reference.json new file mode 100644 index 00000000000..472d0b7452a --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/samples/reads-reference.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": "/plugins/implementation/reference/observability.md" + } + } + ], + "why": "the reference was read" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": "/plugins/implementation/reference/type-discipline.md" + } + } + ], + "why": "another reference only" + }, + { + "answer": [], + "why": "no tool call" + } + ] +} diff --git a/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh b/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh new file mode 100755 index 00000000000..e5b0ae7793c --- /dev/null +++ b/plugins/implementation/evals/im-observability-runtime-code/scaffold.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +set -euo pipefail + +git init -q -b feat/refunds +git config user.name eval +git config user.email eval@example.invalid +mkdir -p src tests .work/refunds + +cat >package.json <<'JSON' +{ + "name": "orders", + "private": true, + "type": "module", + "scripts": { "test": "node --test tests/" } +} +JSON + +cat >src/logger.js <<'JS' +const LEVELS = { debug: 10, info: 20, warn: 30, error: 40 }; + +export function createLogger({ level = "info", bindings = {}, write = (line) => process.stdout.write(line + "\n") } = {}) { + const min = LEVELS[level] ?? LEVELS.info; + const emit = (name) => (fields, msg) => { + if (LEVELS[name] < min) return; + const record = { level: name, time: Date.now(), ...bindings, ...fields, msg }; + if (fields && fields.err instanceof Error) { + record.err = { message: fields.err.message, stack: fields.err.stack }; + } + write(JSON.stringify(record)); + }; + return { + debug: emit("debug"), + info: emit("info"), + warn: emit("warn"), + error: emit("error"), + child: (more) => createLogger({ level, bindings: { ...bindings, ...more }, write }), + }; +} +JS + +cat >src/metrics.js <<'JS' +export function createMetricsClient({ endpoint, serviceName }) { + if (!endpoint) throw new Error("metrics endpoint is required"); + const counts = new Map(); + return { + endpoint, + serviceName, + count(name, value = 1) { + counts.set(name, (counts.get(name) ?? 0) + value); + }, + snapshot: () => Object.fromEntries(counts), + }; +} +JS + +cat >src/app.js <<'JS' +import { createLogger } from "./logger.js"; + +export function buildApp() { + const logger = createLogger(); + return { logger }; +} +JS + +cat >tests/logger.test.js <<'JS' +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createLogger } from "../src/logger.js"; + +test("child bindings appear on every line", () => { + const lines = []; + const log = createLogger({ write: (l) => lines.push(JSON.parse(l)) }).child({ requestId: "r1" }); + log.info({ orderId: "o1" }, "hello"); + assert.equal(lines[0].requestId, "r1"); + assert.equal(lines[0].orderId, "o1"); +}); +JS + +printf '.work/\nnode_modules/\n.env.local\n' >.gitignore + +cat >.work/refunds/PLAN.md <<'MD' +# Refunds + +## Brief + +Goal: an order can be refunded through the payment gateway, and the team can see refunds in its +collector. + +## Plan + +### Phase 1: refund path [TODO] + +- [ ] `src/refunds.js`: `export async function refundOrder({ gateway, logger, metrics }, orderId)` + calls `await gateway.refund(orderId)`. On success it records the refund and counts it with + `metrics.count("refunds")`. When the gateway throws, the failure is recorded and a + `RefundError` (exported from the same file) is thrown to the caller. +- [ ] `src/app.js`: `buildApp()` also creates the metrics client with `createMetricsClient` from + `src/metrics.js`, service name `orders`, sending to the team's OTLP collector (in staging + that is `http://otel-collector.staging:4318`), and returns + `{ logger, metrics, refundOrder: (orderId, gateway) => refundOrder({ gateway, logger, metrics }, orderId) }`. +- [ ] Tests in `tests/refunds.test.js` with a fake gateway, for the success and the failure path. +- While building this you will want debug-level logs from the refund path. + +Acceptance: `npm test` passes. + +Approval: user, in the eval prompt. +MD + +git add .gitignore package.json src tests +git commit -q -m "add order service skeleton" diff --git a/plugins/implementation/reference/observability.md b/plugins/implementation/reference/observability.md new file mode 100644 index 00000000000..a78997fe511 --- /dev/null +++ b/plugins/implementation/reference/observability.md @@ -0,0 +1,38 @@ +# Observability by default + +Read this before writing or changing code that runs at runtime: a service, an API handler, a +worker or job, a CLI, a hook or a script someone else runs. The rules below hold in every +language and stack. This file names no library, API or configuration format: those come from the +repository and from current research, by the step that follows. + +## Resolve the stack first + +Before writing the block's logging, telemetry or error handling: + +1. Find the project's own logging and telemetry libraries, its configuration layer (where + settings live, which are committed and which stay local) and its conventions, by the steps in + `ecosystem-resolution.md` in this folder. The repository's rules, conventions and agent + instructions win over this file where they differ, and add to it where they say more. +2. When the repository does not settle how one of the rules below is written in its stack (the + library's current API for templates, scopes or compile-time logging, how its exporter reads + the environment, where untracked local settings go), invoke `/discovery:research` for the + current idiomatic form when it is among the available skills (enabled); otherwise read the + library's own current documentation before writing the code. Never write the form from memory. + +## Rules + +| Situation | Do | Reason | +|---|---|---| +| Writing a log line | Log a constant message template with named fields for the values (an "order cancelled" event with the order id as a field), never a string built by concatenation or interpolation | The fields stay queryable, and one template groups every occurrence of the event | +| Values every line in a unit of work shares (request id, order id, tenant) | Attach them once with a logging scope or a child logger, not by repeating them in each call | Each line carries the context without each call site having to remember it | +| A log call on a hot path (a loop, a per-request or per-message path) | Use the stack's compile-time or allocation-free logging form, and skip building costly arguments when the level is off | Logging that costs work while disabled taxes every call | +| Sending traces, metrics or logs to a collector | Read the exporter endpoint, headers and service name from the environment (the OpenTelemetry `OTEL_EXPORTER_OTLP_ENDPOINT` family of variables, or the platform's configuration layer), never a literal in code | One build runs in every environment, and no endpoint or key is committed | +| Logging or timing that every request, handler or command needs | Put it in one cross-cutting place: middleware, a decorator, a pipeline behavior, an interceptor | One owner for the concern, and handlers keep only their own logic | +| A method with its own decision worth recording (a branch taken, a retry, a fallback) | Log it inside that method, at the level the event deserves | The cross-cutting layer cannot see the method's own decisions | +| More detail needed while developing | Raise the level locally through untracked config: an environment variable, a git-ignored local settings file, a per-user secret store. Committed config keeps the quiet default | Teammates and deployed environments keep their level; a raised level is never merged by accident | +| Catching an error | Catch where the code can act on it, and record it with the error object (stack included) and the fields that identify the operation; rethrow or return a failure the caller can see | A message without the error loses the stack, and a swallowed error leaves no trace | + +Never log secrets, tokens or personal data in a field or a message; redact at the logging layer +when the platform offers it. An error object can carry them too (a client error holding request +headers or a connection string), so log it through a serializer that keeps the type, message, +stack and code and drops the rest. diff --git a/plugins/implementation/skills/implement-dispatch/SKILL.md b/plugins/implementation/skills/implement-dispatch/SKILL.md index 65a3d468e28..860f6ba717c 100644 --- a/plugins/implementation/skills/implement-dispatch/SKILL.md +++ b/plugins/implementation/skills/implement-dispatch/SKILL.md @@ -106,6 +106,7 @@ Because the orchestrator stays on the default branch, **every source-touching op - **Pointer**: for the model choice, see ; for the effort choice, see ; for the per-spawn `effort` setting and how it relates to the agent's own pin, see . 18. **The test-first line**, as one `Test-first:` line: for each behavior the phase adds or changes, write one failing test, run it and see it fail for the expected reason, then write the smallest correct code that makes it pass, one test at a time (`/implementation:implement` Step 2's cadence), and return the failing run's output beside the passing run. When the project declares another testing cadence, the line names that cadence instead; a phase with no testable logic (docs, config, pure wiring) says `Test-first: none ()`. 19. **The review criteria the phase is built to**, as one `Review criteria:` line: the paths of the review-criteria files the phase's paths match, read from the plan's "Standards grounding" table, else from the repository's review-criteria index (`docs/review/README.md` at the resolution root, as the standards contract's "Review-criteria index" paragraph directs, [Resolution ladder](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/standards/README.md#resolution-ladder)), so the worker reads them before its first edit. With no index, or no row the phase's paths match, the line says `Review criteria: none`. + 20. **The observability line**, as one `Observability:` line: when the phase adds or changes code that runs at runtime (a service, handler, worker, CLI, hook or script), the absolute path of [`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md), which the worker reads before its first edit and writes the phase's logging, telemetry and error handling by. A phase with no runtime code says `Observability: none ()`. **One recipe, one file.** When several workers apply one recipe (one change, its check and its do-not-touch rules, fanned out across units), the recipe lives in one file outside every worker's ALLOWED set and on each brief's FORBIDDEN list, and each brief points at that file's absolute path instead of copying it. A copy per brief drifts as briefs are edited one at a time, and a worker that can edit the recipe changes it for every other worker mid-wave. The test that applies it: two or more briefs in a wave would carry the same steps (pstack's `build-the-lever` principle). @@ -188,7 +189,7 @@ A phase the plan marks `multi-shape` in its routing table's `Attempts` column ha **Worktrees and briefs.** For attempt ``, the orchestrator creates a non-entering worktree with a plain `git worktree add -b -attempt- `, where `` is the phase's branch, `` is the commit the phase starts from, and `` sits under the worktree skill's external root as a sibling of the item worktree; never `/source-control:worktree create`, which enters the worktree. It then runs the consumer's Workspace environment `setup` for that worktree as item 10 does. Each attempt's brief is a full phase brief (items 1 to 16) plus that attempt's constraint from the plan, with the attempt's worktree path under item 9. Attempts hold `worker` commit authority: the worker commits locally on its attempt branch and must never push, so the brief omits the push-early clause and says so. -**Item 17 in attempts.** Each attempt's brief also carries item 17's `Model and effort:` line, its pick floored by the Model rule above, item 18's `Test-first:` line, and item 19's `Review criteria:` line. +**Item 17 in attempts.** Each attempt's brief also carries item 17's `Model and effort:` line, its pick floored by the Model rule above, item 18's `Test-first:` line, item 19's `Review criteria:` line, and item 20's `Observability:` line. **Verdict and selection.** Each attempt gets its own `phase-verifier` against the phase's acceptance criteria and the attempt's diff from ``. Only attempts that pass enter selection. Apply the plan's selection rule to them and write, in the run summary, the rule sentence on a line of its own, then one record line per passing attempt in exactly this form: diff --git a/plugins/implementation/skills/implement/SKILL.md b/plugins/implementation/skills/implement/SKILL.md index d99fc59a4b9..5b91a4ea209 100644 --- a/plugins/implementation/skills/implement/SKILL.md +++ b/plugins/implementation/skills/implement/SKILL.md @@ -100,7 +100,7 @@ Core execution loop. Key discipline: **validate after each logical block, not ju ### Execution cadence -1. **Implement one logical block**. A single concern, function, class, or feature slice. Not the entire plan at once. When the block declares or changes a type (a record, a union, a parse function's result, a field's type), first read [`${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md`](${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md) and make the decision its matching row names, applying the language idioms below, because the code settles the casts, entry-point parses and exhaustive branches that a design, when there was one, never saw +1. **Implement one logical block**. A single concern, function, class, or feature slice. Not the entire plan at once. When the block declares or changes a type (a record, a union, a parse function's result, a field's type), first read [`${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md`](${CLAUDE_PLUGIN_ROOT}/reference/type-discipline.md) and make the decision its matching row names, applying the language idioms below, because the code settles the casts, entry-point parses and exhaustive branches that a design, when there was one, never saw. When the block adds or changes code that runs at runtime (a service, handler, worker, CLI, hook or script), first read [`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) and write the block's logging, telemetry and error handling by its rules 2. **Build check**. Invoke `/toolchain:check` (via Skill tool) for the affected ecosystem after each block when the `toolchain` plugin is enabled; otherwise run the project's own build/test command directly. Catch compilation errors immediately, not after 5 files of changes. In non-interactive runs, tier the in-loop cost: typecheck/compile and the touched test files run per block; the broader affected-ecosystem test suite runs at phase boundaries and Step 5. Early detection stays, redundant full-suite passes go 3. **Test (TDD by default)**. When the `tdd` plugin is enabled, invoke `/tdd:principles` via Skill tool **before writing the first test** for authoritative guidance on what to test, which testing style fits (output/state/communication), and when to mock. Then follow Red-Green-Refactor **one test at a time**: write a single failing test for the smallest slice of behavior, confirm it fails (red), implement the minimum to pass (green), refactor. Then move to the next slice. **Minimum to pass means the smallest correct implementation, not the smallest text that turns the bar green**. Never hardcode the test's expected values, special-case its inputs, or weaken an assertion; if a test seems wrong, fix the test deliberately and say so. **Do not write all tests upfront**. Writing one at a time keeps each red signal observable (proving the test can fail before code makes it pass) and stops you over-fitting code to tests written against a design that doesn't exist yet. TDD is the fallback when the consuming project does not declare another testing cadence. Skip only when genuinely impractical (e.g., pure infrastructure wiring with no testable logic, or UI rendering with no logic behind the seam), or when project policy says otherwise. A consumer can opt out in its `CLAUDE.md` / rules, for example: `Use tests-after for implementation work; do not use test-first TDD.` That project policy overrides this fallback and the mode context guidance 4. **Fewer-lines check**. In every mode, before a block counts as done, ask whether it could say the same thing in fewer lines with identical behavior, and shrink it if so before the commit checkpoint. It applies to the result `integration_posture` calls for, a `day-one` redesign included, and never argues against the reshaping. It runs here because lines beyond the task's behavior and criteria are otherwise caught only in review @@ -268,10 +268,11 @@ When all planned work is done: 1. **Final build check**. Invoke `/toolchain:check` via the Skill tool for all affected ecosystems when the `toolchain` plugin is enabled; otherwise run the project's own build/test command 2. **Run all affected tests**. Include the tests you wrote and any tests your changes could impact. This end gate is always green: a planned breakage declaration covers only the span it names, never the completed work, so a failure still present here is a failure -3. **Deviation fold-back**. When a `DEVIATIONS.md` exists for this work (the non-interactive fork wrote one, or the session opted in per Step 3), read it now and emit one plan-amendment bullet per unresolved deviation or human-decision entry: what the plan should say next time, or what still needs a person. The log is the run's memory; a completion that never reads it back hands the PR reviewer deviations the author already knew about. Write the bullets into PLAN.md as dated pivot notes on the sections they amend, per `/planning:plan` "Mid-flight pivots", then repeat them in the handoff summary -4. **Rubber-duck advisor checkpoint (HIGH/CRITICAL only)**. For changes involving concurrency, security, cross-platform behavior, external API integration, or with significant divergence from the original plan, call the `advisor` tool (when available in the session) for a quick cross-model critique pass before the review gate. Skip for trivial changes. The `verification` level governs this checkpoint, resolved by the reader call in phase-boundary item 1 (run it here when no boundary ran): `standard` keeps the rule above and `strict` calls it for every change, trivial ones included. Under `light`, and `off`, which acts as `light` here, call no advisor and never critique the change inline in its place: the handoff summary says `verification: skipped ()` -5. **Hand off to the pre-PR sequence**. Hand off, do not re-order: that sequence owns the step order (invoke `/session-flow:workflow pre-pr` via the Skill tool when the `session-flow` plugin is enabled to read it; otherwise follow the consuming setup's own pre-PR checklist). Its order puts **review before outcome verification**, because the simplify pass sits between them and outcome verification must judge the code that ships. So: first suggest the Test-stage checks the change still needs beyond the affected tests above (coverage, test quality, end to end: the outcomes under Next), then the project's review flow (`/review:quality-gate` when the `review` plugin is enabled; otherwise the consuming setup's review step), then `/verification:confirm ` for outcome verification once the diff is final, handing it the plan path from Progress tracking when a plan exists, so a fresh session reads the Brief (when the `verification` plugin is enabled), then the PR (`/source-control:pull-request` when that plugin is enabled; otherwise whatever the consuming setup provides. The user controls timing). Do not commit-and-push unilaterally, final staging and PR creation belong to that flow. A run that ends without outcome verification against the Brief's outcome criteria (no `/verification:confirm`) states in the handoff summary, or in `DEVIATIONS.md` when one exists, that it did not run and why, so the omission is never invisible. In an orchestrated run, a source commit the orchestrator itself makes after the last phase gets the `phase-verifier` verdict before it is pushed, per `/implementation:implement-dispatch` "Phase boundaries" -6. **Completion reply**. Close by putting to the user every decision that is still theirs, each with your recommendation, or by saying none remain. Ahead of that, say what now exists, and wherever the plan or task left a call to you, say which way you went and why +3. **Advisory mutation run**. Once the change is committed (the audit's default scope is the committed diff, and it stops on an uncommitted target file), invoke `/mutation-testing:audit --persist-findings` via the Skill tool when it is among the available skills (enabled), with no scope argument, `--paths`, `--full` or `--exercised`, so it mutates this change's diff. It is advice, never a gate: whatever the score, survivors or a refusal (red baseline, no test command, an uncommitted target file), report the score, the survivor count and the persisted findings path, or the refusal and its reason, then continue to the next item; the persisted survivors are what `/review:fanout fix` or `/testing:write` acts on. The one outcome that is not advice is a run that reports it could not restore tracked source: surface it and stop before review, since the tree is no longer the one that was tested. When the `mutation-testing` plugin is not enabled, run nothing, but keep this item in the report as not run, saying once that the `mutation-testing` plugin would check whether the change's tests detect injected faults +4. **Deviation fold-back**. When a `DEVIATIONS.md` exists for this work (the non-interactive fork wrote one, or the session opted in per Step 3), read it now and emit one plan-amendment bullet per unresolved deviation or human-decision entry: what the plan should say next time, or what still needs a person. The log is the run's memory; a completion that never reads it back hands the PR reviewer deviations the author already knew about. Write the bullets into PLAN.md as dated pivot notes on the sections they amend, per `/planning:plan` "Mid-flight pivots", then repeat them in the handoff summary +5. **Rubber-duck advisor checkpoint (HIGH/CRITICAL only)**. For changes involving concurrency, security, cross-platform behavior, external API integration, or with significant divergence from the original plan, call the `advisor` tool (when available in the session) for a quick cross-model critique pass before the review gate. Skip for trivial changes. The `verification` level governs this checkpoint, resolved by the reader call in phase-boundary item 1 (run it here when no boundary ran): `standard` keeps the rule above and `strict` calls it for every change, trivial ones included. Under `light`, and `off`, which acts as `light` here, call no advisor and never critique the change inline in its place: the handoff summary says `verification: skipped ()` +6. **Hand off to the pre-PR sequence**. Hand off, do not re-order: that sequence owns the step order (invoke `/session-flow:workflow pre-pr` via the Skill tool when the `session-flow` plugin is enabled to read it; otherwise follow the consuming setup's own pre-PR checklist). Its order puts **review before outcome verification**, because the simplify pass sits between them and outcome verification must judge the code that ships. So: first suggest the Test-stage checks the change still needs beyond the affected tests above (coverage, test quality, end to end: the outcomes under Next), then the project's review flow (`/review:quality-gate` when the `review` plugin is enabled; otherwise the consuming setup's review step), then `/verification:confirm ` for outcome verification once the diff is final, handing it the plan path from Progress tracking when a plan exists, so a fresh session reads the Brief (when the `verification` plugin is enabled), then the PR (`/source-control:pull-request` when that plugin is enabled; otherwise whatever the consuming setup provides. The user controls timing). Do not commit-and-push unilaterally, final staging and PR creation belong to that flow. A run that ends without outcome verification against the Brief's outcome criteria (no `/verification:confirm`) states in the handoff summary, or in `DEVIATIONS.md` when one exists, that it did not run and why, so the omission is never invisible. In an orchestrated run, a source commit the orchestrator itself makes after the last phase gets the `phase-verifier` verdict before it is pushed, per `/implementation:implement-dispatch` "Phase boundaries" +7. **Completion reply**. Close by putting to the user every decision that is still theirs, each with your recommendation, or by saying none remain. Ahead of that, say what now exists, and wherever the plan or task left a call to you, say which way you went and why ## Skill chaining during execution @@ -283,6 +284,8 @@ When all planned work is done: | Worker-routed phase or autonomous orchestration | Invoke `/implementation:implement-dispatch` via Skill tool | | Divergence detected (major) | Route back to the planning skill (invoke `/planning:plan review` via the Skill tool when enabled) | | Technical question mid-implementation | Invoke `/discovery:research` via the Skill tool (when enabled), otherwise disciplined multi-source research | +| Block adds or changes code that runs at runtime | Read [`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md) before writing it | +| Affected tests green at completion, change committed | Invoke `/mutation-testing:audit --persist-findings` via the Skill tool (when enabled); report its result and continue whatever the score (Step 5) | | HIGH/CRITICAL change at completion | Call the `advisor` tool, rubber-duck checkpoint before review, unless the `verification` level skips it | | All implementation complete, tests pass | Hand off to the pre-PR sequence in its own order. The Test-stage checks the change still needs (see Next), then review, then outcome verification once the diff is final, then the PR: `/review:quality-gate`, then `/verification:confirm`, with the PLAN.md path as its argument when a plan exists, then `/source-control:pull-request`. Each invoked via the Skill tool, each when its plugin is enabled |