Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
bump: minor
---

### Added

- **Observability by default.** `/implementation:implement` reads the new
`reference/observability.md` before writing or changing code that runs at runtime: structured
logs with message templates and scopes or child loggers, compile-time or allocation-free logging
on hot paths, telemetry configured from the environment (the `OTEL_EXPORTER_OTLP_ENDPOINT`
family) instead of literals, cross-cutting logging and timing in middleware, decorators or
pipeline behaviors with in-method logs where the method decides something, log levels raised
only through untracked local config, and caught errors recorded with the exception and the
operation's fields. The rules name no stack: the reference first resolves the project's own
logging library, configuration layer and conventions from the repository, and researches the
current idiomatic form with `/discovery:research` when the repository does not settle it. The
dispatched path reads it too: `/implementation:implement-dispatch` briefs carry an
`Observability:` line, and the `implementer` and `scoped-implementer` agents read the reference
before writing runtime code.
- **Advisory mutation run at completion.** When the `mutation-testing` plugin is enabled, Step 5
runs `/mutation-testing:audit --persist-findings` on the committed change's diff once the
affected tests are green, reports the score, survivors and findings path (or the audit's refusal)
and continues to the review hand-off whatever the score. A run whose tracked source could not be
restored stops before review. Without the plugin, the item stays in the report as not run and says once what the plugin would add.
6 changes: 6 additions & 0 deletions plugins/implementation/agents/implementer.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ conventions it cites. A phase that cannot honor one of them is a divergence: STO
never redesign in place. A brief that says `Design: none`, or carries no excerpt, sets no design
guardrail; that absence alone is not a STOP.

**Runtime code follows the observability reference.** Before writing or changing code that runs at
runtime (a service, handler, worker, CLI, hook or script), read
[`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md)
and write its logging, telemetry and error handling by those rules. This holds even when the brief
does not say it.

**Report discoveries; do not act on them.** Your return carries the extension field
`discoveries:`, after `counts:` and before `open_questions:`, `[]` when there are none, each entry
`{type: stale-dependency | better-option | assumption-confirmed | assumption-doubtful | other,
Expand Down
6 changes: 6 additions & 0 deletions plugins/implementation/agents/scoped-implementer.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ conventions it cites. A phase that cannot honor one of them is a divergence: STO
never redesign in place. A brief that says `Design: none`, or carries no excerpt, sets no design
guardrail; that absence alone is not a STOP.

**Runtime code follows the observability reference.** Before writing or changing code that runs at
runtime (a service, handler, worker, CLI, hook or script), read
[`${CLAUDE_PLUGIN_ROOT}/reference/observability.md`](${CLAUDE_PLUGIN_ROOT}/reference/observability.md)
and write its logging, telemetry and error handling by those rules. This holds even when the brief
does not say it.

**Report discoveries; do not act on them.** Your return carries the extension field
`discoveries:`, after `counts:` and before `open_questions:`, `[]` when there are none, each entry
`{type: stale-dependency | better-option | assumption-confirmed | assumption-doubtful | other,
Expand Down
23 changes: 23 additions & 0 deletions plugins/implementation/evals/id-brief-observability-line/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
schema_version: "1.1"
name: id-brief-observability-line
description: 'Dispatch brief item 20: a phase that adds runtime code gets an `Observability:` line with the path of the plugin''s reference/observability.md, and a docs-only phase gets `Observability: none (<reason>)`. Read-only: the session prints the two lines and dispatches nothing.'
tags: [implement-dispatch, brief, observability, read-only]
runs: 3
plugins: ["../.."]
expected_outcome: 'Phase 2''s brief line reads `Observability:` followed by a path ending in reference/observability.md; phase 3''s reads `Observability: none (...)` with a reason.'
execution:
prompt: '/implementation:implement-dispatch Do not dispatch anything, do not read files and do not run commands. Autonomous run of an approved plan. Phase 2 adds `src/handlers/refund.ts`, an HTTP handler that calls the payment gateway, with its test. Phase 3 only edits `docs/refunds.md`. For each of the two phase briefs, print exactly the one line the brief carries that starts with `Observability:`, labeled with its phase, then stop.'
allowed_tools: [Read, Glob, Grep, Skill]
max_turns: 10
timeout_seconds: 600
graders:
- name: runtime-phase-names-reference
type: regex
pattern: 'Phase 2[^\n]*\n?[^\n]*Observability:[^\n]*reference/observability\.md'
flags: i
target: last_message
- name: docs-phase-none
type: regex
pattern: 'Phase 3[^\n]*\n?[^\n]*Observability:\s*`?none\s*\([^)\n]+\)'
flags: i
target: last_message
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"pass": [
{
"answer": "Phase 2: `Observability: /plugins/implementation/reference/observability.md`\nPhase 3: `Observability: none (docs only, no runtime code)`",
"why": "none with a reason"
},
{
"answer": "**Phase 3**\nObservability: none (edits only docs/refunds.md)",
"why": "label on its own line"
}
],
"fail": [
{
"answer": "Phase 3: `Observability: /plugins/implementation/reference/observability.md`",
"why": "docs-only phase given the reference"
},
{
"answer": "Phase 3: `Observability: none`",
"why": "no reason"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"pass": [
{
"answer": "Phase 2: `Observability: /plugins/implementation/reference/observability.md`\nPhase 3: `Observability: none (docs only, no runtime code)`",
"why": "same line"
},
{
"answer": "**Phase 2**\n`Observability: <plugin-cache>/implementation/1.0.0/reference/observability.md`",
"why": "label on its own line"
}
],
"fail": [
{
"answer": "Phase 2: `Observability: none (handler only)`\nPhase 3: `Observability: none (docs only)`",
"why": "runtime phase given none"
},
{
"answer": "Phase 2: `Observability: follow good logging practice`",
"why": "no reference path"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
schema_version: "1.1"
name: im-mutation-advisory-not-enabled
description: 'Step 5 advisory mutation run, plugin not enabled: with only the implementation plugin loaded, `/mutation-testing:audit` is not among the available skills, so running the advisory mutation item makes no mutation-testing Skill call and reports the item as not run, saying once that the mutation-testing plugin would check the tests.'
tags: [implement, completion, mutation-testing, availability-gate]
runs: 3
plugins: ["../.."]
expected_outcome: 'No Skill call names mutation-testing, and the reply says the advisory mutation item was not run because the mutation-testing plugin is not enabled, naming what it would check.'
execution:
prompt: '/implementation:implement Do not read files or run commands. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Carry out only the Step 5 advisory mutation run item now, report its outcome in at most three sentences, and stop.'
allowed_tools: [Read, Glob, Grep, Skill]
max_turns: 15
timeout_seconds: 600
graders:
- name: no-mutation-skill-call
type: tool_used
tool: Skill
input_match: 'mutation-testing'
min: 0
max: 0
arm: both
- name: no-mutation-run
type: llm
arm: both
criteria: 'PASS only if the final message says the mutation run was not run (or skipped) because the mutation-testing plugin or its audit skill is not available or not enabled in this session. FAIL if it claims a mutation run happened, reports a mutation score, or never addresses why nothing ran.'
- name: names-missing-plugin
type: regex
pattern: '\bmutation[- ]testing\b[\s\S]{0,200}?\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b|\b(?:would|not enabled|isn''t enabled|unavailable|not available)\b[\s\S]{0,200}?\bmutation[- ]testing\b'
flags: i
target: last_message
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "3. Advisory mutation run: not run, the mutation-testing plugin is not enabled.",
"why": "says the plugin is not enabled"
},
{
"answer": "Handoff note: the `mutation-testing` plugin would check whether these tests detect injected faults.",
"why": "says what the plugin would do"
}
],
"fail": [
{
"answer": "1. `/toolchain:check`\n2. `/review:quality-gate`\n3. `/verification:confirm .work/refunds/PLAN.md`",
"why": "the missing plugin is never mentioned"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"pass": [
{
"answer": [],
"why": "no tool call"
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "toolchain:check"
}
}
],
"why": "another skill only"
}
],
"fail": [
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "mutation-testing:audit",
"args": "--persist-findings"
}
}
],
"why": "calls a skill that is not available"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
schema_version: "1.1"
name: im-mutation-advisory-when-enabled
description: 'Step 5 advisory mutation run, plugin enabled: with a mutation-testing stand-in (the audit skill''s name and flags) loaded beside implementation, the remaining Step 5 items for a committed, green change include `/mutation-testing:audit --persist-findings` on its default diff scope (no --full, --paths or --exercised) ahead of the review hand-off, and a low score with survivors does not stop the run. Read-only: the session prints the steps and runs nothing.'
tags: [implement, completion, mutation-testing, availability-gate, read-only]
runs: 3
plugins: ["../..", "mutation-testing-stub"]
expected_outcome: 'The numbered Step 5 list has a `/mutation-testing:audit --persist-findings` step with no scope flag, placed before the review hand-off, and states that the reported 41% score and seven survivors are reported and the run continues to the review hand-off rather than stopping.'
execution:
prompt: '/implementation:implement Do not read files, run commands or invoke skills. The approved plan at .work/refunds/PLAN.md is fully built on branch `feat/refunds`: every phase is [DONE], every change is committed, and the final build and all affected tests are green. Print, as a numbered list, every remaining Step 5 item in the order you would run it, each with its exact command or skill invocation. Then say what you would do next if that mutation run reported a covered-code score of 41% with seven surviving mutants, and stop.'
allowed_tools: [Read, Glob, Grep, Skill]
max_turns: 15
timeout_seconds: 600
graders:
- name: audit-invocation
type: regex
pattern: '\bmutation-testing:audit[ \t]+--persist-findings[ \t]*(?:[`"''),.;]|$)'
flags: mi
target: last_message
- name: audit-before-review
type: regex
pattern: '^(?:(?!\breview\b)[\s\S])*?\bmutation-testing:audit\b[\s\S]*?\breview\b'
flags: i
target: last_message
- name: score-does-not-block
type: llm
arm: both
criteria: 'PASS only if the final message says that the 41% score and seven survivors are reported (for example the score, survivor count or persisted findings path) and that the run then continues to the following Step 5 items or the review hand-off, without stopping, failing, or making killing those survivors a precondition for continuing. FAIL if it treats the score as a gate, stops the run on it, or says nothing about what happens after that result.'
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"name": "mutation-testing",
"version": "0.0.0",
"description": "Eval-only stand-in for the mutation-testing plugin: the audit skill's name and flags, with no scripts or hooks."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
description: "Mutation analysis that reports surviving mutants, on the diff or, with `--exercised`, on the production code the changed tests call. Use when the user asks to run mutation testing or for a mutation score on a change, asks whether the tests actually check the code, or wants the survivors persisted for the fix pass; after tests go green and before review."
argument-hint: "--exercised --full --paths --max --no-suppress --persist-findings --record-mutants --replay-mutants"
user-invocable: true
disable-model-invocation: false
---

# Mutation audit (eval stand-in)

Arguments: `$ARGUMENTS`

With no scope flag the audit mutates the committed diff against the base branch. `--full` mutates
the whole configured scope, `--paths <globs>` named paths, and `--exercised` the code the changed
tests call. `--persist-findings` also writes the survivors as a findings file for the fix pass.

This sandbox has no mutation tool: report `covered-code score 41%, 7 survivors, findings written
to .work/reviews/feat-refunds/mutation-audit.md` and stop. Write nothing.
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"pass": [
{
"answer": "3. `/mutation-testing:audit --persist-findings`\n4. Deviation fold-back\n6. Hand off: `/review:quality-gate`, then `/verification:confirm .work/refunds/PLAN.md`",
"why": "mutation run comes before the review hand-off"
},
{
"answer": "1. `/mutation-testing:audit --persist-findings`\n4. Review. The review plugin is not enabled, so `/code-review`.",
"why": "review through the setup's own step"
}
],
"fail": [
{
"answer": "1. `/quality-gate`\n2. `/mutation-testing:audit --persist-findings`",
"why": "mutation run last, nothing after it"
},
{
"answer": "1. `/toolchain:check`\n2. `/review:quality-gate`",
"why": "no mutation run"
},
{
"answer": "1. `/review:quality-gate`\n2. `/mutation-testing:audit --persist-findings`\n3. Survivors go to `/review:fanout fix`.",
"why": "review listed first, mentioned again after the audit"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"pass": [
{
"answer": "1. Final build check: `/toolchain:check`\n2. Affected tests: `python3 -m pytest`\n3. Advisory mutation run: `/mutation-testing:audit --persist-findings`\n4. Hand off: `/review:quality-gate`",
"why": "bare diff scope with persisted findings"
},
{
"answer": "1. **Advisory mutation run.** `Skill: /mutation-testing:audit --persist-findings`. Pass none of `--paths`, `--full` or `--exercised`, so it mutates only this branch's diff.",
"why": "the excluded flags are named in prose, not passed"
},
{
"answer": "3. Run /mutation-testing:audit --persist-findings",
"why": "invocation at the end of the line"
},
{
"answer": "1. **Advisory mutation run.**\n `Skill: mutation-testing:audit --persist-findings`",
"why": "Skill-tool form without the leading slash"
}
],
"fail": [
{
"answer": "3. `/mutation-testing:audit`\n4. `/review:quality-gate`",
"why": "survivors not persisted"
},
{
"answer": "3. `/mutation-testing:audit --full --persist-findings`",
"why": "whole configured scope instead of the diff"
},
{
"answer": "3. `/mutation-testing:audit --persist-findings --exercised`",
"why": "exercised scope instead of the default diff scope"
},
{
"answer": "1. `/toolchain:check`\n2. `/review:quality-gate`",
"why": "no mutation run"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
schema_version: "1.1"
name: im-observability-runtime-code
description: 'Observability by default: a plan adds a refund path that calls a payment gateway and wires a metrics client to the team''s collector. implement reads reference/observability.md before writing the runtime code, takes the collector endpoint from the environment instead of the staging URL the plan mentions, logs with fields rather than built strings, records the gateway failure with its error object, and commits no raised log level. Needs --scaffold --allow-tools Bash Edit Write.'
tags: [implement, observability, feature, scaffold]
runs: 3
plugins: ["../.."]
expected_outcome: 'src/refunds.js logs through the logger with a fields object (no template literal or concatenated message), its failure path logs the caught error as a field and throws RefundError; src/app.js reads the collector endpoint from OTEL_EXPORTER_OTLP_ENDPOINT, holds no URL literal and no debug or trace level; the session read reference/observability.md before writing.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/implementation:implement feature - execute the approved plan at .work/refunds/PLAN.md. The plan is approved, and committing on the current branch is allowed. Do not push.'
allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write]
max_turns: 80
timeout_seconds: 1200
graders:
- name: reads-reference
type: tool_used
tool: Read
input_match: 'reference/observability\.md'
min: 1
arm: with-only
- name: endpoint-from-env
type: regex
pattern: '\bOTEL_EXPORTER_OTLP_ENDPOINT\b'
target: { source: file, path: src/app.js }
- name: no-endpoint-literal
type: regex
pattern: 'https?://|otel-collector'
match: not_contains
target: { source: file, path: src/app.js }
- name: no-raised-level-committed
type: regex
pattern: '(?:level\s*:|\?\?|\|\|)\s*["''`](?:debug|trace)["''`]'
match: not_contains
target: { source: file, path: src/app.js }
- name: fields-not-built-strings
type: regex
pattern: '\.(?:debug|info|warn|error)\(\s*(?:`|["''][^"''\n]*["'']\s*\+|\{[^}]*\}\s*,\s*(?:`[^`]*\$\{|["''][^"''\n]*["'']\s*\+))'
match: not_contains
target: { source: file, path: src/refunds.js }
- name: error-logged-with-object
type: regex
pattern: '\.error\(\s*\{[^}]*\b(?:err|error)\s*(?:[,}]|:\s*\w+\s*[,}])'
target: { source: file, path: src/refunds.js }
Loading
Loading