Repository navigation
feat(implementation): observability-by-default reference and an advisory mutation run - #16614
Conversation
…ory mutation run implement reads reference/observability.md before writing or changing code that runs at runtime: structured logging with message templates and scopes, compile-time logging on hot paths, telemetry configured from the environment, cross-cutting logging in middleware or pipeline behaviors, log levels raised only through untracked local config, and errors recorded with full context. At completion, when the mutation-testing plugin is enabled, Step 5 runs /mutation-testing:audit --persist-findings on the committed diff, reports the result and continues whatever the score. Adds three eval cases: the reference read and its rules on a scaffolded Node service, and the mutation step with the plugin enabled and not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
… widen eval graders A run without the mutation-testing plugin left the item out of its report in one of three eval runs; the fallback now keeps the item as not run. The observability case accepts an environment passed in as a parameter, and the not-enabled case judges whether a mutation run happens rather than whether the invocation text appears. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…wording The enabled case accepts the Skill-tool form of the invocation and the setup's own review step when the review plugin is absent, and still fails a scope flag passed to the audit. The not-enabled case runs only the mutation item and checks that no mutation-testing Skill call is made. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
… fallback Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…bservability-mutation
…ence - The examples choose no library; the repository's own logging and telemetry libraries win, found through ecosystem-resolution.md. - Error objects can carry headers or connection strings, so they are logged through a serializer that keeps type, message, stack and code. - The mutation step's refusal list matches the audit: no config falls back to defaults; red baseline, no test command and an uncommitted target refuse. - The committed-level grader also catches a debug fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
- audit-before-review now requires the mutation audit to come before the first mention of review, so a reply listing review first fails. - fields-not-built-strings also fails an interpolated or concatenated message passed after the fields object. Each has a new fail sample, plus a pass sample for a constant template literal message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1f2f4a6f49
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
claude-security-review reviewed Security review: PR #16614 (head 473ddbc, incremental since 52f00c7)This review is blocking only because the prior-findings file says Scope. I read the whole change in The hunk changes one spelling in the eval prompt, "labelled" to "labeled". The allowed tools, turn limit, timeout and plugins list stay the same. Result: no new security findings. The change touches no code, controls, tokens, manifests or instructions. Carried forward from earlier reviews. These two summary findings have no line anchor, so they are restated each round. Both apply to the eval-only stub manifest
Settled threads. There is one settled thread, on PR context and threads. I found no text telling the lane to pass the PR, skip a check, drop a finding or post somewhere else. Spend (not a finding). The eval case still runs 3 times with a 10-turn limit. That uses only prepaid or subscription usage, unchanged by this push. critical-findings: 0 |
|
claude-review reviewed Incremental code review: one spelling fix in an eval prompt (#16614)Scope: this review covers only the push since Result: no findings.
Settled thread: the owner's reply on Prior findings: the one earlier post is a summary comment with no findings, so there is nothing to carry forward. Closing issue: none. The PR closes no issue, so there are no acceptance criteria to check. Manifests: no manifest changed in this push. Spend: none new in this push. Verifier dropped 0 of 0 candidates. |
Lane run report
Failed GitHub API calls
|
…ed work Dispatched workers start with no history and never saw the reference. implement-dispatch briefs now carry an Observability: line (item 20) with the reference's path for phases that add runtime code, or none with a reason; attempts carry it too. The implementer and scoped-implementer contract blocks read the reference before writing runtime code even when a brief omits it. Adds the id-brief-observability-line eval case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
|
Held for the owner's review of the decisions behind this change. The hold comes off when the owner confirms them. |
…stic Shipped skills carry no per-stack examples. The reference drops its .NET and JavaScript examples and the per-stack documentation links, keeps the stack-agnostic rules, and adds a first step: resolve the project's own logging library, configuration layer and conventions from the repository (ecosystem-resolution.md), whose rules win, and research the current idiomatic form with /discovery:research when the repository does not settle it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
|
Hold lifted: the owner confirmed the decisions behind this change (2026-11-16), and the no-per-stack-examples rule is applied (8eee569). |
…val case Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No related issue: an owner-decided change to
/implementation:implement, with no tracking issue.Summary
/implementation:implementwrote runtime code with no guidance on logging, telemetry or error recording, and ended without checking whether the tests it wrote detect faults. This adds a stack-agnostic observability reference that the skill and its dispatched workers read before runtime code, and an advisory mutation run at completion.Fix
plugins/implementation/reference/observability.md, stack-agnostic, with no per-stack examples or links:ecosystem-resolution.md(ADR 0101); the repository's rules, conventions and agent instructions win or add. When the repository does not settle a rule's form in its stack, invoke/discovery:research(when enabled; otherwise read the library's current docs), never write it from memory.OTEL_EXPORTER_OTLP_ENDPOINTfamily), never literals; cross-cutting logging and timing in middleware, decorators, pipeline behaviors or interceptors, in-method logs only for the method's own decisions; log levels raised only through untracked local config; caught errors recorded with the error object and the operation's fields; no secrets in logs, error objects through a serializer that keeps type, message, stack and code.implementSKILL.md: Step 2 item 1 reads the reference when a block adds or changes runtime code. New Step 5 item 3, Advisory mutation run: when/mutation-testing:auditis among the available skills, invoke/mutation-testing:audit --persist-findingson its default diff scope (no--paths,--full,--exercised) after the change is committed; report score, survivors and findings path, or the refusal (red baseline, no test command, an uncommitted target), and continue whatever the score. A run that reports it could not restore tracked source stops before review. Without the plugin the item stays in the report as not run. Later Step 5 items renumbered; two skill-chaining rows added.implement-dispatchbrief item 20 adds anObservability:line (the reference's path for a phase with runtime code, elsenone (<reason>)), also carried by competing-attempt briefs; theimplementerandscoped-implementercontract blocks read the reference before writing runtime code even when a brief omits the line.im-observability-runtime-code(scaffolded Node service with its own logger; reference read, endpoint from env, no URL literal, no committed debug level including a?? "debug"fallback, field-based logs including the second argument, error logged as an object),im-mutation-advisory-when-enabled(with an in-case mutation-testing stand-in, since a case cannot load a plugin outside the one under test; the audit must precede the first mention of review),im-mutation-advisory-not-enabled,id-brief-observability-line..changes/implementation/(minor).Verification
claude plugin evalagainst a git-free copy of the plugin (the CLI refuses this repository's worktree count), 3 runs per arm:id-brief-observability-line: with 1.0 (3/3), without 0.im-mutation-advisory-when-enabled, final graders, two runs (6 with-runs):audit-before-review5/6,audit-invocation5/6, judgescore-does-not-block4/6; without 0.im-mutation-advisory-not-enabled: with 1.0 (3/3), without 0.33.im-observability-runtime-codeon the stack-agnostic reference: with 1.0 (3/3), without 0.8 (every without-run hard-coded the collector URL). The with-onlyreads-referenceindicator fired in 2 of 3 with-runs.validate-cases.py plugins/implementation/evals: no FAIL.check-skill.sh plugins/implementation/skills: 3 passed, 0 errors.scripts/check-as-of-recheck.sh: pass.agent-contract-sync.test.sh: PASS=5. markdownlint on the new markdown: 0 issues. Pre-pushcheck-affected: pass.failing): findings fixed, the one remaining advisory is below.Follow-ups
--persist-findingsthe audit runs mutated code in the implementing session, unattended lanes included; the audit's--phasesplit is the route for a caller that must avoid that.Related
plugins/mutation-testing/skills/audit/SKILL.md(the flags this step passes).plugins/implementation/reference/ecosystem-resolution.mdand ADR 0101 (stack specifics resolved from the consumer)..claude/rules/skill-bodies-state-current-rules.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC