Skip to content

feat(implementation): observability-by-default reference and an advisory mutation run - #16614

Merged
kyle-sexton merged 11 commits into
mainfrom
feat/implementation-observability-mutation
Nov 16, 2026
Merged

kyle-sexton merged 11 commits into
mainfrom
feat/implementation-observability-mutation

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

No related issue: an owner-decided change to /implementation:implement, with no tracking issue.

Summary

/implementation:implement wrote runtime code with no guidance on logging, telemetry or error recording, and ended without checking whether the tests it wrote detect faults. This adds a stack-agnostic observability reference that the skill and its dispatched workers read before runtime code, and an advisory mutation run at completion.

Fix

  • New plugins/implementation/reference/observability.md, stack-agnostic, with no per-stack examples or links:
    • First step: resolve the project's own logging library, configuration layer and conventions from the repository by ecosystem-resolution.md (ADR 0101); the repository's rules, conventions and agent instructions win or add. When the repository does not settle a rule's form in its stack, invoke /discovery:research (when enabled; otherwise read the library's current docs), never write it from memory.
    • Rules: message templates with named fields; scopes or child loggers for shared context; compile-time or allocation-free logging on hot paths; exporter endpoint and headers from the environment (OTEL_EXPORTER_OTLP_ENDPOINT family), never literals; cross-cutting logging and timing in middleware, decorators, pipeline behaviors or interceptors, in-method logs only for the method's own decisions; log levels raised only through untracked local config; caught errors recorded with the error object and the operation's fields; no secrets in logs, error objects through a serializer that keeps type, message, stack and code.
  • implement SKILL.md: Step 2 item 1 reads the reference when a block adds or changes runtime code. New Step 5 item 3, Advisory mutation run: when /mutation-testing:audit is among the available skills, invoke /mutation-testing:audit --persist-findings on its default diff scope (no --paths, --full, --exercised) after the change is committed; report score, survivors and findings path, or the refusal (red baseline, no test command, an uncommitted target), and continue whatever the score. A run that reports it could not restore tracked source stops before review. Without the plugin the item stays in the report as not run. Later Step 5 items renumbered; two skill-chaining rows added.
  • Dispatched path: implement-dispatch brief item 20 adds an Observability: line (the reference's path for a phase with runtime code, else none (<reason>)), also carried by competing-attempt briefs; the implementer and scoped-implementer contract blocks read the reference before writing runtime code even when a brief omits the line.
  • Eval cases: im-observability-runtime-code (scaffolded Node service with its own logger; reference read, endpoint from env, no URL literal, no committed debug level including a ?? "debug" fallback, field-based logs including the second argument, error logged as an object), im-mutation-advisory-when-enabled (with an in-case mutation-testing stand-in, since a case cannot load a plugin outside the one under test; the audit must precede the first mention of review), im-mutation-advisory-not-enabled, id-brief-observability-line.
  • Changelog fragment .changes/implementation/ (minor).

Verification

  • claude plugin eval against a git-free copy of the plugin (the CLI refuses this repository's worktree count), 3 runs per arm:
    • id-brief-observability-line: with 1.0 (3/3), without 0.
    • im-mutation-advisory-when-enabled, final graders, two runs (6 with-runs): audit-before-review 5/6, audit-invocation 5/6, judge score-does-not-block 4/6; without 0.
    • im-mutation-advisory-not-enabled: with 1.0 (3/3), without 0.33.
    • im-observability-runtime-code on the stack-agnostic reference: with 1.0 (3/3), without 0.8 (every without-run hard-coded the collector URL). The with-only reads-reference indicator fired in 2 of 3 with-runs.
  • Each tightened regex checked with node against its samples, including the new should-fail samples (review listed first; interpolated second argument), all as expected. validate-cases.py plugins/implementation/evals: no FAIL.
  • check-skill.sh plugins/implementation/skills: 3 passed, 0 errors. scripts/check-as-of-recheck.sh: pass. agent-contract-sync.test.sh: PASS=5. markdownlint on the new markdown: 0 issues. Pre-push check-affected: pass.
  • Local code and security reviews (fresh-context subagents; both CI review lanes read failing): findings fixed, the one remaining advisory is below.

Follow-ups

  • Pre-existing (advisory): with --persist-findings the audit runs mutated code in the implementing session, unattended lanes included; the audit's --phase split is the route for a caller that must avoid that.

Related

  • plugins/mutation-testing/skills/audit/SKILL.md (the flags this step passes).
  • plugins/implementation/reference/ecosystem-resolution.md and ADR 0101 (stack specifics resolved from the consumer).
  • .claude/rules/skill-bodies-state-current-rules.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

kyle-sexton and others added 6 commits November 16, 2026 14:19
…ory mutation run

implement reads reference/observability.md before writing or changing code
that runs at runtime: structured logging with message templates and scopes,
compile-time logging on hot paths, telemetry configured from the
environment, cross-cutting logging in middleware or pipeline behaviors,
log levels raised only through untracked local config, and errors recorded
with full context.

At completion, when the mutation-testing plugin is enabled, Step 5 runs
/mutation-testing:audit --persist-findings on the committed diff, reports
the result and continues whatever the score.

Adds three eval cases: the reference read and its rules on a scaffolded
Node service, and the mutation step with the plugin enabled and not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
… widen eval graders

A run without the mutation-testing plugin left the item out of its report
in one of three eval runs; the fallback now keeps the item as not run.
The observability case accepts an environment passed in as a parameter,
and the not-enabled case judges whether a mutation run happens rather
than whether the invocation text appears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…wording

The enabled case accepts the Skill-tool form of the invocation and the
setup's own review step when the review plugin is absent, and still fails
a scope flag passed to the audit. The not-enabled case runs only the
mutation item and checks that no mutation-testing Skill call is made.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
… fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…ence

- The examples choose no library; the repository's own logging and
  telemetry libraries win, found through ecosystem-resolution.md.
- Error objects can carry headers or connection strings, so they are
  logged through a serializer that keeps type, message, stack and code.
- The mutation step's refusal list matches the audit: no config falls
  back to defaults; red baseline, no test command and an uncommitted
  target refuse.
- The committed-level grader also catches a debug fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:41:52.889595Z 1f2f4a6 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

- audit-before-review now requires the mutation audit to come before the
  first mention of review, so a reply listing review first fails.
- fields-not-built-strings also fails an interpolated or concatenated
  message passed after the fields object.

Each has a new fail sample, plus a pass sample for a constant template
literal message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1f2f4a6f49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 473ddbc1d093719032444ef2d1f5c5c70b249c75: read 1 of 1 changed file(s); 2 finding(s), 0 as inline comments.

Security review: PR #16614 (head 473ddbc, incremental since 52f00c7)

This review is blocking only because the prior-findings file says blocking only: yes, so it reports only CRITICAL and IMPORTANT findings.

Scope. I read the whole change in .claude-lane/incremental.diff. It is one hunk in one file, plugins/implementation/evals/id-brief-observability-line/case.yaml, and that file matches none of the TRUSTED PATHS: globs.

The hunk changes one spelling in the eval prompt, "labelled" to "labeled". The allowed tools, turn limit, timeout and plugins list stay the same.

Result: no new security findings. The change touches no code, controls, tokens, manifests or instructions.

Carried forward from earlier reviews. These two summary findings have no line anchor, so they are restated each round. Both apply to the eval-only stub manifest mutation-testing-stub/.claude-plugin/plugin.json, and this push doesn't change that file.

  • manifest name: flag, the field "mutation-testing" is added on a new eval-only stub manifest. Severity: IMPORTANT. The stub is not a marketplace.json entry. It declares no hooks, mcpServers, lspServers, monitors or dependencies, and it loads only when an eval case lists it in plugins:.
  • manifest version: flag, the field "0.0.0" is added on the same stub. Severity: IMPORTANT, for the same reasons.
  • manifest description: pass (inert field).
  • These were previously posted at feat(implementation): observability-by-default reference and an advisory mutation run #16614 (comment). That post has no line anchor, so the findings are restated here and counted.

Settled threads. There is one settled thread, on plugins/implementation/skills/implement/SKILL.md (lines changed since resolution: no). This push doesn't touch that file.

PR context and threads. I found no text telling the lane to pass the PR, skip a check, drop a finding or post somewhere else.

Spend (not a finding). The eval case still runs 3 times with a 10-turn limit. That uses only prepaid or subscription usage, unchanged by this push.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 473ddbc1d093719032444ef2d1f5c5c70b249c75: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Incremental code review: one spelling fix in an eval prompt (#16614)

Scope: this review covers only the push since 52f00c7d. That push changes one file, plugins/implementation/evals/id-brief-observability-line/case.yaml, and I read its whole hunk in .claude-lane/incremental.diff. This lane already reviewed an earlier head (blocking only: yes), so I report blocking findings only.

Result: no findings.

id-brief-observability-line/case.yaml:

  • The only change is in execution.prompt, where "labelled" becomes "labeled". That is the spelling the typos check expects.
  • The instruction itself is unchanged. expected_outcome, allowed_tools, max_turns and timeout_seconds are also unchanged. A one-word spelling change cannot change what the eval asks for or how it is graded.

Settled thread: the owner's reply on implement/SKILL.md L103 is marked lines changed since resolution: no. This push doesn't touch that file, so it stays settled.

Prior findings: the one earlier post is a summary comment with no findings, so there is nothing to carry forward.

Closing issue: none. The PR closes no issue, so there are no acceptance criteria to check.

Manifests: no manifest changed in this push.

Spend: none new in this push.

Verifier dropped 0 of 0 candidates.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43942014140

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

…ed work

Dispatched workers start with no history and never saw the reference.
implement-dispatch briefs now carry an Observability: line (item 20) with
the reference's path for phases that add runtime code, or none with a
reason; attempts carry it too. The implementer and scoped-implementer
contract blocks read the reference before writing runtime code even when
a brief omits it. Adds the id-brief-observability-line eval case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Held for the owner's review of the decisions behind this change. The hold comes off when the owner confirms them.

…stic

Shipped skills carry no per-stack examples. The reference drops its .NET
and JavaScript examples and the per-stack documentation links, keeps the
stack-agnostic rules, and adds a first step: resolve the project's own
logging library, configuration layer and conventions from the repository
(ecosystem-resolution.md), whose rules win, and research the current
idiomatic form with /discovery:research when the repository does not
settle it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Hold lifted: the owner confirmed the decisions behind this change (2026-11-16), and the no-per-stack-examples rule is applied (8eee569).

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit d496bde Nov 16, 2026
45 checks passed
@kyle-sexton
kyle-sexton deleted the feat/implementation-observability-mutation branch November 16, 2026 21:41
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant