Skip to content

Name every skipped file and unread value in toolchain, code-metrics, testing setup and code-tidying #15705

Description

@kyle-sexton

Parent

Refs #15703
Source: PLAN Phase 2, topic ecosystem-specifics

Depends on

  • none: on the frontier now

Outcome

A changed text source file that no ecosystem claims is named in the /toolchain:check and /toolchain:lint reports instead of being skipped silently, and four other tools name what they cannot read. Nothing is removed.

Demo: run /toolchain:lint on a diff that changes one .rs file in a repository with no Rust record; the report names the file.

What changes:

  • /toolchain:lint and /toolchain:check: an unclaimed text source file is named; binary and asset files stay quiet.
  • The code-metrics suppression audit and its scanner: the report names each file whose language the scanner has no kind for.
  • The testing plugin's setup script: its catch-all arm prints a line for any other detected language.
  • The three code-tidying file-scoping scripts read one shared extension list; the changed-files script gains the seven extensions it lacks, recorded as an intended change.

Method:

  • A characterization test first for each code-tidying caller (one fixture file per extension in the union, the old output asserted, then the named delta), and a negative-path test per script.
  • The two toolchain skill edits ship on a two-version compare of the toolchain check and lint cases plus one new case where a changed .rs file is named.
  • Run budget: 240 runs (18 cases plus 1 new, on the opus and sonnet aliases, three runs per side, after two measuring runs). Select cases with --case, use --runs 3 and a --max-cost-usd ceiling, and start no batch after a usage-limit error.
  • Eval batches run in the one local eval-runner session in the main checkout. Open the pull request with the do-not-merge label; the compare record releases it.

Decisions

  • Q26: every compare runs on the opus and sonnet aliases, three runs per side, the same count on both sides.
  • Owner, 2026-11-12: a compare runs only the changed skills' eval cases, not the plugin's whole suite.

Key interfaces

  • "Executed adapters, single-tool plugins and security floors stay; an adapter names what it cannot read instead of skipping it."
  • Dependency direction: skills depend on contracts and reach each other only by /plugin:skill.

Done when

  • AC4 (third clause): a source file outside every ecosystem is named, not skipped silently; the lint skill no longer says it silently skips.
  • AC8 (first clause): each tool touched here names a file or value it cannot read.
  • Each touched script's test file passes, including its negative-path test.
  • The changed-files script covers the seven added extensions, or reads the shared list.
  • The two-version compare is non-inferior and its record is in the pull request body, with this repository's dogfooding step.
  • AC7: nothing is cut here, so no control result is owed.
  • AC10: no enforcement-placing skill is added here.

Out of scope

  • Removing any bundled default or language page.
  • Accepting new ecosystem ids (a later item).

Activity

  1. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 12, 2026
  2. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    No work-resolve-issue run started in the 1 hour since agent-ready was last added. work-unblock removed and added agent-ready back to start a fresh run on main's workflow files. If this run ends the same way, it re-fires the issue again, waiting longer each time.

  3. removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 13, 2026
  4. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    Size: Large
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: /toolchain:lint and /toolchain:check should name changed text files that no ecosystem claims. The code-metrics scanner, the testing setup script and the three code-tidying scripts should also name what they cannot read. Nothing is removed.

    Recorded decision (Q146, owner answer "go w/ recomemdnations"), quoted from its opening:

    Decision recorded.
    Answer: (a). Every changed text file no ecosystem claims is named, text assets and config files included; binary files stay quiet.

    This settles the open question on which files count as unclaimed. It amends the Outcome line so that only binary files stay quiet, not asset files. A later recorded decision (Q150) concerns the compare on PR #16186 and leaves nothing open here.

    The work touches several subsystems and needs an eval compare, so it is sized Large. No merged pull request in the recent list appears to already fix this, and I found no duplicate.

    Automated triage. A maintainer reviews it.

  5. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. 2 remaining items

  7. github-actions commented on Nov 14, 2026

    @github-actions
    Contributor
    No description provided.
  8. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 14, 2026
  9. melodic-automation-lanes commented on Nov 14, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
    It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    host-only: the issue's done-when needs a two-version eval compare of the toolchain check and lint cases (18 cases plus 1 new, on the opus and sonnet aliases, three runs per side, 240-run budget), run in the one local eval-runner session in the main checkout, with its record in the PR body and the PR opened under the do-not-merge label that the compare record releases. This lane has no eval runner, cannot spend the model-run budget, and opens PRs ready without that label, so it cannot finish the issue as written.

    What the run says it did not do:

    • The whole implementation: toolchain lint and check reports, the code-metrics suppression scanner, the testing setup script's catch-all arm, the shared code-tidying extension list, plus characterization and negative-path tests. Not started, because the eval compare the toolchain edits ship on can…
    • The two-version eval compare (opus and sonnet, three runs per side) and its record in the PR body. It needs the local eval-runner session.
    • Tidy, review, build and lint did not run, because the working tree has no changes.

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43539931117

    To hand it back to the lane, remove host-run and set Work status to agent-ready again.

  10. kyle-sexton commented on Nov 14, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    Owner answer, 2026-11-14T16:21:44Z, in the GitHub Support hub session (transcript 851c547a), to the question below: "go w/ recomemdnations". Posted by the same hub lane after a session clear, from the transcript.

    Question (Q146). In #15705, which changed files should /toolchain:lint and /toolchain:check name as unclaimed?

    Answer: (a), the recommended option. Every changed text file no ecosystem claims is named, text assets and config files included (for example .svg, .css); binary files stay quiet. The options not taken were (b) only files whose extension is in a shared code-extension list, and (c) a new source-extension list inside toolchain.

    This amends the issue's Outcome line that says asset files stay quiet: it now applies to binary files only.

    Acceptance criteria for this decision:

    • /toolchain:lint and /toolchain:check name every changed text file that no resolved ecosystem claims, whatever its extension.
    • A changed binary file outside every ecosystem gets no mention in the reply.
    • An eval case covers each of the two behaviors above with a grader of its own.
  11. kyle-sexton commented on Nov 14, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    Owner answer, 2026-11-14 (about 19:38Z), in the GitHub Support hub session, to the question below: "go w/ recommendations". Posted by that session.

    Question (Q150). The third two-version compare of pull request #16186 (registration 3) read opus MET and sonnet NOT MET on its paired rule: change +0.00, 95% interval -0.06 to +0.07 against a 0.05 margin, with one case, ln-go-fix-filters-non-go, down from 1.00 to 0.67 on one run of three. Registration 3 forbids reruns. What happens to the pull request?

    Answer: (a), the recommended option, an exception to the no-rerun clause. One diagnostic check of the dropped case only (ln-go-fix-filters-non-go, sonnet, 6 runs per side), with its bar written before any run (registration 4). Registration 3's sonnet result stays recorded as NOT MET whatever the check shows. If the after side scores no lower than the before side, the pull request is released with both records in its body; if lower, it stays held for a text fix and a fourth full compare. The options not taken were (b) a text change and a fourth full compare now, and (c) release with no further evidence.

    This amends the issue's done-when item "The two-version compare is non-inferior" for this pull request: it is met by registration 3's record (opus non-inferior; sonnet rules 1 and 3 met, rule 2 missed on one case) together with a registration 4 diagnostic check that meets its bar.

    Result of the check (re-derived from its two result files): both files VALID, before mean 1.00 (6 of 6 runs pass every grader), after mean 1.00 (6 of 6), bar met.

    Acceptance criteria for this decision:

    • The pull request body carries registration 3's record with the sonnet rule 2 miss stated as NOT MET.
    • The pull request body carries registration 4's record, with the bar that was written before the run and its result.
    • Registration 4's result shows the after-side mean on ln-go-fix-filters-non-go at or above the before-side mean, both files VALID.
  12. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions