Skip to content

Generate the rule-to-enforcer inventory and its per-tier count #15711

Description

@kyle-sexton

Parent

Refs #15703
Source: PLAN Phase 12, topic ecosystem-specifics

Depends on

Outcome

/harness-config:audit-automation-gaps prints, read-only, how a repository's enforcement is spread across the ladder's tiers: a count per tier and the rules that sit lower than they could, both rolled up from one generated rule-to-enforcer inventory.

Demo: run the audit against this repository; it prints five tier counts and a "sits lower than it could" list.

What changes:

Method:

  • Tests first for the generator: a fixture repository with one rule per tier yields one row each and a per-tier count; a rule held by guidance that a static check could decide appears under "sits lower than it could".
  • The skill-text change ships on a two-version compare.
  • Run budget: 100 runs (about 3 automation-gaps audit cases, counted first, plus 2 new, on the opus and sonnet aliases, three runs per side, after two measuring runs). Select cases with --case, use --runs 3 and a --max-cost-usd ceiling, and start no batch after a usage-limit error.
  • Eval batches run in the one local eval-runner session in the main checkout. Open the pull request with the do-not-merge label; the compare record releases it.

Decisions

Key interfaces

  • "harness-config (audit-automation-gaps): owns one generated rule-to-enforcer inventory (rule, enforcer, rung, tier). The per-tier count and 'sits lower than it could' are rollups of it."
  • Inventory row: rule, enforcer, rung, tier.
  • Two-way door: the generated inventory.

Done when

  • AC9: the automation-gaps audit reports, read-only, a count per ladder tier and the rules that sit lower than they could.
  • The generator's test file passes.
  • A run of the audit against this repository prints five tier counts (dogfood), named in the pull request body.
  • The Count corrections that became checks and generate the rule-to-enforcer inventory #15650 comment's URL is in the pull request body.
  • The two-version compare is non-inferior and its record is in the pull request body.
  • AC7: nothing is cut here, so no control result is owed.
  • AC10: no new enforcement-placing skill is added here.

Out of scope

Activity

  1. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 12, 2026
  2. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).

  3. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: #15708

    I understood this as: add a generator that produces a rule-to-enforcer inventory (rule, enforcer, rung, tier). Then make /harness-config:audit-automation-gaps print, read-only, a count per tier and a "sits lower than it could" list from that inventory. The issue also asks for a narrowing comment on #15650, which is related work (the inventory moves here). The issue states an exact outcome and done-when, so I have no open questions. It depends on #15708 (the ladder release), which is still open. I found no duplicate and no merged fix.

    Automated triage. A maintainer reviews it.

  4. melodic-automation-lanes commented on Nov 14, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane yet: blocker #15708 is not closed by a listed human or a merge.

  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"2f07f90a4ee2544ecd1a06068dd3a1dafccd9b5310616dfc4b6ba9b2e108de6f","body_sha256":"8ba52f207993a1ad9445df17011f19aa4004a06781a440e8de740a4cd353567b"}

  6. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor
    No description provided.
  7. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  8. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
    It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    host-only: the issue's Method and Done-when need host steps this CI lane cannot take. (1) The skill-text change to /harness-config:audit-automation-gaps 'ships on a two-version compare' run as about 100 eval runs ('--case', '--runs 3', '--max-cost-usd', opus and sonnet aliases) 'in the one local eval-runner session in the main checkout', and the compare record must be in the PR body. This lane can run no eval harness, no claude session and no interpreter. (2) The PR must open with the do-not-merge label so that the compare record releases it, but this lane opens its PR ready. (3) A comment must be posted on #15650 when the PR opens, with its URL in the PR body, and this lane posts no comments. (4) The generator's test file and a dogfood run of the audit (five tier counts) must actually run, and this lane's Bash runs only fixed wrappers, so it cannot execute a test script or the audit even once. A partial PR (generator and tests only) would open ready without the do-not-merge hold, and it would carry a shell generator that never ran, so it was not started.

    What the run says it did not do:

    • Generator and its test file: not written, because this lane cannot run a test script, so the generator would ship without ever running
    • Two-version eval compare (about 100 runs) in the local eval-runner session: needs a host session
    • Comment on #15650 narrowing it to the corrections-to-checks counter: this lane cannot post comments
    • Dogfood run of the audit printing five tier counts: needs a host session
    • Opening the PR with the do-not-merge label: this lane opens PRs ready
    • Dependency #15708 state not checked: this lane has no tracker read, though ladder 3.1.0's five tiers suggest it has landed

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43751910317

    To hand it back to the lane, remove host-run and set Work status to agent-ready again.

  9. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions