Skip to content

Which trust measures can this repository compute today? #15668

Closed

Description

@kyle-sexton

Question

Which trust measures can this repository compute today from its own data: revert rate, the share of pull requests merged with no human touch, and corrections converted to checks?

From the talk

Chapter 1 and chapter 9, slides 3 and 34-38: quality and quantity together; her results shown as a scoreboard. Her own numbers are not targets.

Today and the gap

Pieces exist (/work-items:measure-delivery, /work-items:morning-brief); no single view, and no one has checked which of the three are computable.

What resolves it

For each measure: the data source, whether it is computable now, and the definition used. Evidence from this repository's history and the delivery-metrics literature. Basis: menu priority P1.

Done would add

A finding that sharpens the map's fog on trust measures, and, if computable, a view in an existing skill. Dogfood: the measures computed for this repository.

Activity

  1. added
    wayfind: researchWayfind decision item: external-evidence question; autonomous-capable.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  4. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: a research question asking which of three trust measures (revert rate, share of PRs merged with no human touch, corrections converted to checks) this repository can compute from its own data. For each measure the finding should give the data source, whether it is computable now, and the definition used, with evidence from this repository's history and the delivery-metrics literature. If a measure is computable, an existing skill gains a view.

    The scope is clear enough to start. Reading the repository history and the existing measure-delivery and morning-brief skills should settle it.

    Related, not a duplicate: #15650 covers counting corrections that became checks, which is only the third measure here.

    Automated triage. A maintainer reviews it.

  5. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  6. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane yet: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited; intake's refinement of this body is due and may supply them.

  7. self-assigned this
    on Nov 12, 2026
  8. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor
    No description provided.
  9. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Resolved: of the three trust measures, revert rate is computable today, the no-human-touch merge share is computable only as a lower bound, and corrections converted to checks is not computable because nothing tracked records the conversion. Basis: the research below. A fresh verifier confirmed the definitions and the arithmetic and lowered the confidence of the computed values; the caveats are stated per measure.

    From the talk

    • Slide 37 (chapter 9, Results) shows revert rate as "% of commits reverted" per month; the speaker says the number of pull requests that get reverted is steady at about half a percent. Her numbers are not targets (the map's Out-of-scope).
    • Slides 3 and 33 to 38 show pull request volume, release counts, contributors per month, revert rate and app performance. None shows a share of pull requests merged with no human touch; that measure is ours, not hers.
    • Slide 23 (chapter 6, Corrections) shows a rising step line headed "whenever you correct your agent", with the footer "lint against it or categorically eliminate the problem" and no axis label. It is the talk's only picture of corrections becoming checks, and it gives no definition.
    • These three points were read from our digest of the talk (slide descriptions and transcript), not from a fresh viewing of the video.

    Per measure

    Measure Data source Computable now Definition used Value for this repository
    Revert rate git history of main, through /work-items:measure-delivery Yes Landings whose subject is a revert, over all landings 13 of 7652 landings from 2026-08-20 to 2026-11-11, 0.17%
    No-human-touch merge share GitHub pull request data (author, merger, reviews, comments) Lower bound only Merged pull requests with no human review, comment, approval, merge or commit, over merged pull requests About 61% over a 2.3-day window: 305 of 500 by one read, 315 of 511 by a second
    Corrections converted to checks None tracked No Proven /corrections:codify proposals whose check was later installed, over all proposals None

    Revert rate

    • DORA defines change fail rate as the ratio of deployments that require immediate intervention following a deployment, and deployment rework rate as the ratio of deployments that are unplanned but happen as a result of an incident in production (page last updated January 5, 2026).
    • This repository has no incident record, so the value is a revert-based stand-in for DORA's rework rate, not the DORA measure. plugins/work-items/skills/measure-delivery/SKILL.md:46-69 states that the skill counts each first-parent commit on the default branch as a deployment and detects reverts from git history alone.
    • Confidence: medium. The 13 reverts were matched one for one between git and GitHub's merged pull requests; the 7652 denominator comes from git alone.
    • Not settled: the skill's script resolves 6 of the 13 reverts to an earlier landing and leaves 7 unresolved (one measurement only).

    No-human-touch merge share

    • The nearest published definition is Jellyfish's agentic-merged pull requests: merged pull requests whose only review activity came from AI tools. It covers review only and says nothing about who merged, so the wider definition in the table is ours.
    • It is a lower bound because agent sessions act under the owner's login: plugins/work-items/skills/morning-brief/SKILL.md:244 states that its owner-touch count includes any agent session holding that login. A pull request the owner's login touched cannot be split into human and agent action by login alone.
    • An indication, not a join: in the same window 198 of 510 commits on main carry a Claude co-author trailer, against 191 of the 500 merged pull requests authored by the owner's login.
    • What a full measure needs: one per-pull-request signal that separates the owner's own actions from an agent session's (the co-author trailer, a session trailer or a label).

    Corrections converted to checks

    • plugins/corrections/skills/codify/SKILL.md:153-164,181 states that codify writes proposals and notes under the gitignored .work/codify directory; it records no tracked link from a correction to an installed check.
    • On the one machine read, the corrections findings buffer returned no rows and the checkout has no .work/codify directory.
    • No published ratio for this measure was found (checked: one web search, which returned a conference talk page, an arXiv paper measuring rule recurrence and a pattern-book page on first-pass acceptance; not checked: the analytics docs of DX, Jellyfish and LinearB).
    • What it needs first: a tracked record, such as a trailer or label on the pull request that installs a check codify proposed. That is the subject of Count corrections that became checks and generate the rule-to-enforcer inventory.

    What this means for a view

    • Revert rate needs no new work: /work-items:measure-delivery already reports it.
    • The other two cannot be shown honestly until their missing record exists, so a single trust view is not worth building yet. The two records are the open questions: the per-pull-request human-or-agent signal, and the correction-to-check link (already owned by the work item above).

    Verification

    A fresh verifier re-fetched the DORA and Jellyfish pages and the three skill files and confirmed each quoted definition and line. After one rework round it still failed three rows:

    • the revert-rate value rests on two independent stores, not the three a high-confidence value needs, so it is stated here as medium;
    • the three statements about what our skills do are confirmed as what their SKILL.md files state, not as observed run-time behavior, and are worded that way here;
    • the research index cited the Jellyfish definition for the whole share; this comment limits it to review activity.

    The full research files and the raw command output go on the map's dossier branch.

  10. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Done (2026-11-12): Resolved; see the resolution comment: revert rate is computable now (13 of 7652 landings, 0.17%), the no-human-touch share only as a lower bound, corrections converted to checks not until a tracked record exists.

  11. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Owner decision (2026-11-12, "Let's go with recommendations"): the per-pull-request signal that separates the owner's own actions from an agent session's is the Claude co-author trailer on commits, which sessions already write. Basis: in the measured window 198 of 510 commits on main carry it, against 191 merged pull requests authored by the owner's login; that is an indication, not a per-pull-request join, so the first step of any view is that join.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

wayfind: researchWayfind decision item: external-evidence question; autonomous-capable.

Type

Fields

Work status

ready

Priority

medium

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions