Skip to content

Add clause-dash advisory, judge calibration and frozen holdout cases - #55

Merged
ryanduguid merged 2 commits into
mainfrom
ryan/yt-skill-quality-2026-11-01
Nov 1, 2026
Merged

ryanduguid merged 2 commits into
mainfrom
ryan/yt-skill-quality-2026-11-01

Conversation

@EquityBot

Copy link
Copy Markdown
Owner

Summary

Adds a clause-dash advisory and new stock phrases to laconic-check, a calibration mode for the blocker judge, 10 frozen holdout cases and 3 skill-text additions.

laconic-check lint
├── clause-dashes (new, advisory): clause dashes in 2 or more sentences of one paragraph
│   └── counted once per sentence, so a pair around one aside counts once
├── stock phrases: + innovative, practical solutions, timeless openers
└── exit code: advisory findings never fail the run
scripts/judge.py --calibrate evals/judge-calibration.jsonl   (agreement on the blocker field)
evals/holdout-cases.jsonl   (10 frozen cases, pinned by a digest test)

Skill text: a fiction line in 'Write literally', a social post venue with a regression row, and a limits sentence in the README. Each is mirrored into .cursor/skills/.

Evidence

  • Calibration: on the Style Manual copy, counting raw dashes per paragraph gave 34 findings, mostly paired asides; counting per sentence gives 2, now pinned as an advisory ceiling beside the unchanged 82-finding ceiling for other rules.
  • Judge: judge.py --calibrate agreed with all 10 labelled responses (5 blockers).
  • Checks: cargo test (56), clippy -D warnings, cargo fmt --check, Python unittest (168), run_evals.py validate, claude plugin validate . and git diff --check pass after rebasing on 24ce59c.

Merge danger

Door: two-way. Blast radius: lint output

The new rule is advisory, so lint exit codes do not change; output gains one finding type.

Unverified

check_context_compat.ts did not run (bun not installed locally); this change does not touch its inputs.

…e and frozen holdout cases

- laconic-check: advisory for clause dashes in 2 or more sentences of a paragraph
  (never fails lint), calibrated on the Style Manual (2 advisories); stock phrases
  'innovative', 'practical solutions' and timeless openers.
- judge.py --calibrate with 10 hand-labelled responses; 10 of 10 agreed on
  1 November 2026 (claude-opus-5-5).
- Prose skill: fiction keeps figurative language; social post venue and regression case.
- README limits now say the cases guided rule wording; 10 frozen holdout cases.
@ryanduguid
ryanduguid enabled auto-merge (squash) November 1, 2026 10:02
@sonarqubecloud

sonarqubecloud Bot commented Nov 1, 2026

Copy link
Copy Markdown

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 36 complexity · 0 duplication

Metric Results
Complexity 36
Duplication 0

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

⚠️ Resolve issues before merging

The calibration workflow can still abort unexpectedly when the judge returns no rows, because it indexes the first result without handling an empty response. This should be addressed before merging.

The advisory-only CLI exit-code contract and mirrored skill consistency lack automated tests, leaving two important acceptance areas unverified. The judging entry point also has elevated complexity, which increases maintenance risk but is not itself blocking.

About this PR

  • Add automated coverage for the advisory-only CLI exit-code contract and for consistency between the canonical and mirrored skill files.
1 comment outside of the diff
skills/laconic-prose/SKILL.md

line 1 🟡 MEDIUM RISK
Suggestion: Add the required skill metadata, including an author, to the canonical skill file and synchronize the .cursor/skills/ mirror so the metadata is identical in both copies.

Test suggestions

  • Detect clause dashes across two sentences, including em dashes, en dashes, spaced hyphens and double hyphens, while ignoring paired asides, ranges, compounds, list markers, tables and code.
  • Confirm clause-dash findings are advisory and do not produce a failing lint exit code.
  • Detect the added stock phrases and avoid false positives for legitimate words, protected text and unrelated forms.
  • Run calibration fixtures independently, calculate agreement, report mismatches and reject an empty fixture set.
  • Validate the ten holdout cases and detect any modification through the pinned digest.
  • Verify the new skill guidance is synchronised between canonical and mirrored files.
  • Enforce the Style Manual corpus ceilings for normal findings and clause-dash advisories.
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Confirm clause-dash findings are advisory and do not produce a failing lint exit code.
2. Verify the new skill guidance is synchronised between canonical and mirrored files.

TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback

Comment thread scripts/judge.py
prompt = build_judge_prompt(cases[fixture["case_id"]], {"fixture": fixture["response"]}, labels, rubric)
rows, spent = judge(prompt, (fixture["case_id"], fixture["name"]), labels)
cost += float(spent or 0)
if rows[0]["blocker"] != fixture["blocker"]:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

Handle an empty result explicitly before indexing rows[0], either by recording a calibration error or raising a descriptive ValueError.

TIP React 👍 if this review is right, or reply with what's wrong and how it could be better to help improve the reviewer.

Comment thread scripts/judge.py
return parser


def main(argv: Optional[list[str]] = None) -> int:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

Suggestion: Consider extracting runner setup and calibration handling into focused helpers so main primarily parses arguments, dispatches the selected mode, and reports errors.

See Issue in Codacy
See Complexity in Codacy

TIP React 👍 if this review is right, or reply with what's wrong and how it could be better to help improve the reviewer.

@ryanduguid
ryanduguid merged commit 0fc90f4 into main Nov 1, 2026
19 checks passed
@ryanduguid
ryanduguid deleted the ryan/yt-skill-quality-2026-11-01 branch November 1, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants