[Ezra144israel]

Description

I maintain Agent Teamwork Kit. This adds test-the-test to check whether tests fail when the behavior is broken.

It overlaps with test-gap-audit on weak assertions. This skill focuses on running discriminating checks and keeping changes to tests separately inspectable from changes to the implementation. It includes the objective-integrity reference and the original MIT notice. The source adaptations are single quotes around the description and the spelling correction re-used to reused.

Source: public skill and reference.

Testing

Tested with Copilot CLI 1.0.78 on a small Python example. Copilot invoked the skill and added an ordering assertion. The original count-only test passed the broken sort. The new test failed that same broken sort and passed with the corrected sort. The test did not measure improvement against a run without the skill.

On Awesome Copilot main 613efc8c21d9b3ce642dff361d17006cd9967e28:

• npm run skill:validate: all 436 skills valid.
• npm run plugin:validate: all 104 plugins and the external catalog valid.
• npm start and a repeat npm run build: passed.
• bash eng/fix-line-endings.sh and git diff --check: passed.
• The only generated change is one row in docs/README.skills.md. The 12 existing external-catalog warnings are unchanged.

Prepared with Codex. Copilot generated the example test. Codex checked the file diff and recorded test result.

Pull request checklist

• Read and followed CONTRIBUTING.md and the paid-services guidance. This skill needs no paid service.
• New skill is in the correct folder and follows the naming and frontmatter rules.
• Required reference and MIT notice are included.
• Tested with GitHub Copilot.
• Generated documentation is current.
• Targets main.
• Follows the Code of Conduct and is contributed under the MIT License.

[github-actions]

🔒 PR Risk Scan Results

Scanned 3 changed file(s).

Severity	Count
🔴 High	0
🟠 Medium	0
ℹ️ Info	0

✅ No matching risk patterns were detected in changed files.

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

[github-actions]

🚦 Submission status: 💬 Review in progress

Risk tier: merge-risk:medium — New or substantially changed resource without privileged execution
Required to merge: passing submission-gate checks plus 1 approval from reviewers with write access.

Why this tier

• Adds 3 file(s), e.g. skills/test-the-test/LICENSE
• Changes 524 lines (low-risk limit is 40)

Automated checks

Check	Status	Details
Line endings	✅ Passed	Passed · logs
Spelling	✅ Passed	Passed · logs
Generated README consistency	✅ Passed	Passed · logs
Skill validation	✅ Passed	Passed · logs
Skill lint (vally)	✅ Passed	Passed · logs
Risk scan	✅ Passed	Passed · logs
Contributor reputation	✅ Passed	Passed · logs
Duplicate resource scan	✅ Passed	Passed · logs
PR quality signal	⏭️ Skipped	Skipped by its workflow · logs

Review

• Approvals: 0/1
• Assigned reviewer: not assigned yet — comment /request-review to ask for one
• Review target date: not set
• Still needed: 1 more approval(s)
• No staffed reviewer pool owns these files yet; any reviewer with write access counts as the domain reviewer.

Commands

Command	Who	What it does
/rerun-checks	PR author, maintainers	Re-runs failed or incomplete checks and re-evaluates this gate
/request-review	PR author, maintainers	Asks the review rotation to assign a reviewer (adds needs-reviewer)

_{Updated for 9c94b38 · gate run · This comment is maintained automatically — see submission gate docs.}

[github-actions]

🔍 Vally Lint Results

✅ All checks passed

Scope	Checked
Skills	1
Agents	0
Total	1

Severity	Count
❌ Errors	0
⚠️ Warnings	0
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	✅ test-the-test (2/2 checks passed)
ℹ️	✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️	✓ spec-compliance: All spec checks passed.
ℹ️	✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️	✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️	1 skill(s) linted, 1 passed

Full linter output

### Linting skills/test-the-test
✅ test-the-test (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

[Ezra144israel]

/request-review

[github-actions]

🙋 Added needs-reviewer. Reviewer routing is assigning a reviewer now.

[aaronpowell]

Thanks for this, and for being upfront about the source, the overlap with test-gap-audit, and the limits of your test run.

The part of this skill that adds something Copilot doesn't already do is the evaluator-integrity material:

• keeping evaluator-side changes (tests, fixtures, thresholds, expected values) separately inspectable from implementation changes
• the six-class evaluator-change table
• the positive/negative-control rule ("an evaluator change that silences the negative control is a defect")
• the non-equivalent checks: counterfactual, metamorphic, property-based, and proxy perturbation
• the Level A three-liner: TRUE OBJECTIVE / STRONGEST APPARENT-SUCCESS SHORTCUT / DISCRIMINATING CHECK

That targets a real agent failure: making the test pass by changing the test. I'd like to see the skill built around it. Most of the rest should go. Here's why each section isn't pulling its weight in this repo:

1. Generic testing advice (Vertical Slice Discipline, Test Quality Checklist, What Makes a Bad Test, Database Assertions). These are standard TDD and test-hygiene rules: red→green, test through public interfaces, don't mock internals, cover a failure path. The model already follows them when asked to write good tests, and test-gap-audit already covers weak or mis-scoped tests. Repeating them doesn't change what Copilot does. It does fill the context window and push your distinctive rules further down, so the model is less likely to follow them. Your own test run shows this: the broken-sort case is exactly what Copilot catches without the skill, so it doesn't show the skill made a difference.

2. Seam-Specific Test Expectations table. The rows describe one particular app architecture: server actions, project-scoped DB reads, "authority markers", a natural-language routing seam with "guard/veto vocabulary in object position", and "rendered answer / readback surface". In a typical Copilot user's repo (a Go service, a React library, a CLI), most rows either don't apply or will be misread. If you keep anything here, cut it to 2–3 general rows, such as validation helpers, persistence, and authz boundaries.

3. Agent Teamwork Kit vocabulary throughout. "builder return", "unified return block", "governed crossing", "reviewing seat", "orchestrator seat", "reconvergence", EXECUTOR_OWNED / METHOD_LOCKED, CONTEXT IMPACT: ADVISORY, CONTRACT_RECONVERGENCE_REQUIRED, AUTHORITY_STOP, "pressure-test FAIL", "reviewer NEEDS_REVISION". All of these point to roles, documents and status enums that live in your kit, not here. A user who installs only this skill has no orchestrator seat and nowhere to send reconvergence. Copilot will either ignore these instructions or invent output that looks official but means nothing in their repo. Please rewrite each one in plain terms ("tell the user", "stop and flag it", "note it in your summary") or remove it.

4. reference/objective-integrity.md. About 70% of this file routes into those kit concepts, or is a non-claim and inoculation-prompting disclaimer aimed at reviewers of your framework rather than at the agent doing the work. The five-object decision model and the "routes needing attention here" list (threshold changes, judge biasing, cherry-picked runs, stale evidence) are useful. Fold those into SKILL.md in a few lines and drop the reference file.

5. Sad-Path Acceptance Gate, Return Requirement, Performance and resource boundaries. These require fixed report blocks, including a nine-row matrix and a five-line perf block, every time the skill fires — including any test-writing task. On routine work this is pure overhead, and it's the kind of output users learn to skip; once they do, the instructions that matter get skipped too. If you want an output contract, keep only the Level A three-liner and require it only when a test or evaluator was changed in the same change as the code.

6. Description. Once trimmed, please narrow the description so the skill triggers on the case it is for, e.g. "Use when a change edits tests, fixtures, thresholds, or other pass/fail criteria alongside the implementation, or when a passing test may not prove the behavior." The current "Use when writing tests, reviewing test coverage…" will fire on every testing task and collide with test-gap-audit and the language-specific test skills.

Validation. Once trimmed, please re-test with a scenario that exercises the core. For example, ask Copilot to fix a failing test where the easy path is to loosen the assertion or change the expected value, and show that with the skill it keeps the evaluator change separate, checks a negative control, and refuses the shortcut. Ideally include the same prompt without the skill for comparison.

A focused ~100–150-line version along these lines would be a distinctive addition to the skills catalog.