Memoize skill capability classification by semantic input

[Trecek]

Investigation

Prior investigation completed interactively; the causal benchmark and controlled classifier probe are summarized below.

Problem

PR #4345 correctly added level-aware capability contracts and authenticity validation, but it also made full-document capability classification part of every valid construction through _skill_info_from_frontmatter(). Recipe semantic tests invoke that pure classifier repeatedly with identical (content, skill_name) inputs, turning a correctness check into a reproducible CPU regression.

The general shard more than doubled immediately across the PR:

Measurement	Before a6d14a8	After c4317b4	Delta
General pytest	312.83s	689.64s	+376.81s / 2.204×
Passing tests	22,968	22,996	+28
Selected items	23,520	23,549	+29
Execution pytest	112.94s	122.71s	+9.77s

Primary records: before Actions run, after Actions run.

Both general jobs used Hosted Compute Agent provisioner 20260707.563, Ubuntu image 20260720.247.2, CPython 3.11.14, and uv 0.9.21. The Actions runner application version and Azure region differed, but two serialized local repetitions of the exact commits independently reproduced the regression using the CI general paths, four worksteal workers, experimental features enabled, and task test-check:

Run	Before	After
1	138.08s	265.55s
2	133.40s	275.90s
Mean	135.74s	270.73s

The local mean increase was 99.44%. The second post-PR run was slower, so cold-cache and CI-runner variance do not explain the regression.

Confirmed hot path

The introducing path is:

1. _skill_info_from_frontmatter() calls validate_skill_capability_authenticity() for valid parsed frontmatter.
2. Authenticity validation reaches classify_skill_capability_evidence(content, skill_name), which scans the full document with the capability pattern set.
3. A bare run_semantic_rules(recipe) creates a validation context without a resolver.
4. The dominant skill-content rules call _resolve_skill_md(), which creates a resolver when none is injected. Fifteen sites in rules_skill_content.py resolve and then reread skill files.

The representative test tests/recipe/test_rules_cmd.py::test_bundled_recipes_have_no_bare_rebase_findings calls run_semantic_rules() separately for 15 bundled recipe YAMLs. Direct instrumentation on the exact post-PR tree recorded:

Version	SkillInfo builds	Classifier calls	Test-call time
Before a6d14a8	2,756	classifier did not exist	1.13s
After c4317b4	2,708	2,708	7.40s
After, diagnostic process-wide memo	2,708 resolution-path calls	141 misses / 2,567 hits	1.60s

The normal post-PR probe spent 6.033936 seconds in classify_skill_capability_evidence() and 6.485070 seconds cumulatively in _skill_info_from_frontmatter(). Before the PR, _skill_info_from_frontmatter() consumed 0.229071 seconds.

The diagnostic memo wrapped the classifier in an unbounded worker-process lru_cache; it was an experiment, not the proposed production shape. Its cache key was the classifier's actual arguments, so the 141 misses represent unique (content, skill_name) input pairs across all 15 recipe validations. When those inputs are counted separately per run_semantic_rules() call, there are 272 summed unique pairs and at most 141 in any one call. This distinction matters: a validation-scoped cache alone cannot reproduce the measured 141-miss result across the whole test.

Attribution and ruled-out causes

• Pytest call phases increased by 500.43 summed seconds and accounted for 99.61% of the measured phase delta; setup increased 1.90 seconds and teardown 0.05 seconds.
• Collection-only measurements were 14.72 seconds before and 14.76 seconds after.
• Outer-process CPU increased by 502.90 seconds, almost exactly matching the 502.38-second phase increase. Added CPU was 96.77% user mode, consistent with parsing and regex classification rather than blocking subprocess or filesystem work.
• tests/recipe contributed 429.42 seconds, or 85.48%, of the measured phase delta. tests/cli contributed 54.39 seconds and tests/server 11.57 seconds.
• The gap between pytest wall time and summed phase time divided across four workers changed by only 1.88 seconds, ruling out xdist scheduling imbalance as the driver.
• Fixture changes did alter exercised code, but the controlled classifier probe and recipe-directory attribution—not setup timing alone—establish that they are not the primary cause.
• Only 28 additional tests passed and 29 additional items were selected. Existing bundled-recipe tests rose from roughly one second to 8–9.5 seconds.

Largest recipe-file deltas included test_rules_ci.py (+39.34 summed seconds), test_bundled_recipes_general.py (+37.00s), test_rules_contracts.py (+27.16s), and test_merge_prs_queue_common.py (+21.17s).

Required outcome

Make capability-evidence classification reusable across recipe validations when—and only when—the classifier's semantic inputs are identical.

The production implementation must bound retained cache entries and prevent arbitrarily large individual documents from defeating that bound. A finite process-local pure-function memo is acceptable because changed project-local content produces a different key; an unbounded cache of full skill documents is not. A validation-scoped layer may be added, but it is insufficient by itself to match the controlled result across the 15 validations in the representative test.

The key must include the effective logical skill name as well as the complete canonical content, or an equivalently collision-safe representation. Digest alone is incorrect: classify_skill_capability_evidence() uses skill_name to distinguish a self-reference from a cross-skill reference.

Requirements

REQ-CACHE-001: Repeated classifier invocations with identical canonical content and effective logical skill name must reuse one previously computed immutable evidence result within the bounded cache lifetime.

REQ-CACHE-002: A change to either canonical content or effective logical skill name must not reuse a result computed for the prior input.

REQ-CACHE-003: The cache must remain coherent and must not expose mutable results under same-process concurrent callers; duplicate computation on a simultaneous cold miss is acceptable, but callers must converge on the same correct resident value.

REQ-CACHE-004: Omitted, empty, and explicit skill_name values that resolve to the same effective frontmatter name must share one semantic cache entry.

REQ-MEM-001: Classifier reuse must have an explicit finite bound and must not retain an unbounded number of full skill documents or evidence tuples.

REQ-MEM-002: Cache admission or byte accounting must prevent an arbitrarily large skill document from defeating the retention bound; oversized inputs must still classify correctly when not retained.

REQ-CONTRACT-001: Evidence classification, source spans, ordering, genuine-versus-artifact decisions, detected capabilities, authenticity diagnostics, exact-role contracts, projection behavior, and resume authority introduced by #4345 must remain functionally unchanged.

REQ-TEST-001: Deterministic serial tests must count underlying full-document scans separately from public classifier invocations and prove one scan per unique (canonical content, effective logical skill name) input while the entry remains resident.

REQ-TEST-002: Tests with identical content under different logical skill names must prove that self-reference and cross-skill-reference classification cannot collide.

REQ-TEST-003: Tests must prove that edited project-local content produces a fresh classification result without requiring process restart or global cache clearing.

REQ-TEST-004: Cache-bound tests must prove that retained entries cannot grow without limit and that eviction does not change classifier correctness.

REQ-TEST-005: Tests must prove that oversized inputs remain correct without causing mandatory cache retention.

REQ-TEST-006: Same-process threaded tests must prove cache-state coherence and unchanged results under concurrent callers.

REQ-TEST-007: Tests must prove that skill_name=None, skill_name="", and the equivalent explicit frontmatter name normalize to one cache entry.

REQ-PERF-001: The implementation PR must include paired before/after measurements using approved Taskfile commands for the representative bundled-recipe test and the general shard, reporting underlying scan counts and elapsed time.

Acceptance criteria

• After clearing the new bounded cache, the representative test's 2,708 public classifier invocations execute no more underlying full-document scans than its dynamically measured unique (content, skill_name) inputs while resident; the recorded diagnostic workload had 141 unique inputs and 2,567 reusable calls.
• The representative test produces identical semantic findings and recovers most of its measured 6.27-second call-time regression; the diagnostic memo reduced it from 7.40 seconds to 1.60 seconds versus the 1.13-second pre-PR baseline.
• Two serialized general-shard measurements on equivalent base and implementation revisions show that the regression is materially reduced. Deterministic scan-count and correctness assertions are the merge gate; wall time is corroborating evidence, not a flaky test SLA.
• Verification uses task test-check or other approved Taskfile commands; pytest is never invoked directly.

Scope boundaries

Primary implementation scope:

• src/autoskillit/workspace/skill_capabilities.py
• focused capability-classifier tests
• operation-count coverage for the representative recipe test

This issue does not remove or weaken #4345 capability validation. It also does not redesign recipe _LOAD_CACHE, all semantic-rule content access, projection-cache validation, doctor reporting, or the project-local resolver. Those paths may contain additional repeated work, but the controlled probe isolates classifier reuse as the immediate regression and nearly restores the representative test to baseline.

The planned two-way general-shard split is capacity work, not evidence that this CPU regression is fixed.

Historical context

Commit 7f6149f748161146a480b6346cc9ae9f6d5d0f73 previously added DefaultSkillResolver caching and resolver threading specifically to eliminate redundant recipe-validation I/O. Commit c066608929ebf858c7b903c87e330759afc01abf, an intermediate #4345 branch commit, optimized direct invocation closure scans but did not cover recipe semantic-rule classification. This issue uses a different lifetime: a finite memo over the classifier's complete semantic inputs, so freshness does not depend on path or resolver identity.