[BeatSeat]

Governing Issue and delivery slice

Refs #455

• Accepted scope / delegation: task: add GPU measurement-validity and profiler-attribution guidance to inference-benchmark-design #455 is proposed; scope acceptance is requested there. No closure is claimed.
• What this PR delivers: A 175-line GPU measurement-validity reference, a six-line skill pointer, pinned public-source metadata, matrix navigation, three regenerated projections and the installed-resource fixture increment from 923 to 924.
• What it does not deliver: A benchmark implementation, new skill, campaign threshold, empirical validation or performance claim.

GPU E2E timings can move with clock, thermal, power and sharing conditions; profiler replay can introduce different execution conditions. The reference gives a procedure for retaining paired timings, checking conditional within-run attribution and Amdahl predictions, and connecting gains to work counters and a predicted profiler signature. It covers timing/cache/baseline/JIT pitfalls and keeps the four supplied numerical observations explicitly unverified.

The existing inference-benchmark-design owner already defines comparable workloads, measurement integrity and evidence interpretation. Under the knowledge-base ownership contract, this is an owner-local reference extending that procedure; it does not need a separate skill identity or registry entry. Candidate state, empirical not_run and efficacy not_evaluated remain explicit.

Knowledge authorship declaration

{
  "schema_version": 1,
  "head_sha": "8c0b894a2dc9b8572700ca3f80b119e75f2377db",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Codex CLI",
      "model_id": "gpt-6.1-sol",
      "effort": "xhigh",
      "role": "author"
    }
  ]
}

Changed artifacts and evidence

Requirement	Actual change	Check / evidence and result	Remaining gap
Concise, source-grounded procedure	methods/inference-benchmark-design/references/gpu-measurement-validity.md (175 lines); SKILL.md adds six lines	Ten public-source identities; citation labels resolve; retained document hashes checked against retrieved bytes	No GPU/profiler or efficacy execution
Attribution and pitfalls	Same-run region shares, conditional internal clock, serial/critical-path limits, Amdahl check, count metrics, explanation-first adoption, five timing-trap categories	Official versioned NVIDIA/PyTorch docs, pinned Triton source and robust-kbench v1 inspected	Normalization/adoption is an unverified author proposal; legacy counters require target queries
Preserve evidence state	Owner metadata and four task-supplied observations	Normalized checkout/package reads succeed; observations labelled unverified with unavailable raw evidence	No supplied campaign artifacts
Keep projections and packaging consistent	Both matrices; target generated record, _provenance.json, coverage view; tests/check_installed.py count 924	Both generators and checks pass; installed-wheel inventory and new-reference byte checks pass	Independent review pending

Canonical skill paths above are relative to src/inferrsi/knowledge/data/skill_library/. The registry, unrelated skill records and generated navigation README remain unchanged.

Checks and dependencies

All Python commands used the requester-specified interpreter and checkout PYTHONPATH. Final Git-dependent checks used a temporary Git 2.49.1 build, with the repository tools unchanged and native --no-lazy-fetch support.

• tools/build_skill_index.py --source-revision 3671a98ad340fb19e6595d1c8d818c7e9ca2577a, then --check: passed, 80 records current (regenerated after the two-commit split; generated churn limited to the target record, _provenance.json and the coverage view).
• tools/build_skill_coverage.py, then --check: passed. Generated churn is limited to the target record, shared provenance and coverage view.
• python -m pytest tests/test_skill_catalog.py tests/test_skill_index.py tests/test_skill_library.py tests/test_skill_library_readers.py tests/test_skill_coverage_view.py tests/test_knowledge_*.py --basetemp <temporary-test-dir> -rs: 479 passed, 5 skipped, 112 subtests passed. The final head retains the same canonical/generated bytes; the later one-line installed-count adjustment was exercised directly below.
• python -m ruff check src tests: passed, including the installed-count adjustment.
• Built and noneditably installed a wheel from the final head with temporary build dependencies. Ran tests/check_installed.py outside the checkout with isolated imports and checkout-resource comparison: passed, all 924 resource files byte-identical. The installed normalized reader also retrieved the new reference byte-for-byte.
• Checkout and package normalized read/resource, citation/line-count checks and retained public-document digest comparisons: passed.
• tools/knowledge_declaration_paths.py --base origin/main --head HEAD: passed; 7 protected paths, 0 policy paths, 3 generated paths (overlapping classifications), 2 other paths.
• Offline assess() from tools/check_knowledge_model_policy.py against this PR body, final head, base-relative paths and trusted base policy: passed, declaration compliance only. No GitHub status was published; the remote CLI/status workflow was not run against an uncreated PR.
• git diff --check, clean-worktree check, DCO sign-off and Agent-assisted-by trailer inspection on both commits: passed.

Resolved validation failures: The initial generator invocation used system Git 2.43, which lacks --no-lazy-fetch; final generation/checks used native Git 2.49.1. A broad-suite run found the paper source needed the normalized id field; it was corrected and the complete selected suite rerun. The first installed check exposed the expected resource-count increment; the fixture now expects 924 and the final-head check passes. Initial build-dependency downloads timed out; a bounded pinned-dependency retry succeeded.

Skipped / not run: One browser test requires unavailable Playwright; two fresh-venv tests remain opt-in; two NTFS-junction cases are Windows-only. The separate noneditable wheel check above ran successfully. Full ready/nightly CI, independent review, GPU profiling, model quality, L1–L4 campaign execution and efficacy evaluation were not run.

Limits

• The attribution/adoption procedure is proposed guidance. Its internal-clock correction requires common drift and an unaffected reference; overlapping kernels require critical-path analysis. No numeric gate is introduced.
• The four numerical cases are task-supplied author reports with no raw samples/configurations/profiles supplied. They are neither public-source findings nor evidence of efficacy.
• Metric spellings have an official legacy NCU mapping; availability and semantics still need inspection on the actual GPU/tool version. PyTorch's naming advice is explicitly an inference from its documented module/build identity contract, not a claim that every duplicate-name load fails.
• The unversioned SMI page is identified by its inspected HTML hash in metadata. Raw third-party documents/code are not redistributed. Source review does not grant execution permission or promote this skill.
• The wheel build emitted setuptools package-discovery warnings for data directories; the complete installed-resource inventory check nevertheless passed. Packaging configuration was not expanded in this slice.
• Follows the two-step workflow in docs/knowledge-contribution-workflow.md: canonical files are committed first (3671a98a, the pinned source_revision), then records are regenerated from that commit and committed (8c0b894a). The source revision is an ancestor of the head, so no separate bundle is needed.

Remaining work and Issue closure

• Remaining Issue requirements / follow-up: Maintainer scope decision on task: add GPU measurement-validity and profiler-attribution guidance to inference-benchmark-design #455, independent review and the remote required checks.
• Goal-acceptance owner or accepted closure delegation: Maintainer to designate; no closure delegation was granted.
• Goal ready for acceptance? Not claimed.

[github-actions]

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

• Next: run the pinned source/index --check below at head 8c0b894a2dc9b8572700ca3f80b119e75f2377db.
• Blockers: Executable snapshot entries are not supported. No unsafe fallback; details below.
• Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 60; Canonical library paths: 3; Generated projection paths: 2; Other paths: 55. These are paths, not independent claims.
• Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.

Author verification steps (nonpaid; not yet run)

1. Bind the checkout. Run git rev-parse HEAD; it must equal 8c0b894a2dc9b8572700ca3f80b119e75f2377db. If it differs, obtain a current plan; do not reuse old results.
2. Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:

python tools/build_skill_index.py --source-revision 3671a98ad340fb19e6595d1c8d818c7e9ca2577a --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.

• inference-benchmark-design
  Selected by the trusted path mapping: src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md; src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json (+1 more in the plan).
  Unsupported: Executable snapshot entries are not supported. No snapshot prepared; task metadata is in details. Do not relax snapshot safety.
  Uncovered scope by location: Canonical library paths: 3; Generated projection paths: 2; Other paths: 55. Path counts are not claim counts; no metadata-only conclusion is inferred.

3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
   Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1797341910 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

• Retained evidence is planned / unexecuted.
• No supported comparison cases were selected.
• Uncovered/unsupported scope needs a case or explicit reviewer disposition.
  Plan SHA-256: 497cd20678afbe369cdea4b5bed22ff96317693ed9ebbd547f1c396d7c721d50
  Base: cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 · Head: 8c0b894a2dc9b8572700ca3f80b119e75f2377db
  Trusted implementation: cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill	Comparison	Cases	Estimated calls
inference-benchmark-design	unsupported snapshot: Executable snapshot entries are not supported	—	0

Selection/task metadata (not a relevance verdict):

{"adapter":"audit","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md","src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json","src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md"],"tasks":[{"evidence_scope":"http_fixed_workload","id":"audit-threshold","request":"Audit the fixed input report using its contract. Compute the median of per-trial request means, including every request in each trial, and the percentage reduction relative to the baseline aggregate.","split":"holdout"}]}

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"3671a98ad340fb19e6595d1c8d818c7e9ca2577a","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (60)

• uncovered: "AGENTS.md"
• uncovered: "CHANGELOG.md"
• uncovered: "docs/contracts/current-design.md"
• uncovered: "docs/module-layout.md"
• uncovered: "docs/reference-matrix.md"
• uncovered: "docs/searchspace-matrix.md"
• uncovered: "docs/skill-coverage.md"
• uncovered: "src/inferrsi/_json_v1.py"
• uncovered: "src/inferrsi/apps/api.py"
• uncovered: "src/inferrsi/apps/archive.py"
• uncovered: "src/inferrsi/apps/artifact_reader.py"
• uncovered: "src/inferrsi/apps/bridge.py"
• uncovered: "src/inferrsi/apps/context_provider.py"
• uncovered: "src/inferrsi/apps/legacy/evaluation.py"
• uncovered: "src/inferrsi/apps/legacy/run.py"
• uncovered: "src/inferrsi/apps/result.py"
• uncovered: "src/inferrsi/apps/views/attempt.py"
• uncovered: "src/inferrsi/arguments.py"
• uncovered: "src/inferrsi/claim.py"
• uncovered: "src/inferrsi/contracts.py"
• uncovered: "src/inferrsi/contracts/init.py"
• uncovered: "src/inferrsi/contracts/claim.py"
• uncovered: "src/inferrsi/contracts/json_v1.py"
• uncovered: "src/inferrsi/contracts/runspec.py"
• uncovered: "src/inferrsi/execution.py"
• uncovered: "src/inferrsi/harness/init.py"
• uncovered: "src/inferrsi/harness/execution_service.py"
• uncovered: "src/inferrsi/harness/execution_spec.py"
• uncovered: "src/inferrsi/harness/run_store.py"
• uncovered: "src/inferrsi/harness/runner.py"
• uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json"
• uncovered: "src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json"
• unsupported: "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md"
• unsupported: "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json"
• unsupported: "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md"
• uncovered: "src/inferrsi/runspec.py"
• uncovered: "tests/_gate_policy_support.py"
• uncovered: "tests/architecture_real_ports.py"
• uncovered: "tests/check_installed.py"
• uncovered: "tests/h01_run_store_child.py"
• uncovered: "tests/test_apps_assess.py"
• uncovered: "tests/test_apps_evidence.py"
• uncovered: "tests/test_apps_execution_binding.py"
• uncovered: "tests/test_apps_output.py"
• uncovered: "tests/test_apps_prepare.py"
• uncovered: "tests/test_claim.py"
• uncovered: "tests/test_contracts_package.py"
• uncovered: "tests/test_dashboard_workflow.py"
• uncovered: "tests/test_first_wave_isolation_policy.py"
• uncovered: "tests/test_flare_adapter.py"
• uncovered: "tests/test_harness_execute.py"
• uncovered: "tests/test_harness_run_store.py"
• uncovered: "tests/test_knowledge_model_policy.py"
• uncovered: "tests/test_legacy_run_split.py"
• uncovered: "tests/test_module_boundaries.py"
• uncovered: "tests/test_package_root_inventory.py"
• uncovered: "tests/test_smoke.py"
• uncovered: "tests/test_target_module_import_gate.py"
• uncovered: "tests/test_trial_check_reports.py"
• uncovered: "tests/test_trial_checks.py"

[amy-why-3459]

Thanks for the careful caveats and the two-step generation. I reviewed this against current main cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 and head 8c0b894a2dc9b8572700ca3f80b119e75f2377db (branch point a92c77ccaed68426c83154cff7619745f34d0058 / source revision 3671a98ad340fb19e6595d1c8d818c7e9ca2577a, then the regeneration commit).

1. Matrix section. The new row in docs/reference-matrix.md was added under "Task-specific benchmark selection (issue [RFC] Benchmark selection by task for InferRSI #60)", so docs/skill-coverage.md also labels it as an [RFC] Benchmark selection by task for InferRSI #60 source, and the searchspace-matrix paragraph was inserted under the same heading. This reference is general GPU measurement methodology, not a task-selection source for [RFC] Benchmark selection by task for InferRSI #60. Please move the matrix row and the searchspace note to a dedicated short section so the generated coverage label is accurate.

2. Rebase onto current main. The branch is based on a92c77ccaed68426c83154cff7619745f34d0058. Package shared technical contracts and retire root modules #453 (cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332) landed afterwards and added C1 assertions to tests/check_installed.py while keeping the installed-resource count at 923. A merge may apply the 923 → 924 change cleanly, but please rebase and, if the source-revision commit changes, regenerate the records and update the declaration head_sha before merge.

I also checked: the new reference is 175 lines; its SHA-256 78d750321e65b69f1cc2006997d2e66494284e65cc7eb55d9f2c88e17aedb87c is present in the generated skill record; the Triton v3.2.0 tag is 9641643da6c52000c807b5eeed05edaec4402a67; and do_bench matches the write-based flush / preliminary-estimate / per-sample event description. Status remains candidate / empirical not_run. I did not re-download the NVIDIA/PyTorch documents to re-check the retained hashes, and I did not rerun the generators or the installed-wheel check. #455 is still Proposed, with no acceptance owner yet.

I am not approving this head until the matrix placement is fixed.