[WS1][C2] Canonical workload, logical identity, and reference command (#268) - #292
[WS1][C2] Canonical workload, logical identity, and reference command (#268)#292maxiaosong1124 wants to merge 8 commits into
Conversation
Freeze the WS1 numerical SSOT for issue RL-Align#267: four-judgment tolerance rows, dtype/TF32/FP8 policy, comparison roles, chain logprob aggregates, shared resolver, and op_checks wiring so forward and gradient accuracy no longer share one threshold path. Add schema tests, usage docs, and a migration checklist for remaining private-atol call sites (C3/C4/C8). Closes RL-Align#267
Record acceptance-criteria mapping, verification commands, and residual scope so issue RL-Align#267 can close without implying full RL-Align#266 exit.
Freeze the full Qwen3-8B Dense logical workload SSOT for WS1 closeout C2: manifest pins (config fingerprint, weight content hash, 2x2 Batch/Chunk matrix, varlen fixtures, packing, dual backend profiles, representative case_ids), logical identity restore after pad/pack/chunk, singleton_aggregate vs BN multiset plan, registry-resolved candidate binding, and a single reference command. Document registry-vs-runtime actual boundary and Triton missing_required reds without silent fallback. Closes RL-Align#268
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe PR adds a versioned gtest tolerance contract, provenance-aware operator checks, a canonical Qwen3-8B WS1 workload, reference tooling, candidate runtime evidence, tests, and C2 closeout documentation. ChangesWS1 contract and operator checks
Canonical workload and evidence
C2 planning and documentation
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Operator as gtest operator check
participant Candidate as backend candidate
participant Contract as tolerance contract
participant Report as candidate report
Operator->>Candidate: execute candidate with provenance
Operator->>Contract: resolve judgment, roles, dtype, and backend tolerance
Contract-->>Operator: validated tolerance specification
Operator->>Report: record comparison result and provenance
sequenceDiagram
participant CLI as ws1_reference.py
participant Manifest as WS1 manifest
participant Workload as ws1_workload.py
participant Payload as reference payload
CLI->>Manifest: validate workload identity and selected cell
CLI->>Workload: build reference_payload
Workload->>Payload: include fixture hashes and layout digests
CLI-->>Payload: emit JSON or human-readable output
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (13)
rl_engine/kernels/gtest/tolerance.py (2)
260-262: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueRaise
ContractResolveErrorwhen a declared profile has no profile contract.Line 260 validates
backend_profileagainstpolicy.backend_profiles. Line 262 then indexesbackend_profile_contractsdirectly. A contract that lists a profile inbackend_profilesbut omits it frombackend_profile_contractsraisesKeyErrorinstead of the module's contract error. Schema validation only requires the two WS1 profiles, so a third declared profile can reach this path.♻️ Proposed fix
- profile_contract = contract["policy"]["backend_profile_contracts"][provenance.backend_profile] + profile_contracts = contract["policy"]["backend_profile_contracts"] + if provenance.backend_profile not in profile_contracts: + raise ContractResolveError( + f"missing backend_profile_contracts entry for {provenance.backend_profile!r}" + ) + profile_contract = profile_contracts[provenance.backend_profile]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/kernels/gtest/tolerance.py` around lines 260 - 262, Update the validation in the backend profile resolution flow before the `profile_contract` lookup to verify that `provenance.backend_profile` exists in `contract["policy"]["backend_profile_contracts"]`; raise `ContractResolveError` for a declared profile without a contract, preserving the existing error behavior for unknown profiles.
936-964: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueNormalize all supported PyTorch FP8 dtypes to
"float8"Map the concrete torch FP8 variants, including
torch.float8_e4m3fnuzandtorch.float8_e5m2fnuz, and their bare names to"float8". This makes them use the documented FP8 out-of-scope error instead ofunsupported dtype.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/kernels/gtest/tolerance.py` around lines 936 - 964, Update the dtype normalization logic in the shown torch-dtype branch to recognize all supported PyTorch FP8 variants, including float8_e4m3fn, float8_e5m2, float8_e4m3fnuz, and float8_e5m2fnuz, using both qualified and bare names. Return "float8" for each variant so they reach the documented FP8 out-of-scope handling instead of raising unsupported dtype.tests/test_op_checks.py (2)
251-280: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCover the gold
reference_dtypeguard as well.This test exercises the candidate branch at
rl_engine/kernels/gtest/op_checks.pylines 336-340. The sibling branch at lines 341-345 raisesgold output dtype ... disagrees with provenance reference_dtypeand has no test. Add a case whosegold_fnreturns BF16 instead of FP32, and assertContractResolveErrorwithmatch="gold output dtype". Without it, a regression in the reference-dtype guard passes CI.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_op_checks.py` around lines 251 - 280, Extend test_ws1_report_checks_observed_output_dtype_against_provenance with a gold_fn case that returns bfloat16 where provenance.reference_dtype is float32. Run the operator suite using this gold function and assert ContractResolveError with match="gold output dtype", covering the gold reference_dtype validation branch.
191-221: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a
triton_cuda_bf16happy-path case.This test covers the
cuda_bf16profile end to end. Thetriton_cuda_bf16profile appears intests/test_op_checks.pyonly at line 242, wherebackend="triton"is used to trigger a mismatch. No test runs a candidate withbackend_profile="triton_cuda_bf16",requested_backend="triton", andactual_backend="triton"throughrun_operator_suite.The contract declares both profiles as required for WS1. Add one case that mirrors this test with the triton profile, so a regression in
backend_profile_contractsfortritonfails here and not only in the contract unit tests.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_op_checks.py` around lines 191 - 221, Add a happy-path test alongside test_ws1_report_persists_roles_and_backend_provenance using backend_profile="triton_cuda_bf16", requested_backend="triton", actual_backend="triton", and backend="triton". Run it through run_operator_suite with the existing logp case, then assert the same judgment, role fields, and serialized backend provenance expectations, including absence of a baseline output.rl_engine/kernels/gtest/tolerance_contract.json (2)
217-224: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
approx_kl0cannot fail beforemax_abs_dlogpat these thresholds.
approx_kl0ismean(exp(dlogp) - 1 - dlogp), which is approximatelymean(dlogp^2 / 2)for smalldlogp. At the BF16 limitmax_abs_dlogp = 5.0e-2, the largest possibleapprox_kl0is about1.25e-3, well under the declared5.0e-2. The FP32 pair is even further apart:1.0e-5ondlogpboundsapprox_kl0near5.0e-11against a1.0e-5threshold.
judge_logprob_aggregatesrequires all three metrics, butapprox_kl0can never be the metric that fails. The gate therefore reduces tomax_abs_dlogpplusclipfrac0. Ifapprox_kl0is meant to catch distributed drift that a single-token maximum misses, set it near the square of themax_abs_dlogpscale, for example1.0e-3for BF16 and1.0e-10for FP32. If the loose value is deliberate, record the reason in the contract.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/kernels/gtest/tolerance_contract.json` around lines 217 - 224, The approx_kl0 thresholds in the contract are too loose to detect drift before max_abs_dlogp, especially for BF16 and FP32. Update the by_execution_dtype thresholds for approx_kl0 to meaningful values near the square of each max_abs_dlogp scale (approximately 1.0e-3 for BF16 and 1.0e-10 for FP32), or document the deliberate rationale for retaining the current thresholds.
127-158: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueConfirm the gradient thresholds are calibrated, not copied.
Every
gradient_accuracycell holds the sameatol/rtolas the matchingforward_accuracycell. The schema now keeps the rows independent, which is the stated goal of C1. The values themselves still mirror the forward table. BF16 backward passes usually accumulate more error than the forward pass, so identical thresholds can be optimistic forreductionandattention. Confirm these numbers come from measured gradient error, or record them as provisional in the contract so a later gate can tighten or loosen them with evidence.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/kernels/gtest/tolerance_contract.json` around lines 127 - 158, Review the gradient_accuracy thresholds for elementwise, reduction, logprob, and attention against measured backward-pass errors rather than copying forward_accuracy values. Update the cells with calibrated atol/rtol values, or explicitly mark the current thresholds as provisional using the contract’s existing status/schema conventions so they can be revised when evidence is available.rl_engine/kernels/gtest/op_checks.py (1)
244-266: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
resolve_toleranceruns twice at both comparison sites._resolve_tolerancealready callsresolve_tolerancefor contracts that contain ajudgmentsblock, then returns onlyatolandrtol. Both call sites then callresolve_toleranceagain with identical arguments purely to readcomparison_lhs_roleandcomparison_rhs_role. The cell lookup, dtype-policy resolution, and support validation therefore execute twice per comparison, and the thresholds and the roles travel through two separate code paths that can drift.
rl_engine/kernels/gtest/op_checks.py#L244-L266: callresolve_toleranceonce withjudgment="gradient_accuracy", keep the returned spec, and readatol,rtol, and both role fields from it. Use_resolve_toleranceonly when"judgments"is absent from the contract.rl_engine/kernels/gtest/op_checks.py#L315-L331: apply the same single-resolution pattern withjudgment="forward_accuracy"and drop the separateforward_speccall.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/kernels/gtest/op_checks.py` around lines 244 - 266, The gradient comparison at rl_engine/kernels/gtest/op_checks.py lines 244-266 and the forward comparison at lines 315-331 each resolve tolerance twice; update both sites to call resolve_tolerance once when the contract has a judgments block, retaining its returned spec for atol, rtol, comparison_lhs_role, and comparison_rhs_role, and use _resolve_tolerance only when judgments is absent. Apply judgment="gradient_accuracy" at the anchor and judgment="forward_accuracy" at the sibling, removing the separate gradient_spec and forward_spec calls.tests/test_ws1_workload.py (2)
264-273: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the unused
inverselist.Lines 265-267 build
inverse, but line 272 recomputes the same mapping withlist(perm).index(old_i). Theinverselist is never read.♻️ Proposed cleanup
# Restoring original order via inverse permutation. - inverse = [0] * len(perm) - for new_i, old_i in enumerate(perm): - inverse[old_i] = new_i + inverse = [0] * len(perm) + for new_i, old_i in enumerate(perm): + inverse[old_i] = new_i # samples in permuted are batch.samples[perm[i]]; map back: restored_samples = [] for old_i in range(len(batch.samples)): - # find which permuted index holds original old_i - new_i = list(perm).index(old_i) - restored_samples.append(permuted.samples[new_i]) + restored_samples.append(permuted.samples[inverse[old_i]])This keeps
inverseand drops the redundant linear search.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_ws1_workload.py` around lines 264 - 273, Remove the unused inverse list and its construction from the restoration logic near the restored_samples loop, leaving the existing list(perm).index(old_i) mapping behavior unchanged.
403-416: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffThis test couples to the source text and to line numbers that will rot.
Two brittleness concerns:
- Lines 407 and 412 assert on exact substrings of
operator_specs.py, including'"{spec_name}": OperatorSpec('. A formatting change in that file, for example Black splitting the call across lines, breaks the assertion even though the registry is still correct. Import the registry and check the mapping keys instead of grepping the source.- Lines 413-416 parse
algorithm_sourceaspath:lineand only assert that the line number is within the file length. That check passes for any line in a long file, so it does not prove the pin points at the declaration. The manifest pins values such asrl_engine/kernels/ops/cuda/matmul/det_gemm.py:3, and those line numbers drift on any edit above the declaration.Consider pinning a symbol name in
algorithm_sourceinstead of a line number, and assert that the named symbol exists in the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_ws1_workload.py` around lines 403 - 416, Replace source-text grepping in test_declared_candidates_resolve_to_real_operator_specs with an imported operator registry and assert that all manifest operator_spec_map values resolve to registry keys. Replace algorithm_source path:line parsing with a stable path:symbol reference, then assert the referenced symbol is declared in the target file rather than validating only a line range; update the manifest entries accordingly.rl_engine/testing/__init__.py (1)
16-48: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider exporting
apply_paddingandrestore_logical_order_from_paddedtoo.The package exports
apply_packing,apply_chunking, andrestore_logical_order, but not the padding counterparts.rl_engine/testing/ws1_workload.pydefinesapply_padding(line 778) andrestore_logical_order_from_padded(line 851), and the padding path is one of the three transforms the manifest pins underlogical_identity.restore_before_compare_after. Consumers that need the padded layout must reach into the module directly, which weakens the public surface.♻️ Proposed export addition
from .ws1_workload import ( LogicalBatch, PhysicalLayout, WorkloadError, WS1Manifest, apply_chunking, apply_packing, + apply_padding, build_logical_batch, fixture_hash, load_manifest, reference_payload, restore_logical_order, + restore_logical_order_from_padded, )Add the matching entries to
__all__.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/testing/__init__.py` around lines 16 - 48, Update the testing package exports in __all__ to include apply_padding and restore_logical_order_from_padded, matching the existing exports for apply_packing, apply_chunking, and restore_logical_order. Keep the current imports and export ordering consistent with the corresponding symbols from ws1_workload.rl_engine/testing/ws1_workload.py (1)
973-981: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winThe identity hash only covers the keys listed in
_REQUIRED_TOP_LEVEL.
_manifest_identity_payloadselects keys from the fixed_REQUIRED_TOP_LEVELtuple.validate_manifestuses the resulting hash to enforce the rule stated at lines 281-284: any numerics-affecting edit must change the identity. A future top-level manifest section that is not added to_REQUIRED_TOP_LEVELis silently excluded from both the identity hash andfixture_hash, so a numerics-affecting edit in that section would pass validation with an unchangedfixture_identity_sha256.Hash the full manifest minus
fixture_identity_sha256instead of an allowlist. That keeps the invariant true as the schema grows.♻️ Proposed change
def _manifest_identity_payload(raw: Mapping[str, Any]) -> dict[str, Any]: - return {k: raw[k] for k in _REQUIRED_TOP_LEVEL if k != "fixture_identity_sha256"} + # Hash every declared section so new manifest keys cannot escape the identity. + return {k: v for k, v in raw.items() if k != "fixture_identity_sha256"}This changes the digest, so
fixture_identity_sha256inrl_engine/testing/ws1_manifest.jsonmust be regenerated with it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/testing/ws1_workload.py` around lines 973 - 981, Update _manifest_identity_payload to include every top-level manifest field except fixture_identity_sha256, rather than filtering through _REQUIRED_TOP_LEVEL, so future numerics-affecting sections are covered by manifest_identity_hash and validate_manifest. Regenerate fixture_identity_sha256 in the manifest to match the updated digest.rl_engine/testing/ws1_manifest.json (1)
367-370: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winCross-validate
candidate_case_idsagainstrepresentative_cases.Three fixtures declare
candidate_case_ids, but_validate_fixturesinrl_engine/testing/ws1_workload.py(lines 505-510) only checks that the list is non-empty forshort_full_model_fixtureandlong_full_model_fixture. It never checks that the referenced IDs exist inrepresentative_cases, and it never inspectsrepresentative_full_model_fixtureat all. A renamed or removedcase_idwould leave a dangling reference that no test detects. The IDs are correct today, so this is a durability gap rather than a current defect.♻️ Proposed validator addition in rl_engine/testing/ws1_workload.py
def _validate_fixture_case_refs( fixtures: Mapping[str, Any], cases: Sequence[Mapping[str, Any]] ) -> None: known = {c["case_id"] for c in cases} for name in ( "short_full_model_fixture", "long_full_model_fixture", "representative_full_model_fixture", ): fixture = fixtures.get(name) if not fixture: continue for case_id in fixture.get("candidate_case_ids", []): if case_id not in known: raise WorkloadError( f"{name} references unknown case_id {case_id!r}" )Call it from
validate_manifestafter_validate_representative_cases.Also applies to: 411-414, 427-430
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/testing/ws1_manifest.json` around lines 367 - 370, Update validation around validate_manifest and _validate_representative_cases to cross-check every candidate_case_ids entry in short_full_model_fixture, long_full_model_fixture, and representative_full_model_fixture against the case_id values in representative_cases. Add a focused helper such as _validate_fixture_case_refs that skips absent fixtures and raises WorkloadError identifying the fixture and unknown ID, then invoke it after representative-case validation.scripts/ws1_reference.py (1)
21-36: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the unused
_ensure_repo_on_pathhelper.
_load_workload_moduleloads the workload by file path, and both modules use only standard-library imports. Remove the helper and its call.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/ws1_reference.py` around lines 21 - 36, Remove the unused _ensure_repo_on_path helper and delete its invocation from the script; leave _load_workload_module unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/contributing/testing.md`:
- Around line 9-11: Update the quick-start check_operator command in the testing
guide to include the small-shape arguments --batch 1, --seq 2, and --vocab 17,
matching the corresponding smoke command in gtest-usage.md while preserving the
existing operator, candidate, device, and dtype options.
In `@docs/design/ws1-c2-268-workload-plan.md`:
- Around line 3-4: Remove trailing double spaces used for Markdown hard breaks
in docs/design/ws1-c2-268-workload-plan.md lines 3-4 and line 174, replacing
them with blank lines or <br> as appropriate; also remove the trailing spaces on
the **Issue:** line in docs/design/ws1-c2-268-closeout-evidence.md lines 3-4.
Ensure both documents pass the trailing-whitespace hook while preserving
intended rendering.
- Around line 141-152: Correct the workload API block to match
rl_engine/testing/ws1_workload.py: replace build_logical_batch(workload_id) with
build_logical_batch(manifest=None, *, cell_id=None, sample_ids=None), rename
matrix_cells() to matrix_cell_ids(), and replace get_cell(cell_id) with
get_matrix_cell(manifest, cell_id). Leave the other API entries unchanged.
In `@rl_engine/kernels/gtest/op_checks.py`:
- Around line 161-166: Update the dtype comparison in
validate_backend_provenance to normalize both values with the shared normalizer
from tolerance.py, comparing _dtype_name(case.dtype) against
_dtype_name(candidate.provenance.execution_dtype). Keep the existing mismatch
error and validation flow unchanged.
In `@rl_engine/testing/ws1_workload.py`:
- Around line 765-775: Run Black on rl_engine/testing/ws1_workload.py#L765-L775
and scripts/ws1_reference.py#L88-L104, and run both Black and isort on
tests/test_ws1_workload.py#L8-L19; commit the resulting formatter changes, or
run pre-commit run --all-files to apply all required hooks.
- Around line 304-321: Update the weight_snapshot required-key list in the
validator to include weight_files_total_size_bytes before it is read for the
shard-size comparison, ensuring omissions raise WorkloadError rather than
KeyError. Also apply the same guarded-access pattern in _validate_primary_matrix
for N, sample_ids, and chunk, and in _validate_fixtures for padding, packing,
and primary_seq_len; use a small _require helper if appropriate.
In `@tests/test_tolerance_contract.py`:
- Around line 471-483: Update test_clip_interval_endpoints_count_as_inside to
derive lo and hi from float32 torch.exp values, then construct dlogp from the
corresponding exact log bounds so the implementation’s ratio0 values equal the
interval endpoints. Keep the inclusive-endpoint assertion that agg.clipfrac0 is
0.0.
In `@tests/test_ws1_workload.py`:
- Around line 465-482: Add a finite timeout to the subprocess.run call in
test_ws1_reference_cli_emits_identity, preserving the existing command arguments
and result assertions while ensuring a blocked CLI fails promptly.
---
Nitpick comments:
In `@rl_engine/kernels/gtest/op_checks.py`:
- Around line 244-266: The gradient comparison at
rl_engine/kernels/gtest/op_checks.py lines 244-266 and the forward comparison at
lines 315-331 each resolve tolerance twice; update both sites to call
resolve_tolerance once when the contract has a judgments block, retaining its
returned spec for atol, rtol, comparison_lhs_role, and comparison_rhs_role, and
use _resolve_tolerance only when judgments is absent. Apply
judgment="gradient_accuracy" at the anchor and judgment="forward_accuracy" at
the sibling, removing the separate gradient_spec and forward_spec calls.
In `@rl_engine/kernels/gtest/tolerance_contract.json`:
- Around line 217-224: The approx_kl0 thresholds in the contract are too loose
to detect drift before max_abs_dlogp, especially for BF16 and FP32. Update the
by_execution_dtype thresholds for approx_kl0 to meaningful values near the
square of each max_abs_dlogp scale (approximately 1.0e-3 for BF16 and 1.0e-10
for FP32), or document the deliberate rationale for retaining the current
thresholds.
- Around line 127-158: Review the gradient_accuracy thresholds for elementwise,
reduction, logprob, and attention against measured backward-pass errors rather
than copying forward_accuracy values. Update the cells with calibrated atol/rtol
values, or explicitly mark the current thresholds as provisional using the
contract’s existing status/schema conventions so they can be revised when
evidence is available.
In `@rl_engine/kernels/gtest/tolerance.py`:
- Around line 260-262: Update the validation in the backend profile resolution
flow before the `profile_contract` lookup to verify that
`provenance.backend_profile` exists in
`contract["policy"]["backend_profile_contracts"]`; raise `ContractResolveError`
for a declared profile without a contract, preserving the existing error
behavior for unknown profiles.
- Around line 936-964: Update the dtype normalization logic in the shown
torch-dtype branch to recognize all supported PyTorch FP8 variants, including
float8_e4m3fn, float8_e5m2, float8_e4m3fnuz, and float8_e5m2fnuz, using both
qualified and bare names. Return "float8" for each variant so they reach the
documented FP8 out-of-scope handling instead of raising unsupported dtype.
In `@rl_engine/testing/__init__.py`:
- Around line 16-48: Update the testing package exports in __all__ to include
apply_padding and restore_logical_order_from_padded, matching the existing
exports for apply_packing, apply_chunking, and restore_logical_order. Keep the
current imports and export ordering consistent with the corresponding symbols
from ws1_workload.
In `@rl_engine/testing/ws1_manifest.json`:
- Around line 367-370: Update validation around validate_manifest and
_validate_representative_cases to cross-check every candidate_case_ids entry in
short_full_model_fixture, long_full_model_fixture, and
representative_full_model_fixture against the case_id values in
representative_cases. Add a focused helper such as _validate_fixture_case_refs
that skips absent fixtures and raises WorkloadError identifying the fixture and
unknown ID, then invoke it after representative-case validation.
In `@rl_engine/testing/ws1_workload.py`:
- Around line 973-981: Update _manifest_identity_payload to include every
top-level manifest field except fixture_identity_sha256, rather than filtering
through _REQUIRED_TOP_LEVEL, so future numerics-affecting sections are covered
by manifest_identity_hash and validate_manifest. Regenerate
fixture_identity_sha256 in the manifest to match the updated digest.
In `@scripts/ws1_reference.py`:
- Around line 21-36: Remove the unused _ensure_repo_on_path helper and delete
its invocation from the script; leave _load_workload_module unchanged.
In `@tests/test_op_checks.py`:
- Around line 251-280: Extend
test_ws1_report_checks_observed_output_dtype_against_provenance with a gold_fn
case that returns bfloat16 where provenance.reference_dtype is float32. Run the
operator suite using this gold function and assert ContractResolveError with
match="gold output dtype", covering the gold reference_dtype validation branch.
- Around line 191-221: Add a happy-path test alongside
test_ws1_report_persists_roles_and_backend_provenance using
backend_profile="triton_cuda_bf16", requested_backend="triton",
actual_backend="triton", and backend="triton". Run it through run_operator_suite
with the existing logp case, then assert the same judgment, role fields, and
serialized backend provenance expectations, including absence of a baseline
output.
In `@tests/test_ws1_workload.py`:
- Around line 264-273: Remove the unused inverse list and its construction from
the restoration logic near the restored_samples loop, leaving the existing
list(perm).index(old_i) mapping behavior unchanged.
- Around line 403-416: Replace source-text grepping in
test_declared_candidates_resolve_to_real_operator_specs with an imported
operator registry and assert that all manifest operator_spec_map values resolve
to registry keys. Replace algorithm_source path:line parsing with a stable
path:symbol reference, then assert the referenced symbol is declared in the
target file rather than validating only a line range; update the manifest
entries accordingly.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: d0258e01-442b-4c49-8741-b6b5495f01b2
📒 Files selected for processing (15)
docs/contributing/gtest-usage.mddocs/contributing/testing.mddocs/design/ws1-c2-268-closeout-evidence.mddocs/design/ws1-c2-268-workload-plan.mdrl_engine/kernels/gtest/__init__.pyrl_engine/kernels/gtest/op_checks.pyrl_engine/kernels/gtest/tolerance.pyrl_engine/kernels/gtest/tolerance_contract.jsonrl_engine/testing/__init__.pyrl_engine/testing/ws1_manifest.jsonrl_engine/testing/ws1_workload.pyscripts/ws1_reference.pytests/test_op_checks.pytests/test_tolerance_contract.pytests/test_ws1_workload.py
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
rl_engine/testing/ws1_workload.py (3)
496-517: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winGuard
padding["modes"],packing["status"], and the fixture lookups.Lines 496, 501, 508, and 517 read keys directly after the enclosing object passed
_require. A manifest that omitsmodes,status, or one of the three fixture entries raisesKeyErrorinstead ofWorkloadError. Use_requirefor these reads to keep the validator fail-closed.🐛 Proposed fix
- if "right" not in padding["modes"] or "left" not in padding["modes"]: + modes = _require(padding, "modes", context="fixtures.padding") + if "right" not in modes or "left" not in modes: raise WorkloadError("padding.modes must include left and right") @@ - if packing["status"] not in { + status = _require(packing, "status", context="fixtures.packing") + if status not in { @@ - fixture = fixtures[name] + fixture = _require(fixtures, name, context="fixtures")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/testing/ws1_workload.py` around lines 496 - 517, Update the validator around the padding and packing checks to retrieve padding["modes"], packing["status"], and each fixture named in the loop via _require, preserving the existing WorkloadError validation and fail-closed behavior when any key is missing.
324-333: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winApply
_requireto element-level field reads, not only to container reads. The new_requirehelper guards top-level and container lookups, but the validators still index fields inside list elements and nested objects directly. Each unguarded read raisesKeyErrorinstead ofWorkloadErroron a malformed manifest.load_manifestandscripts/ws1_reference.pycatch onlyWorkloadError, so these paths produce a traceback instead of the intended exit code.
rl_engine/testing/ws1_workload.py#L324-L333: guardsize_bytes,filename, andsha256on each shard, and reject shard entries that are not mappings, before summing sizes and callingweight_snapshot_hash.rl_engine/testing/ws1_workload.py#L438-L446: readcell_idandbatch_modethrough_requirewith contextprimary_matrix.cell.rl_engine/testing/ws1_workload.py#L496-L517: readpadding["modes"],packing["status"], and the three named fixture entries through_require.rl_engine/testing/ws1_workload.py#L645-L650: readshape["M"]andshape["mode"]defensively, matching thecase["shape"].get(key)pattern already used at line 706.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/testing/ws1_workload.py` around lines 324 - 333, Update rl_engine/testing/ws1_workload.py at lines 324-333, 438-446, 496-517, and 645-650: use _require for all specified nested and element-level field reads, reject non-mapping shard entries before accessing fields, and defensively read shape["M"] and shape["mode"] consistent with the existing case["shape"].get pattern. Ensure malformed manifests raise WorkloadError rather than KeyError; scripts/ws1_reference.py requires no direct change because it is corrected by these validator updates.
438-446: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winRoute cell field reads through
_require.Line 438 reads
c["cell_id"]and line 444 readscell["batch_mode"]directly. A cell that omits either key raisesKeyErrorinstead ofWorkloadError. The rest of this function now uses_require; apply it here too.🐛 Proposed fix
- cell_ids = [c["cell_id"] for c in cells] + cell_ids = [_require(c, "cell_id", context="primary_matrix.cell") for c in cells] if set(cell_ids) != set(_REQUIRED_MATRIX_CELLS): raise WorkloadError( f"primary_matrix.cells must be exactly {_REQUIRED_MATRIX_CELLS}, got {cell_ids}" ) for cell in cells: - mode = cell["batch_mode"] + mode = _require(cell, "batch_mode", context="primary_matrix.cell")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rl_engine/testing/ws1_workload.py` around lines 438 - 446, Update the cell validation in the surrounding function to retrieve both cell_id and batch_mode through the existing _require helper instead of direct dictionary indexing. Ensure missing fields consistently raise WorkloadError while preserving the current required-cell and batch-mode validation behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rl_engine/testing/ws1_workload.py`:
- Around line 645-650: Update the GEMM and attention validation in the profile
case checks to read shape sub-keys defensively via the established get-based
pattern used by _validate_fixture_case_bindings. Ensure missing M or mode values
are handled by the existing set comparisons and raise the intended WorkloadError
rather than KeyError.
- Around line 685-714: Guard the expected-shape lookup in the case-validation
loop so unknown fixture IDs or families produce WorkloadError rather than
KeyError. Update expected_shapes to derive its fixture keys from by_fixture_id,
and handle missing family entries explicitly while preserving existing mismatch
validation for supported combinations.
In `@scripts/ws1_candidate_evidence.py`:
- Around line 155-157: Update the case execution flow around run_case so each
seed is derived from manifest.seed and the case’s stable case_id rather than
enumerate(cases), ensuring filtering by --profile or --case-id does not alter
generated inputs. Store the derived seed in every case result, and add coverage
comparing a default run with an isolated case selection to verify identical
results for the same workload_id and case_id.
In `@tests/test_ws1_workload.py`:
- Around line 516-526: Make the candidate evidence CLI help assertion in
test_candidate_evidence_cli_help_is_available resilient to argparse line
wrapping by pinning the subprocess environment’s COLUMNS value, or assert on a
shorter stable help-text token. Add the os import if using the environment
override, while preserving the existing command invocation and return-code
checks.
---
Outside diff comments:
In `@rl_engine/testing/ws1_workload.py`:
- Around line 496-517: Update the validator around the padding and packing
checks to retrieve padding["modes"], packing["status"], and each fixture named
in the loop via _require, preserving the existing WorkloadError validation and
fail-closed behavior when any key is missing.
- Around line 324-333: Update rl_engine/testing/ws1_workload.py at lines
324-333, 438-446, 496-517, and 645-650: use _require for all specified nested
and element-level field reads, reject non-mapping shard entries before accessing
fields, and defensively read shape["M"] and shape["mode"] consistent with the
existing case["shape"].get pattern. Ensure malformed manifests raise
WorkloadError rather than KeyError; scripts/ws1_reference.py requires no direct
change because it is corrected by these validator updates.
- Around line 438-446: Update the cell validation in the surrounding function to
retrieve both cell_id and batch_mode through the existing _require helper
instead of direct dictionary indexing. Ensure missing fields consistently raise
WorkloadError while preserving the current required-cell and batch-mode
validation behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: c3a98db7-3462-4cc5-b5be-f018a5cb930e
📒 Files selected for processing (15)
docs/contributing/testing.mddocs/design/ws1-c2-268-closeout-evidence.mddocs/design/ws1-c2-268-workload-plan.mdrl_engine/kernels/gtest/op_checks.pyrl_engine/kernels/gtest/tolerance.pyrl_engine/kernels/gtest/tolerance_contract.jsonrl_engine/testing/__init__.pyrl_engine/testing/ws1_manifest.jsonrl_engine/testing/ws1_workload.pyscripts/ws1_candidate_evidence.pyscripts/ws1_reference.pytests/test_op_checks.pytests/test_tolerance_contract.pytests/test_ws1_candidate_evidence.pytests/test_ws1_workload.py
🚧 Files skipped from review as they are similar to previous changes (8)
- docs/contributing/testing.md
- scripts/ws1_reference.py
- rl_engine/testing/init.py
- rl_engine/kernels/gtest/op_checks.py
- rl_engine/testing/ws1_manifest.json
- docs/design/ws1-c2-268-workload-plan.md
- tests/test_tolerance_contract.py
- rl_engine/kernels/gtest/tolerance_contract.json
| gemm_m = {int(c["shape"]["M"]) for c in profile_cases if c["family"] == "gemm"} | ||
| if len(gemm_m) < 2: | ||
| raise WorkloadError(f"profile {profile} GEMM cases require multiple M values") | ||
| attn_modes = {c["shape"]["mode"] for c in profile_cases if c["family"] == "attention"} | ||
| if attn_modes != {"prefill", "decode"}: | ||
| raise WorkloadError(f"profile {profile} attention cases require prefill+decode") |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Guard the shape sub-keys.
The required-key loop guarantees shape exists, but not shape["M"] or shape["mode"]. A GEMM case without M, or an attention case without mode, raises KeyError instead of WorkloadError. _validate_fixture_case_bindings at line 706 already uses case["shape"].get(key); use the same defensive read here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rl_engine/testing/ws1_workload.py` around lines 645 - 650, Update the GEMM
and attention validation in the profile case checks to read shape sub-keys
defensively via the established get-based pattern used by
_validate_fixture_case_bindings. Ensure missing M or mode values are handled by
the existing set comparisons and raise the intended WorkloadError rather than
KeyError.
| expected_shapes = { | ||
| "short_full_model_seq8": { | ||
| "gemm": {"M": int(short["seq_len"])}, | ||
| "logprob": {"B": 1, "T": int(short["seq_len"]) - int(short["prompt_len"])}, | ||
| }, | ||
| "long_full_model_seq32": { | ||
| "attention": {"B": 1, "Sq": 1, "Skv": int(long["seq_len"]), "mode": "decode"} | ||
| }, | ||
| "rep_full_model_seq16": { | ||
| "gemm": {"M": primary_total_tokens}, | ||
| "attention": { | ||
| "B": len(fixtures["samples"]), | ||
| "Sq": primary_max_seq, | ||
| "Skv": primary_max_seq, | ||
| "mode": "prefill", | ||
| }, | ||
| }, | ||
| } | ||
| for case in cases: | ||
| required = expected_shapes[case["fixture_id"]][case["family"]] | ||
| mismatched = { | ||
| key: (case["shape"].get(key), value) | ||
| for key, value in required.items() | ||
| if case["shape"].get(key) != value | ||
| } | ||
| if mismatched: | ||
| raise WorkloadError( | ||
| f"case {case['case_id']} shape does not derive from fixture " | ||
| f"{case['fixture_id']}: {mismatched}" | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Two unguarded lookups on a hardcoded shape table.
Line 704 indexes expected_shapes[case["fixture_id"]][case["family"]] without guards, and the loop runs over every case.
- The outer key set is hardcoded to
short_full_model_seq8,long_full_model_seq32, andrep_full_model_seq16. Theby_fixture_idmap above derives its keys from the manifest, but this table does not. If a manifest renames afixture_id, this raisesKeyErrorinstead ofWorkloadError. - The inner key set is partial.
short_full_model_seq8declares onlygemmandlogprob,long_full_model_seq32onlyattention, andrep_full_model_seq16onlygemmandattention._validate_representative_casespermits up to three cases per family per profile. A manifest that adds alogprobcase on the long fixture therefore crashes here instead of validating.
Callers catch only WorkloadError, so both paths surface as a traceback.
🐛 Proposed fix
for case in cases:
- required = expected_shapes[case["fixture_id"]][case["family"]]
+ fixture_shapes = expected_shapes.get(case["fixture_id"])
+ if fixture_shapes is None:
+ raise WorkloadError(
+ f"case {case['case_id']} pins unknown fixture_id {case['fixture_id']!r}"
+ )
+ required = fixture_shapes.get(case["family"])
+ if required is None:
+ continueDeriving expected_shapes keys from by_fixture_id instead of literals would also remove the duplicate pin.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| expected_shapes = { | |
| "short_full_model_seq8": { | |
| "gemm": {"M": int(short["seq_len"])}, | |
| "logprob": {"B": 1, "T": int(short["seq_len"]) - int(short["prompt_len"])}, | |
| }, | |
| "long_full_model_seq32": { | |
| "attention": {"B": 1, "Sq": 1, "Skv": int(long["seq_len"]), "mode": "decode"} | |
| }, | |
| "rep_full_model_seq16": { | |
| "gemm": {"M": primary_total_tokens}, | |
| "attention": { | |
| "B": len(fixtures["samples"]), | |
| "Sq": primary_max_seq, | |
| "Skv": primary_max_seq, | |
| "mode": "prefill", | |
| }, | |
| }, | |
| } | |
| for case in cases: | |
| required = expected_shapes[case["fixture_id"]][case["family"]] | |
| mismatched = { | |
| key: (case["shape"].get(key), value) | |
| for key, value in required.items() | |
| if case["shape"].get(key) != value | |
| } | |
| if mismatched: | |
| raise WorkloadError( | |
| f"case {case['case_id']} shape does not derive from fixture " | |
| f"{case['fixture_id']}: {mismatched}" | |
| ) | |
| expected_shapes = { | |
| "short_full_model_seq8": { | |
| "gemm": {"M": int(short["seq_len"])}, | |
| "logprob": {"B": 1, "T": int(short["seq_len"]) - int(short["prompt_len"])}, | |
| }, | |
| "long_full_model_seq32": { | |
| "attention": {"B": 1, "Sq": 1, "Skv": int(long["seq_len"]), "mode": "decode"} | |
| }, | |
| "rep_full_model_seq16": { | |
| "gemm": {"M": primary_total_tokens}, | |
| "attention": { | |
| "B": len(fixtures["samples"]), | |
| "Sq": primary_max_seq, | |
| "Skv": primary_max_seq, | |
| "mode": "prefill", | |
| }, | |
| }, | |
| } | |
| for case in cases: | |
| fixture_shapes = expected_shapes.get(case["fixture_id"]) | |
| if fixture_shapes is None: | |
| raise WorkloadError( | |
| f"case {case['case_id']} pins unknown fixture_id {case['fixture_id']!r}" | |
| ) | |
| required = fixture_shapes.get(case["family"]) | |
| if required is None: | |
| continue | |
| mismatched = { | |
| key: (case["shape"].get(key), value) | |
| for key, value in required.items() | |
| if case["shape"].get(key) != value | |
| } | |
| if mismatched: | |
| raise WorkloadError( | |
| f"case {case['case_id']} shape does not derive from fixture " | |
| f"{case['fixture_id']}: {mismatched}" | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rl_engine/testing/ws1_workload.py` around lines 685 - 714, Guard the
expected-shape lookup in the case-validation loop so unknown fixture IDs or
families produce WorkloadError rather than KeyError. Update expected_shapes to
derive its fixture keys from by_fixture_id, and handle missing family entries
explicitly while preserving existing mismatch validation for supported
combinations.
| results = [ | ||
| run_case(case, seed=manifest.seed + i, device=device) | ||
| for i, case in enumerate(cases) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Derive each case seed from stable case identity.
Line 156 derives the seed from the filtered cases index. --profile and --case-id change that index. The same workload_id and case_id can then execute different random tensors.
Derive the seed from manifest.seed and case_id. Record that seed in each case result. Add coverage that compares a default run with an isolated case selection.
Proposed direction
+import hashlib
+
+def _case_seed(workload_id: str, manifest_seed: int, case_id: str) -> int:
+ material = f"{workload_id}:{manifest_seed}:{case_id}".encode("utf-8")
+ return int.from_bytes(hashlib.sha256(material).digest()[:8], "big")
+
- run_case(case, seed=manifest.seed + i, device=device)
- for i, case in enumerate(cases)
+ run_case(
+ case,
+ seed=_case_seed(manifest.workload_id, manifest.seed, case["case_id"]),
+ device=device,
+ )
+ for case in cases🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/ws1_candidate_evidence.py` around lines 155 - 157, Update the case
execution flow around run_case so each seed is derived from manifest.seed and
the case’s stable case_id rather than enumerate(cases), ensuring filtering by
--profile or --case-id does not alter generated inputs. Store the derived seed
in every case result, and add coverage comparing a default run with an isolated
case selection to verify identical results for the same workload_id and case_id.
| def test_candidate_evidence_cli_help_is_available(): | ||
| proc = subprocess.run( | ||
| [sys.executable, str(CANDIDATE_EVIDENCE_SCRIPT), "--help"], | ||
| check=False, | ||
| capture_output=True, | ||
| text=True, | ||
| cwd=str(REPO_ROOT), | ||
| timeout=60, | ||
| ) | ||
| assert proc.returncode == 0, proc.stderr | ||
| assert "representative candidates on a real GPU" in proc.stdout |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
The help-text assertion can fail because argparse wraps the description.
argparse wraps the description to the terminal width reported by shutil.get_terminal_size(), which honors the COLUMNS environment variable. If CI sets a narrow width, the 41-character phrase at line 526 splits across lines and the substring match fails. Pin the width, or assert on a shorter token.
🔧 Proposed fix
+ env = {**os.environ, "COLUMNS": "200"}
proc = subprocess.run(
[sys.executable, str(CANDIDATE_EVIDENCE_SCRIPT), "--help"],
check=False,
capture_output=True,
text=True,
cwd=str(REPO_ROOT),
timeout=60,
+ env=env,
)Add import os at the top of the module.
The Ruff S603 and ast-grep command-injection hints on this call are false positives. Every argument is a literal or a path derived from __file__.
🧰 Tools
🪛 ast-grep (0.45.1)
[error] 516-523: Command coming from incoming request
Context: subprocess.run(
[sys.executable, str(CANDIDATE_EVIDENCE_SCRIPT), "--help"],
check=False,
capture_output=True,
text=True,
cwd=str(REPO_ROOT),
timeout=60,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🪛 Ruff (0.16.1)
[error] 517-517: subprocess call: check for execution of untrusted input
(S603)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_ws1_workload.py` around lines 516 - 526, Make the candidate
evidence CLI help assertion in test_candidate_evidence_cli_help_is_available
resilient to argparse line wrapping by pinning the subprocess environment’s
COLUMNS value, or assert on a shorter stable help-text token. Add the os import
if using the environment override, while preserving the existing command
invocation and return-code checks.
Source: Linters/SAST tools
1 participant
Summary
Lands WS1 Closeout C2 (#268): the fixed full Qwen3-8B Dense logical workload identity used by later gates (C3–C11).
ws1-qwen3-8b-dense-primary-v3): official config fingerprint, weight shard content hash, seed, varlen token fixtures, left/right pad, packing, 2×2 Batch/Chunk matrix, clip_interval, RNG policy, dual profiles (cuda_bf16/triton_cuda_bf16), representativecase_ids.(sample_id, token_position)identity; pad/pack/chunk restore; B1singleton_aggregatevs BN same multiset.scripts/ws1_reference.pyemits workload_id / seed / dtype / fixture digests.actual_*= registry-resolved candidate paths; runtime GPU observation owned by C3/C8/C10/C11. Triton missing candidates aremissing_required(red), not N/A.Depends on / stacks with: C1 PR #290 (this branch is based on
feat/ws1-c1-tolerance-contract-267). Merge #290 first or land as a stack.Does not claim full WS1 EXIT (#266).
Acceptance (#268)
See
docs/design/ws1-c2-268-closeout-evidence.mdfor the full AC map.Test plan
python -m pytest tests/test_ws1_workload.py -q→ 33 passedpython -m pytest tests/test_tolerance_contract.py -q→ C1 still greenpython scripts/ws1_reference.py --dtype bf16 --cell-id BN/full --emit-json -Residual (not C2)
embedding,lm_head,logprob(tracked in manifest)Closes #268
Summary by CodeRabbit
New Features
Documentation
Tests