Skip to content

update(ssm): add BF16 timestep precision regression coverage - #2209

Merged
inureyes merged 5 commits into
lablup:mainfrom
Jaeyeong-CHOI:fix/2142-ssm-bf16-timestep
Oct 12, 2026
Merged

inureyes merged 5 commits into
lablup:mainfrom
Jaeyeong-CHOI:fix/2142-ssm-bf16-timestep

Conversation

@Jaeyeong-CHOI

@Jaeyeong-CHOI Jaeyeong-CHOI commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Current main contains the production correction through PR #2270: the shared compiled SSM preprocessing casts both dt and dt_bias to float32 before addition. This PR preserves the contributor's original diagnosis and commit history while adding the isolated regression coverage that remains absent from main.

Root cause and coverage

  • The defect was in compute_dt_compiled, before backend dispatch, rather than in the CUDA recurrence body or the float32 reference. MLX selected BF16 for float32 + bfloat16, so the processed timestep was rounded before the float32 state update.
  • The regression uses zero state, x = B = 1, C = D = 0, and exactly representable inputs, making every next-state element equal host-f32 softplus(-2.5) without depending on a second GPU oracle.
  • Float32 and BF16 activations run across BF16-to-float32-to-BF16 bias transitions. Existing tolerances are unchanged.
  • The test is split into a child module to keep the parent parity file below the 500-line workflow limit.

Validation

  • Focused CUDA build at candidate a9696bd8cb3d0550d312ce5b9c561c544f9886e4: passed.
  • Focused GB10 runtime under gpu-lock: 6 passed, 0 failed, 0 ignored; all six SSM parity cases executed in 9.65 seconds.
  • Rust formatting and diff whitespace checks: passed.
  • Full CUDA workspace gate under gpu-lock: 12,247 passed, 0 failed, and 389 ignored across 163 suites, including doc tests.
  • Broad CUDA Clippy with all workspace targets and -D warnings: passed.

PR #2270's preserved GB10 evidence reduced Granite state normalized RMS/max from 2.532402801e-3 / 4.127457397e-2 to 2.838613609e-8 / 1.701009653e-6 without changing tolerances, and its deterministic Granite fused-kernel versus graph comparison matched all 64 generated token IDs. This final PR does not claim a second production correction.

Bilingual technical reports are included under TECHNICAL_REPORTS/.

Closes #2142

Jaeyeong-CHOI and others added 3 commits October 7, 2026 17:41
Current main independently carries the float32 timestep promotion from PR lablup#2270. Resolve the overlapping wrapper change to the mainline form while retaining PR lablup#2209's focused mixed-dtype regression coverage.

Refs lablup#2142
Keep the focused mixed-bias precision regression in a child module so the parity suite remains below the 500-line file limit. The six CUDA cases and their tolerances are unchanged, and the focused test target compiles with the CUDA feature.

Refs lablup#2142
@inureyes inureyes changed the title fix(ssm): preserve float32 timestep computation with BF16 bias test(ssm): pin float32 timestep with BF16 bias Oct 12, 2026
@inureyes inureyes changed the title test(ssm): pin float32 timestep with BF16 bias update(ssm): add BF16 timestep precision regression coverage Oct 12, 2026
Record the independent float32 timestep oracle and the completed GB10 CUDA workspace, focused parity, contract, and Clippy checks in English and Korean reports. Distinguish the regression coverage from the production correction already merged in PR lablup#2270.
Include the shared CI dependency fix from main so the BF16 timestep regression coverage passes cargo-deny without suppressing advisories. The SSM test implementation is unchanged.

Refs lablup#2142, lablup#2278.
@inureyes inureyes added the status:done Completed label Oct 12, 2026
@inureyes
inureyes merged commit 84554b5 into lablup:main Oct 12, 2026
27 checks passed
inureyes added a commit that referenced this pull request Oct 12, 2026
Include the merged SSM precision regression coverage before the final contract and A/B script validation for the portable output directory fix.

Refs #2141, #2209.
inureyes added a commit that referenced this pull request Oct 12, 2026
Merge origin/main at 2cb4e20 after the portable A/B output directory fix landed.

This incorporates the #2209 and #2276 prerequisite changes while leaving the split-MTP implementation and regression-test paths unchanged.

Validation:
- configured repository contracts
- cargo deny check
- cargo fmt --all -- --check
- staged and unstaged whitespace checks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:done Completed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(cuda): ssm_update_kernel bf16 parity tests fail on GB10

2 participants