[samggreenberg]

Closes #4938

Why

numpy 2.5.2 and later bundle OpenBLAS 0.3.34. That release added a Zen 4 tuning block (OpenMathLib/OpenBLAS#5868) which fires on an AMD host with a 1 MB L2, an L3 that is a multiple of 32 MB, and AVX-512. It rewrites the GEMM blocking of whatever core table is active, including one forced by OPENBLAS_CORETYPE (OpenMathLib/OpenBLAS#6021).

Under the Haswell pin that app.py, gridenv.sh and the cell runner set (#4481), this has two effects on the L40S nodes' EPYC 9534:

• float64 GEMM with K ≥ 320, and float32 or float64 SVD, overrun the Haswell kernel's stack and segfault;
• float32 GEMM and syrk return different bits than on every other CPU.

Upstream fixed it in OpenMathLib/OpenBLAS#6027 (merged 2026-09-13). The fix is in no release, and the newest scipy-openblas wheels (0.3.34.237) predate it.

The block requires an L2 of exactly 1024 KB, and the Haswell table reads the L2 size nowhere else. So OPENBLAS_L2_SIZE=2048 switches the block off, which is what upstream's fix does for a forced Haswell core.

The GRID is not exposed today: the venv runs numpy 2.2.6 (OpenBLAS 0.3.29) and scipy's 0.3.31.dev. A fresh install resolves numpy 2.5.4, so a server on a Zen 4/5 machine would crash.

What changes

• app.py, gridenv.sh, _expcommon.cpu_kernel_pins() (the cell runner) and pile/fhibe_supply.py default OPENBLAS_L2_SIZE=2048 whenever the core is Haswell.
  • An exported value wins; an empty one lets OpenBLAS detect the cache.
  • Under another core it is not set, because older cores' blocking follows the L2 size.
• Tests: the app prelude, the cell runner (only with Haswell), and gridenv.sh, checked by sourcing a copy (default, another core, exported value wins).
• docs/DEPLOYMENT.md gets a row for the variable; CHANGELOG.md gets a Fixed entry.
• Probes and report: docs/experiments/2026-10-15-openblas-zen4-4938/.

Results

• Probes: numpy 2.2.6, 2.4.6, 2.5.1 and 2.5.4 × L2 setting × 24 BLAS/LAPACK ops, on 6 nodes covering 5 CPU models.
  • Without the setting, only numpy 2.5.4 on the EPYC 9534 misbehaves: 6 ops crash and 11 differ.
  • With it, every op agrees on every node.
  • The setting changes no result on numpy 2.2.6, 2.4.6 or 2.5.1, or in scipy's OpenBLAS, on any node, so no GRID baseline moves.
• The app's own suite on the EPYC 9534 with numpy 2.5.4:
  • Without the setting: 11 segfaults, all in the np.linalg.svd of vtscore/media/structural_tiles.py:fit_projection (fitting a document collection's Stage-1 projection). pytest gave up after 10,720 tests.
  • With it: 0 segfaults, and the suite completed (16,646 passed).
  • The remaining 26 failures occur in both runs and come from the numpy 2.5 overlay: 11 are numba refusing numpy 2.5, and 15 are audio tests. The 18 that are not UMAP pass by ID on the venv's numpy 2.2.6.

Test plan

• test_cpu_kernel_pins.py, test_torch_threads_default.py, test_experiments_common.py, test_job_records.py and test_eval_app_sync.py: 129 passed on rack4n06 (Silver 4114) and rack7n11 (EPYC 7713).
• The suite on rack4n01 (EPYC 9534), numpy 2.5.4, bug arm vs fix arm, as above.
• The suite's 7 doc/index gates, deptry and ruff. pyright on the changed files: only the 2 app.py errors that dev already has.
• GRID suite (suite-grid, advisory).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q8KvKjgBL7pTwLPQT1g77M