Windows COFF archive generation can silently fail and cache stale output

[gregcotten]

Direct static_library generation can exit 0 after failing to replace an existing Windows COFF .lib. With the generator cache enabled, the invocation can then store that old library under the requested pipeline's correct key. The writer failure also reproduces with caching disabled.

Reproducer

The standalone CPU reproducer and results use a generator computing output(x) = input(x) + offset. Each request runs in a fresh process with stable function names, output paths and arguments, targeting x86-64-windows-no_runtime.

Tested on Windows x64 with MSVC 19.42, SDK 10.0.22621.0 and the shared halide-bin==22.0.0.dev390 wheel. Relevant upstream source is unchanged at a95f4de3b2b9279830e9b2c90983461ab2a4d8a1.

1. Generate A (offset=1) and an unlocked B (offset=2) control; verify different archive hashes.
2. Restore A and open its .lib with CreateFileW(GENERIC_READ, FILE_SHARE_READ, OPEN_EXISTING), allowing reads but denying writes and deletion.
3. Generate B with -e c_header,static_library, HL_DEBUG_CODEGEN=1 and HL_CACHE_DIR pointing to an empty directory.
4. Release the handle and request B again using that cache.
5. Repeat the locked generation with HL_CACHE_DIR unset.

Observed results, using SHA-256 prefixes:

Operation	Exit	Cache message	Archive hash
A control	0	stores A	A: 705cdf8d44bd
B control	0	stores B	B: bedc7d16a66d
B cold compile while A is locked	0	stores B key 76cd390cb00b	A: 705cdf8d44bd
B after unlock, using that cache	0	hit for B key 76cd390cb00b	A: 705cdf8d44bd
B while A is locked, caching disabled	0	none	A: 705cdf8d44bd

Handles are released in finally. This exercises Halide's direct COFF writer; an archive assembled by MSBuild/lib.exe takes a different path. The condition follows the Windows output target, so it is not inherently limited to a Windows host. Cross-host reproduction remains unverified.

Expected behavior and fix

An archive output error should fail generation with a diagnostic identifying the destination. Failed generation must not publish a cache entry. After the obstruction is removed, generation should produce B's archive.

The COFF branch of create_static_library opens, writes, flushes and closes an ofstream without checking its state.

A local patch checks opening and final stream state and flushes explicitly before the writer's seeks. The latter is needed by another regression on the tested MSVC implementation: a same-process byte-range lock permits opening/truncation but blocks writing. Checking only opening and final state still falsely succeeds because a seek can lose a buffered write error. Explicit flushes make that error observable. Successful archives remain byte-identical, without adding an archive-sized memory buffer.

The patch passes a full shared Halide build and three affected correctness tests, independently and together with the restore fix. The combined build also passes both normal CMake cache integration tests. The original wheel fails the new regressions; the byte-range-lock regression also rejects the incomplete open/end-only fix. Patched macOS/Linux hosts, static Halide linkage and full CI remain unverified.

This producer defect is separate from the failed cache restoration in #9437 / #9438. The fixes can be reviewed independently: this one prevents failed generation from publishing wrong bytes; #9438 prevents failed restoration from reporting a hit. B's distinct key receiving stale output does not demonstrate a key collision.

Proposed fix and regression tests: #9440.