cache: don't let shared-cache pruning kill concurrent builds - #14
Merged
Conversation
The prune sweep unlinked every ircache_*.tmp??????? unconditionally, on the assumption that a temp file in the cache dir is a leftover from a crashed compiler. That holds for a private cache but not for one shared between concurrent compilers: it deletes temp files other processes are still writing, so their rename() fails with ENOENT and cacheObjectFile() calls fatal(), killing an otherwise healthy compilation. Only remove temp files older than a grace period, so genuine orphans are still cleaned up but live ones survive.
Storing is a pure optimization - the object file has already been written to its output location and nothing in the compilation reads it back - yet every failure in cacheObjectFile() called fatal(), so a cache-write hiccup killed the whole compilation. Warn and skip caching instead; the worst consequence is a cache miss next time. recoverObjectFile() keeps fatal(): it removes the output file before copying from the cache, so a failure there really does lose the object.
ehudt-weka
approved these changes
Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the farm-wide build failures tracked in WEKAPP-651245:
https://wekaio.atlassian.net/browse/WEKAPP-651245
Root cause
CachePruner.doPrune()unlinked every temp file in the cache directoryunconditionally:
No age check, no ownership check — contrast
pruneForExpiryright below it,which does gate on
timeLastAccessed. The implicit assumption is that a tempfile in the cache dir is a leftover from a crashed compiler, so it is garbage.
That holds for a private cache. It does not hold for a cache shared between
concurrent compilers. Temp files are created inside the cache directory
(
storeCacheFileName+".tmp%%%%%%%") because an atomicrename()requiressource and destination on the same filesystem; their only privacy is the random
7-char suffix, which prevents name collisions but is matched exactly by
.tmp???????. So one compiler's prune pass deletes other compilers' in-flighttemp files, their
rename()fails withENOENT, andcacheObjectFile()callsfatal()— killing an otherwise healthy compilation.Evidence from the affected build farm
Weka shares one LDC cache across 22 build servers on a cluster filesystem.
ircache_prune_timestamphad a byte-identical mtime on three separatebuilders, i.e. one prune clock for the whole farm: every
-cache-prune-interval, whichever compiler process notices the staletimestamp sweeps temp files on behalf of every machine.
log, on an object that had already been written successfully.
flight; ~4% of sweeps landing on a temp implies an ~80ms temp lifetime, which
matches a
copy_file+renameof a ~34MB object.Changes
cache: don't delete in-flight temp files when pruning a shared cache—only remove temp files older than a grace period (1 hour), so genuine
orphans are still cleaned up but live ones survive.
cache: warn instead of aborting when storing into the cache fails—defense in depth. Storing is a pure optimization: the object file has
already been written to its output location by the caller and nothing in the
compilation reads it back, so a store failure means "cache miss next time",
not "build invalid".
recoverObjectFile()deliberately keepsfatal()— itremoves the output file before copying from the cache, so a failure there
really does lose the object.
Commit 1 fixes the known race; commit 2 makes any future cache-write failure a
slower build rather than an outage. They are independent if you only want one.
Testing
driver/cache_pruning.dcompiles standalone (ldc2 -o- -c); the module hasno LDC dependencies by design.
driver/cache.cppcompiles with the project's real build flags extractedfrom a configured ninja tree.
🤖 Generated with Claude Code