Repository navigation
Match Lake's trace githash to the toolchain that built the Mathlib cache - #1
Merged
Merged
Conversation
Lake keys every build trace on the compiler's githash. The Mathlib artifact cache is produced by the official leanprover/lean4:v4.30.0 release, whose `lean --githash` is d024af099ca4bf2c86f649261ebf59565 dc8c622; nixpkgs' lean4 is built from the same source but reports the literal tag, "v4.30.0". So every artifact `lake exe cache get` fetched failed Lake's trace check, and `lake build Mathlib` in the deps derivation recompiled all of Mathlib: 8,474 jobs, four and a quarter hours on a GitHub runner, on every run of every consumer, because nothing downstream can cache a 10 GB output. The same mismatch is why the earlier deps derivation rebuilt the import closure and why an added import then tried to rebuild inside the read-only store. Lake honours LEAN_GITHASH as the detected githash (Lake/Config/Env), so the flake pins the release commit as `leanGithash` and exports it wherever lake runs: in deps, in the oracle, and in the dev shell. With matching traces `lake build Mathlib` is a replay. Measured, the whole deps derivation now takes about two minutes on x86_64-linux and aarch64-linux builders and seven on an aarch64-darwin workstation, with no module compiled. Refreshing the hashes showed what actually differed between platforms: exactly the natively compiled products of `lake exe cache` -- the `.c.o.export` objects under build/ir with their .hash and .trace records, and the `Cache.*` modules' own artifacts, whose traces embed the native facet. The oracle imports none of it, so the cleanup removes it, and the normalized tree is byte-identical on x86_64-linux, aarch64-linux and aarch64-darwin (119,869 files compared, and reproduced across two x86_64 machines). The deps derivation therefore declares one hash for every platform instead of one per kernel; the case-insensitive-filesystem hypothesis is withdrawn. The scrub's greps now tolerate an empty match set, which stdenv's pipefail otherwise turns into a silent build failure. AGENTS.md records leanGithash as the fifth pin that must move with the toolchain, and the CI comment describes the measurement legs in terms of the single hash. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XEq9CB5JypSjKRn8MjRg72
Lake keys its cached lakefile elaboration on the githash as well as its build traces. A downstream that runs the oracle in place with a lake reporting a different githash -- the C++ ledger's check phase, which puts this flake's `lean` output on PATH and runs `lake env lean --run` inside the store tree -- judges the elaborated lakefile stale and tries to rewrite it there: "permission denied" on .lake/config/0/lakefile.olean.lock, and the bisimulation fails before the oracle runs. The `lean` package now wraps lake to set LEAN_GITHASH to the pin the oracle was built with, so consumers need know nothing about it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XEq9CB5JypSjKRn8MjRg72
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every consumer of this flake has been recompiling all of Mathlib on every build, because Lake's build traces are keyed on the compiler's githash and the two Leans involved disagree about it:
leanprover/lean4:v4.30.0release, whoselean --githashisd024af099ca4bf2c86f649261ebf59565dc8c622;lean44.30.0 is built from the same source but reports the literal tag,v4.30.0.So every artifact
lake exe cache getfetched failed the trace check, andlake build Mathlibin thedepsderivation rebuilt 8,474 jobs from source: 4h19m of a 4h27m colddepsbuild onubuntu-latest(this repo's own CI, 03abb3e), and the same hours on everynix buildof the C++ ledger repository, which cannot cache a 10 GB output. The proof is in the artifacts: every.oleanin the currentdepsoutput carries githashv4.30.0; in the pre-03abb3e91 output, 7,370 still carriedd024af0…and only the 731 in the import closure had been rebuilt. That mismatch is also why the earlier derivation rebuilt the import closure at all, and why an added import then tried to rebuild inside the read-only store.Fix: pin the release commit as
leanGithashand export it asLEAN_GITHASHwherever lake runs (deps,oracle, dev shell). Lake honours it as the detected githash (Lake/Config/Env.lean), solake build Mathlibbecomes a replay.Measured (
nix build .#deps -L, no module compiled):One hash for every platform. Refreshing the hashes with full-tree checksums (119,869 files) showed that the only files that ever differed between platforms were the natively compiled products of
lake exe cache: the.c.o.exportobjects underbuild/irwith their.hash/.tracerecords, and theCache.*modules' own artifacts, whose traces embed the native facet. The oracle imports none of it, so the cleanup now removes it, and the normalized tree is byte-identical on x86_64-linux, aarch64-linux and aarch64-darwin (also reproduced across two different x86_64 machines). The per-kerneloutputHashand the case-insensitive-filesystem hypothesis are gone. The scrub's greps are made tolerant of an empty match set, which stdenv'spipefailotherwise turns into a silent build failure.Consumers. Lake keys its cached lakefile elaboration on the githash as well, so a downstream that runs the oracle in place with a lake reporting a different githash (the C++ ledger's check phase: this flake's
leanon PATH,lake env lean --runin the store tree) fails with "permission denied" on.lake/config/0/lakefile.olean.lock. The second commit wraps theleanpackage'slaketo setLEAN_GITHASH, verified by running the oracle driver from the store tree with only that toolchain on PATH and no variable in the environment.Docs:
AGENTS.mdrecordsleanGithashas the fifth pin that must move with the toolchain; the CI comment describes the measurement legs in terms of the single hash (x86_64-darwin remains the one platform not yet seen to reproduce it).Test plan
nix build .#deps -Lon x86_64-linux (two machines), aarch64-linux and aarch64-darwin: identical hash, all 8,474 Mathlib jobs replayed.nix build .#oracle -Lon aarch64-darwin builds against the new tree; the driver smoke test fromci.ymlpasses.🤖 Generated with Claude Code
https://claude.ai/code/session_01XEq9CB5JypSjKRn8MjRg72