Skip to content

Match Lake's trace githash to the toolchain that built the Mathlib cache - #1

Merged
jwiegley merged 2 commits into
mainfrom
johnw/lean-githash
Sep 22, 2026
Merged

jwiegley merged 2 commits into
mainfrom
johnw/lean-githash

Conversation

@jwiegley

@jwiegley jwiegley commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Summary

Every consumer of this flake has been recompiling all of Mathlib on every build, because Lake's build traces are keyed on the compiler's githash and the two Leans involved disagree about it:

  • the Mathlib artifact cache is produced by the official leanprover/lean4:v4.30.0 release, whose lean --githash is d024af099ca4bf2c86f649261ebf59565dc8c622;
  • nixpkgs' lean4 4.30.0 is built from the same source but reports the literal tag, v4.30.0.

So every artifact lake exe cache get fetched failed the trace check, and lake build Mathlib in the deps derivation rebuilt 8,474 jobs from source: 4h19m of a 4h27m cold deps build on ubuntu-latest (this repo's own CI, 03abb3e), and the same hours on every nix build of the C++ ledger repository, which cannot cache a 10 GB output. The proof is in the artifacts: every .olean in the current deps output carries githash v4.30.0; in the pre-03abb3e91 output, 7,370 still carried d024af0… and only the 731 in the import closure had been rebuilt. That mismatch is also why the earlier derivation rebuilt the import closure at all, and why an added import then tried to rebuild inside the read-only store.

Fix: pin the release commit as leanGithash and export it as LEAN_GITHASH wherever lake runs (deps, oracle, dev shell). Lake honours it as the detected githash (Lake/Config/Env.lean), so lake build Mathlib becomes a replay.

Measured (nix build .#deps -L, no module compiled):

platform machine deps derivation
x86_64-linux 256-core builder 1m53s – 2m16s
aarch64-linux 10-core builder 2m09s – 4m05s
aarch64-darwin M3 Ultra ~6.5 min

One hash for every platform. Refreshing the hashes with full-tree checksums (119,869 files) showed that the only files that ever differed between platforms were the natively compiled products of lake exe cache: the .c.o.export objects under build/ir with their .hash/.trace records, and the Cache.* modules' own artifacts, whose traces embed the native facet. The oracle imports none of it, so the cleanup now removes it, and the normalized tree is byte-identical on x86_64-linux, aarch64-linux and aarch64-darwin (also reproduced across two different x86_64 machines). The per-kernel outputHash and the case-insensitive-filesystem hypothesis are gone. The scrub's greps are made tolerant of an empty match set, which stdenv's pipefail otherwise turns into a silent build failure.

Consumers. Lake keys its cached lakefile elaboration on the githash as well, so a downstream that runs the oracle in place with a lake reporting a different githash (the C++ ledger's check phase: this flake's lean on PATH, lake env lean --run in the store tree) fails with "permission denied" on .lake/config/0/lakefile.olean.lock. The second commit wraps the lean package's lake to set LEAN_GITHASH, verified by running the oracle driver from the store tree with only that toolchain on PATH and no variable in the environment.

Docs: AGENTS.md records leanGithash as the fifth pin that must move with the toolchain; the CI comment describes the measurement legs in terms of the single hash (x86_64-darwin remains the one platform not yet seen to reproduce it).

Test plan

  • nix build .#deps -L on x86_64-linux (two machines), aarch64-linux and aarch64-darwin: identical hash, all 8,474 Mathlib jobs replayed.
  • nix build .#oracle -L on aarch64-darwin builds against the new tree; the driver smoke test from ci.yml passes.
  • CI here (ubuntu-latest, macos-latest, plus the two measurement legs) should now take minutes instead of hours; the C++ repository's flake will be bumped in a follow-up PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XEq9CB5JypSjKRn8MjRg72

jwiegley and others added 2 commits September 21, 2026 23:58
Lake keys every build trace on the compiler's githash.  The Mathlib
artifact cache is produced by the official leanprover/lean4:v4.30.0
release, whose `lean --githash` is d024af099ca4bf2c86f649261ebf59565
dc8c622; nixpkgs' lean4 is built from the same source but reports the
literal tag, "v4.30.0".  So every artifact `lake exe cache get`
fetched failed Lake's trace check, and `lake build Mathlib` in the
deps derivation recompiled all of Mathlib: 8,474 jobs, four and a
quarter hours on a GitHub runner, on every run of every consumer,
because nothing downstream can cache a 10 GB output.  The same
mismatch is why the earlier deps derivation rebuilt the import
closure and why an added import then tried to rebuild inside the
read-only store.

Lake honours LEAN_GITHASH as the detected githash (Lake/Config/Env),
so the flake pins the release commit as `leanGithash` and exports it
wherever lake runs: in deps, in the oracle, and in the dev shell.
With matching traces `lake build Mathlib` is a replay.  Measured, the
whole deps derivation now takes about two minutes on x86_64-linux
and aarch64-linux builders and seven on an aarch64-darwin
workstation, with no module compiled.

Refreshing the hashes showed what actually differed between
platforms: exactly the natively compiled products of `lake exe cache`
-- the `.c.o.export` objects under build/ir with their .hash and
.trace records, and the `Cache.*` modules' own artifacts, whose
traces embed the native facet.  The oracle imports none of it, so
the cleanup removes it, and the normalized tree is byte-identical on
x86_64-linux, aarch64-linux and aarch64-darwin (119,869 files
compared, and reproduced across two x86_64 machines).  The deps
derivation therefore declares one hash for every platform instead of
one per kernel; the case-insensitive-filesystem hypothesis is
withdrawn.  The scrub's greps now tolerate an empty match set, which
stdenv's pipefail otherwise turns into a silent build failure.

AGENTS.md records leanGithash as the fifth pin that must move with
the toolchain, and the CI comment describes the measurement legs in
terms of the single hash.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XEq9CB5JypSjKRn8MjRg72
Lake keys its cached lakefile elaboration on the githash as well as
its build traces.  A downstream that runs the oracle in place with a
lake reporting a different githash -- the C++ ledger's check phase,
which puts this flake's `lean` output on PATH and runs `lake env lean
--run` inside the store tree -- judges the elaborated lakefile stale
and tries to rewrite it there: "permission denied" on
.lake/config/0/lakefile.olean.lock, and the bisimulation fails before
the oracle runs.  The `lean` package now wraps lake to set
LEAN_GITHASH to the pin the oracle was built with, so consumers need
know nothing about it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XEq9CB5JypSjKRn8MjRg72
@jwiegley
jwiegley merged commit 101c328 into main Sep 22, 2026
3 of 4 checks passed
@jwiegley
jwiegley deleted the johnw/lean-githash branch September 22, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant