ci(measure): a scorer or schema step whose inputs are unchanged takes its output from a score cache (#1237)

.github/workflows/measure.yml

@@ -64,6 +64,14 @@
 # probe-<probe>.txt and probe_score-<probe>.txt:
 #
 #   gh workflow run measure.yml -f ref=my-branch -f base=none -f nordic=true -f models="nordic-office-*" -f scorers=none -f probe="probe_a probe_b" -f probe_scorer="a_vs_ifc -"
+#
+# A scorer or schema step whose every input is as it was in an earlier run is
+# not run again: its output comes from the score cache (tools/ci/measure-cache.py;
+# its key, what is never cached and how it is saved are in docs/maintaining.md).
+# Exports, rvt-info, rvt-gltf, the witness exports, diagnostics and probes always
+# run. `score_cache=off` runs every step, as before.
+#
+#   gh workflow run measure.yml -f ref=my-branch -f base=main -f score_cache=off
 
 name: Measure
 
@@ -150,6 +158,10 @@ on:
         description: Only these steps on the models, tools/re scorers and info, gltf, schema, witness (e.g. "qtos_vs_ifc witness"; a tools/re script in no scorer list runs as an IFC scorer), none for none, empty for all; every model is still exported and hashed
         required: false
         default: ""
+      score_cache:
+        description: 'The score cache (tools/ci/measure-cache.py, docs/maintaining.md): empty for the shared one, off to run every step, or a name for a cache of its own that this run saves to from any ref (the shared one is read as well)'
+        required: false
+        default: ""
   # A change to the measurement itself measures its own PR against main.
   pull_request:
     paths:
@@ -317,11 +329,98 @@ jobs:
           python-version: "3.12"
       - name: Install IfcOpenShell
         run: python -m pip install --require-hashes -r tools/ci/requirements-ifc.txt
+      # The score cache (tools/ci/measure-cache.py, docs/maintaining.md): a
+      # scorer or schema step whose every input is as it was in an earlier run
+      # takes that run's output. Every run restores the newest cache of its
+      # shard, before the measured code builds; it is saved only where the
+      # Rust cache is (the workflow dispatched from main and the code measured
+      # main's, so a branch's build scripts can never write what a later main
+      # run reads), or under a name of its own that `score_cache` gives a run
+      # from any ref, which main's runs never read. Entries are named by what
+      # they were made from, so the sides and the flags share them. Each shard
+      # keeps a cache of its own, named by the shard whatever the shard is
+      # called, and ends its name with a `.` that no shard name holds, so one
+      # shard's prefix is never another's.
+      # The held-out models, BIMcollab's and the pairs once held out (spent-*)
+      # are never cached (RE-252).
+      - name: Score cache key
+        id: score-cache-key
+        shell: bash
+        env:
+          SCORE_CACHE: ${{ inputs.score_cache }}
+          SHARD: ${{ matrix.shard }}
+          SAVE_FROM_MAIN: ${{ github.ref == 'refs/heads/main' && (matrix.side == 'base' && (inputs.base == '' || inputs.base == 'main') || matrix.side != 'base' && inputs.ref == '') }}
+        run: |
+          enabled=true
+          save=false
+          main=measure-scores-v2-main-
+          case "$SCORE_CACHE" in
+            off)
+              enabled=false
+              ;;
+            "")
+              own=$main
+              [ "$SAVE_FROM_MAIN" = true ] && save=true
+              ;;
+            *)
+              if ! [[ "$SCORE_CACHE" =~ ^[A-Za-z0-9._]{1,40}$ ]]; then
+                echo "::error::score_cache is off, or a name of up to 40 letters, digits, dots and underscores"
+                exit 1
+              fi
+              own="measure-scores-v2-x-$SCORE_CACHE-"
+              save=true
+              ;;
+          esac
+          {
+            echo "enabled=$enabled"
+            echo "save=$save"
+            echo "prefix=$own$SHARD."
+            echo "restore<<END"
+            echo "$own$SHARD."
+            [ "$own" != "$main" ] && echo "$main$SHARD."
+            echo "END"
+          } >> "$GITHUB_OUTPUT"
+          if [ "$enabled" = true ]; then
+            echo "MEASURE_CACHE_DIR=$GITHUB_WORKSPACE/_score-cache" >> "$GITHUB_ENV"
+            echo "MEASURE_CACHE_WORK=$RUNNER_TEMP/score-cache-work" >> "$GITHUB_ENV"
+          fi
+          echo "score cache: enabled=$enabled save=$save key prefix $own$SHARD."
+      # The three binaries every shard and every run of a commit builds alike
+      # (about 70 s of each job) are kept under the commit's full SHA, the
+      # toolchain and the profile: what the cache holds is what that code
+      # builds to, so a run of any ref may save it, and only a run of that
+      # commit restores it. A ref that is not a full SHA is built each time.
+      # The probe examples are always built.
+      - name: Binary cache key
+        id: binaries-key
+        shell: bash
+        env:
+          MEASURED: ${{ matrix.side == 'base' && needs.plan.outputs.base_sha || needs.plan.outputs.head_sha }}
+        run: |
+          if [[ "$MEASURED" =~ ^[0-9a-f]{40}$ ]]; then
+            echo "key=rvt-bin-v1-$MEASURED-$RUNNER_OS-$RUNNER_ARCH-$(rustc -vV | sha256sum | cut -c1-16)-ci" >> "$GITHUB_OUTPUT"
+          else
+            echo "the measured ref is not a full SHA, so its binaries are built each time"
+          fi
+      - name: Restore the binaries
+        id: binaries
+        if: steps.binaries-key.outputs.key != ''
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: |
+            measured/target/ci/rvt-ifc
+            measured/target/ci/rvt-gltf
+            measured/target/ci/rvt-info
+          key: ${{ steps.binaries-key.outputs.key }}
       - name: Build
         working-directory: measured
         run: |
           git log -1 --format='%H %s' > ../measured-commit.txt
-          cargo build --profile ci --bin rvt-ifc --bin rvt-gltf --bin rvt-info
+          if [ "$BINARIES_CACHED" = true ]; then
+            echo "rvt-ifc, rvt-gltf and rvt-info restored from the binary cache"
+          else
+            cargo build --profile ci --bin rvt-ifc --bin rvt-gltf --bin rvt-info
+          fi
           # A base older than a probe has no such example; it is skipped there.
           # Probes live in the rvt-probes crate (probes/examples) since it was
           # split out; a ref from before that has them in examples/. Both build
@@ -343,6 +442,28 @@ jobs:
           fi
         env:
           PROBE: ${{ inputs.probe }}
+          BINARIES_CACHED: ${{ steps.binaries.outputs.cache-hit }}
+      # Every shard of a commit misses at once and saves the same key; the one
+      # that reserves it first stores it and the others are told it is taken.
+      # A run from a pull request saves nothing, its cache being its own.
+      - name: Save the binaries
+        if: ${{ success() && github.event_name == 'workflow_dispatch' && steps.binaries-key.outputs.key != '' && steps.binaries.outputs.cache-hit != 'true' }}
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: |
+            measured/target/ci/rvt-ifc
+            measured/target/ci/rvt-gltf
+            measured/target/ci/rvt-info
+          key: ${{ steps.binaries-key.outputs.key }}
+      # Restored after the Build step, so that no build script of the measured
+      # ref ever sees a writable score cache.
+      - name: Restore the score cache
+        if: steps.score-cache-key.outputs.enabled == 'true'
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: _score-cache
+          key: ${{ steps.score-cache-key.outputs.prefix }}${{ matrix.side }}-${{ github.run_id }}-${{ github.run_attempt }}
+          restore-keys: ${{ steps.score-cache-key.outputs.restore }}
       - name: Fetch the reference models
         run: tools/fetch-reference-models.sh models
       # The office models' host being down is NOT MEASURED; a file that no
@@ -507,6 +628,25 @@ jobs:
           # The runner's CPU, so the compare job can say which machines the
           # wall times came from.
           grep -m1 'model name' /proc/cpuinfo | cut -d: -f2- | xargs > "results/runner-$SHARD.txt"
+      # Entries not used for 21 days go, then the oldest past 256 MiB; the cache
+      # is saved only if this run kept an entry, pruned one or refreshed one not
+      # used for three days (tools/maintainer/clean_caches.py deletes the
+      # generations it supersedes).
+      - name: Finish the score cache
+        id: score-cache
+        if: ${{ !cancelled() && steps.score-cache-key.outputs.enabled == 'true' }}
+        shell: bash
+        env:
+          SHARD: ${{ matrix.shard }}
+        run: |
+          mkdir -p results
+          python tools/ci/measure-cache.py finish --report "results/score-cache-$SHARD.log"
+      - name: Save the score cache
+        if: ${{ !cancelled() && github.event_name == 'workflow_dispatch' && steps.score-cache.outputs.changed == 'true' && steps.score-cache-key.outputs.save == 'true' }}
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: _score-cache
+          key: ${{ steps.score-cache-key.outputs.prefix }}${{ matrix.side }}-${{ matrix.side == 'base' && needs.plan.outputs.base_sha || needs.plan.outputs.head_sha }}-${{ github.run_id }}-${{ github.run_attempt }}
       # The held-out models and BIMcollab's carry no licence: only their
       # counts leave the runner (RE-252).
       - name: Keep only the unlicensed models' counts

docs/maintaining.md

@@ -94,6 +94,111 @@ when a release ships.
   ([`changelog.d/`](../changelog.d/README.md)) change in the same pull request
   as the capability they describe.
 
+## Measure's score cache
+
+Measure runs about 40 scorers and four schema checks on every model, and most
+pull requests move the export of only a few of them. A step whose inputs are as
+they were in an earlier run takes that run's output from the score cache
+(`tools/ci/measure-cache.py`) instead of running; exports, `rvt-info`,
+`rvt-gltf`, the witness exports, diagnostics and probes always run. A hit
+writes the cached output and a `<step> cached` line to the model's
+`timings.log`; every output is the one a run would have written.
+
+- **Key.** The sha256 of the step's name and arguments; the content of every
+  file argument (the rvt-rs IFC without its `FILE_NAME` and `IFCOWNERHISTORY`
+  lines, which carry the export time, so a rebuild that writes the same model
+  hits; Revit's export; the glTF; the validation findings); the scorer and
+  every module of `tools/re` and `tools/ci` it reaches by import, importlib
+  path or name, and every file of `tools/re` and `tools/ci` that is not Python
+  (the Measure script, the requirements, the schemas); the value of every
+  environment variable the scorers read (apart from the two that size the mesh
+  cache), which is `PSETS_MISSING` for `psets_vs_ifc` and `qtos_vs_ifc`; the
+  files a scorer reads that its arguments do not name (`qtos_vs_ifc`'s
+  `diagnostics.json`, `unfilled_openings_vs_ifc`'s sibling Revit exports); the
+  Python minor version (not the patch: a runner image moves it, 3.12.15 to
+  3.12.16 in run 41769277902, which cost two of six shards every hit), the
+  platform, the sha256 of `tools/ci/requirements-ifc.txt`, the installed
+  version of every package it pins, the versions IfcOpenShell and numpy
+  report, `RVT_IFC_FLAGS`, and the sha256 of `measure-cache.py` itself, so a
+  change to what is kept or looked up starts a cache. Each entry therefore
+  hashes the rvt-rs IFC, Revit's reference IFC (or the glTF), the command-line
+  arguments, the scorer and its transitive `tools/re` and `tools/ci` imports,
+  the environment variables they read and the library versions.
+- **What is never cached.** A step whose closure lists or walks a directory,
+  takes a directory from an argument, opens a file by a literal path, runs a
+  program, draws a number or a date, reads the IFC's export time or reads an
+  environment variable by a computed name, or one given a directory as an
+  argument: the cache tells the reason in `score-cache-<shard>.log` and runs
+  it. A new scorer that reads anything but the files it is given is run every
+  time until `EXTRA_INPUTS` in `measure-cache.py` names what it reads. A step
+  is kept only if it finished with status 0 and neither its output nor any file
+  it also wrote (the validation findings, the rules summary) is over 4 MiB or
+  shows a failure a rerun could mend (`unreadable:`, a traceback, a
+  `MemoryError`, a dead worker); `ifc_validate_summary.py --both` exits 1 when
+  a check failed, where its separate runs print the failure and exit 0. An
+  entry that lacks a file the step wrote is a miss.
+  The held-out models, BIMcollab's and the pairs that were held out
+  (`holdout-*`, `bimcollab-*`, `spent-*`: BIM Whale's AdvancedProject was spent
+  and held out again on 2026-11-03), of which only counts leave the runner
+  (`tools/ci/scrub-holdout-results.sh`), are never looked up or stored, because
+  the Actions cache of a public repository can be restored by a workflow that a
+  fork's pull request changes: nothing the scrub removes may be in it, and a
+  pair's results do not become safe by being spent. Each entry names its model,
+  and the end of every run removes any entry that names one of those models or
+  none and prints the entries by model prefix, so the saved cache is shown to
+  hold none. That gives up caching a share of the work, not a small one: in
+  the unfiltered run 41748430211 the 18 held-out models took 32.8 of the 119
+  task-minutes of model steps (28%); they run only with `-f holdout=true`.
+- **Where it lives.** Each shard of the run, whatever it is called, restores
+  the newest `measure-scores-v2-main-<shard>.*` cache after the measured code
+  has built (a build script never sees a writable cache) (a run given `models`
+  is the shard `all`, with a cache of its own,
+  built by earlier runs of that kind). It is saved only where rust-cache is saved: the
+  workflow dispatched from `main` and the side measuring `main`'s own code
+  (`inputs.ref` empty, or `base` `main`), because the build scripts of any
+  other ref run on the runner and must not write what a later `main` run
+  reads. A pull request run, and a run of any other ref, only reads. Keying on
+  the scorers makes an entry reachable only by a run whose scorers, files and
+  libraries hash alike, but that is not safe by construction: a build script of
+  the measured ref could rewrite the installed IfcOpenShell, which the key
+  names only by version, and the cache would then hold what that run produced.
+  That is why only `main`'s own code saves it. A run dispatched with `-f
+  score_cache=<name>` reads and saves a cache of its own,
+  `measure-scores-v2-x-<name>-<shard>.*`, from any ref (GitHub scopes a
+  branch's caches to that branch, and the shared cache never reads the name);
+  this is how a change to the cache itself is proved. `-f score_cache=off`
+  runs every step as before.
+- **Bounds.** Entries not used for 21 days are removed, then the oldest past
+  256 MiB; a run saves a new cache only if it kept an entry, pruned one or
+  refreshed one unused for three days, and
+  `tools/maintainer/clean_caches.py` deletes the generations a newer one
+  supersedes. An entry is a scorer's text output (kilobytes), so the cache is
+  a few MiB, not GiB. `results/score-cache-<shard>.log` in the artifact lists every
+  hit, miss, kept and uncached step with the first twelve characters of its key.
+- **Checking it.** The run after a change to an export is the check that the
+  key follows the IFC: its changed models miss and the rest hit. To check a
+  change to the cache or to a scorer, run Measure twice on one branch with
+  `-f score_cache=<name>` (cold, then warm) and once with `-f
+  score_cache=off`, and `diff -r` the three results apart from `*.log`,
+  `commit.txt` and `runner*.txt` (`diagnostics.json` is not left out: it is
+  byte-identical across runs of one commit, and `qtos_vs_ifc` reads it); to
+  check that a changed input misses, run the same name at two commits whose
+  exports differ for some model, and the second must miss on the models that
+  moved and hit on the rest, and diff clean against a run with the cache off
+  at that commit; a warm run with `-f
+  psets_missing=Pset_WallCommon.Reference` must miss `psets_vs_ifc` on every
+  model and hit everything else.
+
+Measure also keeps the three binaries its Build step makes (`rvt-ifc`,
+`rvt-gltf`, `rvt-info`) under `rvt-bin-v1-<full SHA>-<OS>-<arch>-<toolchain
+hash>-ci`, so the shards of a run, and every run that measures `main`'s commit
+as its base, build them once (the build is about 70 seconds of each job). What
+the cache holds is a function of its key, the code of that commit built by that
+toolchain, and only a run of that commit restores it, so a run of any ref may
+save it (a pull request's run does not: its caches are its own). A ref that is
+not a full SHA is built each time, the probe examples always are, and
+`clean_caches.py` keeps the newest sixteen.
+
 ## Dependencies and the MSRV
 
 `rust-version` in `Cargo.toml` is the oldest stable Rust the crate builds on, and