Repository navigation
ci(measure): a scorer or schema step whose inputs are unchanged takes its output from a score cache (#1237) - #1317
Draft
DrunkOnJava wants to merge 10 commits into
Draft
DrunkOnJava wants to merge 10 commits into
DrunkOnJava wants to merge 10 commits into
Conversation
… its output from a score cache, output unchanged (#1237) Measure re-ran every scorer and schema check on every model for head and for base, though most pull requests move the export of a few models. tools/ci/measure-cache.py keys each step on its name and arguments, the content of its file arguments (the rvt-rs IFC without the FILE_NAME and IFCOWNERHISTORY lines that carry the export time), the scorer and every tools/re and tools/ci module it reaches, the environment variables they read, the files they read beside their arguments, the Python, platform and pinned package versions and RVT_IFC_FLAGS, and keeps the output of a step that finished with status 0. A step whose closure does something the key cannot see is not cached, and the held-out models and BIMcollab's are never looked up or stored. Every run restores the newest cache; only the workflow dispatched from main on main's own code saves it, and score_cache=<name> gives a run of any ref a cache of its own.
…s before (#1237) An exception in the helper exited 1, which the Measure script reads as a miss and would then store under a stale key. A failed lookup now says not cached (2), a failed store keeps nothing, and a lookup forgets its key file first.
…ne parse of the IFC, output unchanged (#1237) ifc_validate_summary.py --both MODEL STATEMENTS RULES.txt runs the default validation and the rules check on one ifcopenshell.open and writes each exactly as its own run does, an unreadable file or a failing check being the finding of both or of that one as before. The schema function runs it as one step, schema-validate-rules, cached with the validation findings and the rules summary.
…y that names a held-out one, and the key adds the library versions IfcOpenShell and numpy report (#1237) The Actions cache of a public repository can be restored by a workflow that a fork's pull request changes, so nothing tools/ci/scrub-holdout-results.sh removes may be in it. holdout-* and bimcollab-* were never looked up or stored; now store refuses them as well, every entry carries its model in meta.json, and finish removes an entry that names a held-out model or none and prints the entries by model prefix. The key also hashes requirements-ifc.txt and the versions ifcopenshell and numpy report, worked out once a run.
Resolves measure.yml against the sharded layout (#1311): the score cache is keyed per shard and by the measured SHA, a run given models restores the other shards' caches too, and each shard writes its own score-cache-<shard>.log.
… rule in the Measure script's header (#1237)
… is, and a run given models restores only its own (#1237) The shard names are the plan job's to choose, so no step lists them. A shard's prefix ends in a dot that no shard name holds, so one shard's prefix is never another's, and the extra restores of the four fixed shards for a run given models are gone: that run keeps a cache of its own, built by earlier runs of its kind.
…HA, the toolchain and the profile (#1237) The shards of a run, and every run that measures main's commit as its base, built the same three binaries (about 70 seconds of each job). rvt-bin-v1-<SHA>-<OS>-<arch>-<toolchain>-ci holds rvt-ifc, rvt-gltf and rvt-info; a hit skips their cargo build, a ref that is not a full SHA is built each time, the probe examples always are, a run of a pull request saves nothing, and clean_caches.py keeps the newest sixteen.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Measure's scorers and schema checks no longer run again on a model whose inputs have not changed: each step's output is kept in a content-addressed score cache, and the schema validation and the EXPRESS rules check share one parse of the IFC. Every output is byte-identical to an uncached run (runs below).
Motivation
Run 41748430211 (unfiltered, 47 minutes): the Measure step was 43 of them, CPU-bound, 164 task-minutes on four cores; in a typical pull request more than 90% of the models export byte-identical to main, yet every scorer ran on head again and base re-scored main from scratch. Refs #1237.
Changes
tools/ci/measure-cache.py(new):lookup,storeandfinishfor the score cache. A hit writes the cached output and a<step> cachedline totimings.log.tools/ci/measure-reference-models.sh:cscore(the tools/re scorers in the two lists,ports_vs_ifcandrevit_property_sets_vs_ifc) andcached(the four schema steps) look a step up before running it. Exports,rvt-info,rvt-gltf, the witness exports, diagnostics, probes and the scorers named only throughscorersalways run.tools/ci/ifc_validate_summary.py:--both MODEL STATEMENTS RULES.txtruns the validation and the rules check on oneifcopenshell.open, each written exactly as its own run does; the schema function runs it as one step,schema-validate-rules..github/workflows/measure.yml: ascore_cacheinput, a key step,actions/cache/restorebefore the build (per shard; a run givenmodelsalso restores the other shards'),finishandactions/cache/saveafter the Measure step.tools/maintainer/clean_caches.py: the new key family;docs/maintaining.md: the operator note.Testing
Evidence is being added as the proof runs finish (unfiltered cold, warm and cache-off runs on the sharded workflow, pinned to main 46aa05f). Smoke proofs so far, on core-interior, einhoven, re1-architecture and the 2024 samples: cache off 8.48 task-minutes; cold, every output identical to cache off; warm, 130 of 130 steps cached, 0.28 task-minutes, every output identical.
No changelog fragment (the earlier
ci(measure)pull requests, #1285 and #1295, carry none) and notools/check-local.sh(no local compute).Breaking changes
None.
Related issues
Refs #1237.