refactor(ci): moveprovider benchmark, scoredirector reporting by triceo · Pull Request #157 · TimefoldAI/timefold-solver-benchmarks

triceo · 2026-09-08T13:05:00Z

No description provided.

# Conflicts: # .github/workflows/performance_move_provider.yml

The two scenarios were singleDraw (1 draw + commit) and manyDraws (10 draws + commit), and the commit dominated both. A fit of A + B*n to them shows the move provider does 2 % of the work in singleDraw and 15 % in manyDraws, so neither measured move generation - which is what a move-provider benchmark exists for. Replace them with commitMove and drawOnly. commitMove is the old singleDraw, renamed to say what it measures: apply one move and settle the neighborhood network. It calculates no score, so it is not a whole step. drawOnly draws 500 candidates and commits none, so the solution never changes and its speed is generation alone. manyDraws goes away; it was exactly commitMove + 9 * drawOnly. drawOnly carries @OperationsPerInvocation, so its speed is drawn moves for each second, and its movedValues counter divided by that speed is the mean move size. Its draw count is frozen: a bi move iterator retires dead left tuples for the life of one iterator, so a longer invocation draws from a more pruned pool and reports a faster number. The report keeps its five columns but now ranks both costs, with one ranking line for each scenario above each table, and it explains all four columns at the bottom. It also maps a baseline's singleDraw onto commitMove, so a baseline jar built from an older ref still joins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The table shows a delta but no margin of error, so a reader cannot tell whether a delta is bigger than the noise it sits in. Every cell already marked with a high score error now prints one line below its family table, naming both margins and the delta they must be weighed against. A clean table prints no such line, and a margin on a zero score prints as absent rather than crashing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The legend named every verdict but explained none, and "undetermined" is the one label that cannot explain itself. It now says what makes a row undetermined, that it fails the build, and what to do about it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Do not merge; revert after the experiment run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Run 33599612547 raised the inline budget on both sides and the per-fork modes survived, so the 4.0 band derived from within-mode spread does not hold yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Do not merge; revert after the experiment run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Background compilation let each JVM race C2 against its own profile and keep the outcome for life, splitting the forks of one benchmark into two speeds 10-30 % apart. -Xbatch removes the race: SUB_LIST_SWAP/drawOnly went from a 1.37x per-fork spread (CV 12 %) to 1.01x (CV 0.2 %) in run 33606514634. Forks drop from 40 to 20, since they no longer average over two speeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The new summarizers can crash on edge-case JMH output (division by zero) and the move-provider flush cadence does not match its “bounded periodic” contract, risking unstable/biased measurements.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR refactors the CI performance workflows by switching both the score-director and move-provider benchmarks to a subjob-per-slice model that uploads raw JMH JSON artifacts, and adds Python summarizers that aggregate those artifacts into a single markdown regression report (failing CI on regressions/undetermined/missing data). It also introduces a new move-provider microbenchmark suite (Java) with supporting configuration and “no-op” constraint providers to minimize non-measured overhead.

Changes:

Add summarize-scoredirector.py and summarize-moveprovider.py to aggregate per-subjob artifacts into a single CI report with verdicts and links to benchmark assets.
Introduce a new MoveProvider microbenchmark suite (cases, JMH benchmarks, runner main, dataset loading, and a runnable non-JMH test).
Update GitHub Actions workflows to build baseline/SUT once, run benchmark subjobs on GitHub-hosted ARM runners, archive raw results + asset URLs, and run a final summarizing job.

File summaries

File	Description
summarize-scoredirector.py	New: aggregates per-example score-director benchmark artifacts into one markdown table + CI exit code.
summarize-moveprovider.py	New: aggregates per-provider/scenario move-provider benchmark artifacts into tables + CI exit code.
src/test/java/ai/timefold/solver/benchmarks/micro/moveprovider/MoveProviderBenchmarkTest.java	New: non-JMH runnable lifecycle check across all built-in move providers.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/problems/MoveProviderProblem.java	New: implements commitMove/drawOnly benchmark scenarios with undo/flush behavior.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/NoOpVehicleRoutingConstraintProvider.java	New: no-op constraints to avoid join overhead for move-provider benchmarks.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/NoOpTaskAssigningConstraintProvider.java	New: no-op constraints to avoid join overhead for move-provider benchmarks.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/NoOpPatientAdmissionScheduleConstraintProvider.java	New: no-op constraints to avoid join overhead for move-provider benchmarks.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/NoOpMeetingSchedulingConstraintProvider.java	New: no-op constraints to avoid join overhead for move-provider benchmarks.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/MoveProviderCase.java	New: common interface for move-provider enum cases and moved-values counting.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/Main.java	New: benchmark entrypoint wiring configuration → JMH runner, plus score error warnings.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/ListMoveProviderCase.java	New: list-variable move-provider case definitions and moved-values counters.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/jmh/MovedValueCounter.java	New: JMH aux counter for moved values to normalize throughput in CI reporting.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/jmh/ListVariableMoveProviderBenchmark.java	New: JMH benchmark subclass for list-variable providers.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/jmh/BasicVariableMoveProviderBenchmark.java	New: JMH benchmark subclass for basic-variable providers.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/jmh/AbstractMoveProviderBenchmark.java	New: shared JMH config/scenarios/constants for move-provider benchmarks.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/Example.java	New: dataset + solver-config selection for move-provider benchmark suite.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/Configuration.java	New: benchmark config parsing for selecting provider subsets and JMH parameters.
src/main/java/ai/timefold/solver/benchmarks/micro/moveprovider/BasicMoveProviderCase.java	New: basic-variable move-provider case definitions and moved-values counters.
src/main/java/ai/timefold/solver/benchmarks/micro/common/AbstractMain.java	Update: add `-Xbatch` JVM arg to reduce compilation nondeterminism in perf runs.
src/main/java/ai/timefold/solver/benchmarks/micro/common/AbstractConfiguration.java	Update: treat blank selection properties as “run nothing” instead of “run everything”.
run-moveprovider.sh	New: local runner script for move-provider benchmark (mirrors score-director script).
moveprovider-benchmark.properties	New: example properties file for selecting move providers and benchmark parameters.
.github/workflows/performance_score_director.yml	Update: refactor into per-example benchmark subjobs + final summarizing report job.
.github/workflows/performance_quickstarts.yml	Update: bump default baseline version.
.github/workflows/performance_move_provider.yml	Update: refactor into per-provider benchmark subjobs + final summarizing report job; tune fork/warmup/measurement iterations.

Review details

Suppressed comments (2)

summarize-scoredirector.py:110

evaluate_row() can raise ZeroDivisionError when the baseline score is 0, because it divides by old["score"] unconditionally. Even if JMH normally reports non-zero throughput, a failed/degenerate run could produce 0 and crash the summarizer instead of reporting an undetermined/missing row.

def evaluate_row(old: "dict | None", new: "dict | None") -> tuple["float | None", Verdict, bool]:
    """Returns (delta_pct, verdict, high_error). delta_pct is None only for MISSING."""
    if old is None or new is None:
        return None, MISSING, False
    delta_pct = (new["score"] / old["score"] - 1) * 100
    error_limit = TOLERANCE_PCT / 100 / HIGH_ERROR_BAND_FRACTION

summarize-moveprovider.py:178

evaluate_row() can raise ZeroDivisionError when the baseline score is 0, because it divides by old["score"] unconditionally. If a JMH run ever produces 0 throughput (for example due to a degenerate/failed run), the summarizer would crash instead of producing a verdict.

def evaluate_row(old: "dict | None", new: "dict | None", tolerance_pct: float) -> tuple["float | None", Verdict, bool]:
    """Returns (delta_pct, verdict, high_error). delta_pct is None only for MISSING.

    tolerance_pct belongs to the scenario, not to the report; see TOLERANCE_PCT.
    """
    if old is None or new is None:
        return None, MISSING, False
    delta_pct = (new["score"] / old["score"] - 1) * 100
    error_limit = tolerance_pct / 100 / HIGH_ERROR_BAND_FRACTION

Files reviewed: 25/27 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    private void maybeFlushConstraintStreamSession() {
+        if (invocationCounter >= AbstractMoveProviderBenchmark.FLUSH_EVERY_N_STEPS && scoreDirector.requiresFlushing()) {
+            scoreDirector.calculateScore();
+        }
+    }


+        url_path = os.path.join(provider_dir, "assets-url.txt")
+        if os.path.exists(url_path):
+            url = open(url_path).read().strip()


+    for baseline_path in glob.glob(os.path.join(data_dir, "*", "baseline-results.json")):
+        example_dir = os.path.dirname(baseline_path)
+        sut_path = os.path.join(example_dir, "sut-results.json")
+        baseline.update(load_side(baseline_path))
+        if os.path.exists(sut_path):
+            sut.update(load_side(sut_path))
+        url_path = os.path.join(example_dir, "assets-url.txt")
+        if os.path.exists(url_path):
+            url = open(url_path).read().strip()
+            if url:
+                for example in load_side(baseline_path):
+                    asset_urls[example] = url


triceo and others added 30 commits September 8, 2026 13:05

build the perf benchmark

e2a1992

introduce the CI

282e10d

# Conflicts: # .github/workflows/performance_move_provider.yml

temporary

3384c8f

another temporary

387e0ec

fix baseline

66b69c5

throw earlier

60bf031

fix assign/unassign

a345c3f

sort table

c360eb7

Increase warmup and measurements

08d9aa1

Make sure we only test moves, not score

129f70c

Moar memory

92277b3

Fix memory use

d6d4913

Bump forks to 4

fc167c1

Fix another harness bug

e602378

Improve the table

2ef67f1

Switch to 8 forks

7b1576d

Switch to 16 forks

b379036

Finally go to 20 forks

7acdd52

improve the table

08d304b

Refactor

fe9a9fc

Tweak

019e7b5

Fix the measurements

0d0c152

chore: TEMP raise inline budget to test fork bimodality

b683650

Do not merge; revert after the experiment run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix(moveprovider): restore the wider drawOnly tolerance

a70a025

Run 33599612547 raised the inline budget on both sides and the per-fork modes survived, so the 4.0 band derived from within-mode spread does not hold yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore: TEMP test -Xbatch against fork bimodality

dc860a1

Do not merge; revert after the experiment run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Bump to 2.6.0

9ca8277

triceo added 8 commits September 8, 2026 13:05

Hide moves

92af4d9

Prepare for merging

8b49d62

Formatting

4bcccb1

Bech to benchmarking again

383c58d

Eliminate VDSO

4cabe9f

feat(ci): improve scoredirector benchmark reporting

6129281

Final changes

0e2efad

Slightly increase tolerance

ea81652

Copilot AI lite review requested due to automatic review settings September 8, 2026 13:05

triceo deployed to internal September 8, 2026 13:05 — with GitHub Actions Active

triceo changed the title ~~refactor(ci): moveprovider benchmark, scoredirector reporting refactor~~ refactor(ci): moveprovider benchmark, scoredirector reporting Sep 8, 2026

triceo merged commit fc223f4 into main Sep 8, 2026
6 of 7 checks passed

triceo deleted the pillars-unified-report-scoredirector branch September 8, 2026 13:05

Copilot started reviewing on behalf of triceo September 8, 2026 13:05

View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Navigation Menu

Repository navigation

refactor(ci): moveprovider benchmark, scoredirector reporting - #157

refactor(ci): moveprovider benchmark, scoredirector reporting#157
triceo merged 38 commits into
mainfrom
pillars-unified-report-scoredirector

triceo commented Sep 8, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

triceo commented Sep 8, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (2)

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants