Skip to content

Calibrate benchmarks against Library Checker problems #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shakayami merged 2 commits into from
Jul 31, 2026
Merged

Calibrate benchmarks against Library Checker problems #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3d36850
Calibrate benchmark workloads against Library Checker problems
claude Jul 31, 2026
e695010
Benchmark only the modules a pull request touches
claude Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 151 additions & 0 deletions .github/workflows/benchmark-full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
name: Benchmark (full sweep)

# Pull requests only benchmark the modules they touch, which keeps Actions time
# down but means no single PR ever sees the whole picture. This sweep runs
# every workload on master and compares against the previous sweep, so drift
# that accumulates across many small PRs still shows up somewhere.

on:
schedule:
# 03:00 UTC every Monday.
- cron: '0 3 * * 1'
workflow_dispatch:
inputs:
scale:
description: 'ACL_BENCH_SCALE (a fraction, or "max" for the judge limits)'
required: false
default: '1.0'
passes:
description: 'Interleaved measurement passes'
required: false
default: '3'

permissions:
contents: read
actions: read

concurrency:
group: benchmark-full
cancel-in-progress: false

jobs:
full-sweep:
runs-on: ubuntu-latest
timeout-minutes: 300
env:
ACL_BENCH_SCALE: ${{ github.event.inputs.scale || '1.0' }}
ACL_BENCH_ROUNDS: '1'
PASSES: ${{ github.event.inputs.passes || '3' }}
ARTIFACT_NAME: benchmark-full-results
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest pytest-benchmark

- name: Run full suite
run: |
set -x
for pass in $(seq 1 "$PASSES"); do
pytest benchmarks/test_time.py --benchmark-only \
--benchmark-json="current-time-$pass.json"
done
python benchmarks/memory_bench.py --output current-memory.json

# Fetch the previous sweep so the summary shows deltas rather than bare
# numbers. Absent (first run, expired retention), the report still
# renders with the current times and n/a deltas.
- name: Download previous sweep
id: previous
continue-on-error: true
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const runs = await github.rest.actions.listWorkflowRuns({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: 'benchmark-full.yml',
status: 'success',
per_page: 10,
});
for (const run of runs.data.workflow_runs) {
if (run.id === context.runId) continue;
const arts = await github.rest.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: run.id,
});
const art = arts.data.artifacts.find(
a => a.name === process.env.ARTIFACT_NAME && !a.expired
);
if (!art) continue;
const zip = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: art.id,
archive_format: 'zip',
});
fs.writeFileSync('previous.zip', Buffer.from(zip.data));
core.setOutput('found', 'true');
core.setOutput('sha', run.head_sha);
core.info(`Comparing against run ${run.id} (${run.head_sha})`);
return;
}
core.setOutput('found', 'false');
core.info('No previous sweep artifact available.');

- name: Unpack previous sweep
run: |
mkdir -p previous
if [ "${{ steps.previous.outputs.found }}" = "true" ] && [ -f previous.zip ]; then
unzip -o -q previous.zip -d previous
fi
# compare.py needs a base to read even when there is no history yet.
if ! ls previous/current-time-*.json >/dev/null 2>&1; then
echo '{"benchmarks": []}' > previous/current-time-1.json
fi
if [ ! -f previous/current-memory.json ]; then
echo '{"benchmarks": []}' > previous/current-memory.json
fi

- name: Compare against previous sweep
run: |
python benchmarks/compare.py \
--base-time previous/current-time-*.json \
--pr-time current-time-*.json \
--base-memory previous/current-memory.json \
--pr-memory current-memory.json \
--base-label 'Previous' --pr-label 'Now' \
--title '📊 Full benchmark sweep' \
--output report.md

- name: Publish to job summary
run: |
{
echo "Commit: \`${{ github.sha }}\`"
if [ "${{ steps.previous.outputs.found }}" = "true" ]; then
echo "Previous sweep: \`${{ steps.previous.outputs.sha }}\`"
else
echo "No previous sweep to compare against."
fi
echo
cat report.md
} >> "$GITHUB_STEP_SUMMARY"

- name: Upload results
uses: actions/upload-artifact@v4
with:
name: benchmark-full-results
retention-days: 90
path: |
current-time-*.json
current-memory.json
report.md
103 changes: 82 additions & 21 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,24 @@ permissions:
contents: read
pull-requests: write

concurrency:
group: benchmark-${{ github.event.pull_request.number }}
cancel-in-progress: true

env:
# Fraction of the Library Checker maximum constraints to run at. The query
# mix and operand types are scale-invariant, so a scaled run keeps the shape
# that makes these benchmarks predictive while fitting in a CI budget.
ACL_BENCH_SCALE: '0.2'
# One timed round per pass; the passes below provide the repetitions, and
# interleaving them is what makes base and PR comparable.
ACL_BENCH_ROUNDS: '1'
PASSES: '3'

jobs:
benchmark:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout PR branch
uses: actions/checkout@v4
Expand All @@ -33,46 +48,92 @@ jobs:
python -m pip install --upgrade pip
pip install pytest pytest-benchmark

- name: Run PR benchmarks (time)
working-directory: pr
run: pytest benchmarks/test_time.py --benchmark-only --benchmark-json=../pr-time.json
- name: List changed files
uses: actions/github-script@v7
with:
script: |
const files = await github.paginate(github.rest.pulls.listFiles, {
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number,
per_page: 100,
});
require('fs').writeFileSync(
'changed-files.txt',
files.map(f => f.filename).join('\n') + '\n'
);

- name: Run PR benchmarks (memory)
working-directory: pr
run: python benchmarks/memory_bench.py --output ../pr-memory.json
# A change to segtree.py cannot make factorize faster, so measuring
# factorize would only burn Actions minutes and give noise another chance
# to invent a row. Everything runs when the workloads or the harness
# themselves change, because then the comparison is what is in question.
- name: Select affected benchmarks
id: select
run: |
names=$(python pr/benchmarks/select_workloads.py \
--changed-files changed-files.txt --format csv)
echo "names=$names" >> "$GITHUB_OUTPUT"
echo "Selected: ${names:-<none>}"

- name: Run base benchmarks (time)
working-directory: base
- name: Run benchmarks (interleaved passes)
if: steps.select.outputs.names != ''
env:
ACL_BENCH_ONLY: ${{ steps.select.outputs.names }}
run: |
if [ -f benchmarks/test_time.py ]; then
pytest benchmarks/test_time.py --benchmark-only --benchmark-json=../base-time.json
else
echo '{"benchmarks": []}' > ../base-time.json
fi
set -x
for pass in $(seq 1 "$PASSES"); do
(cd pr && pytest benchmarks/test_time.py --benchmark-only \
--benchmark-json="../pr-time-$pass.json")
if [ -f base/benchmarks/test_time.py ]; then
(cd base && pytest benchmarks/test_time.py --benchmark-only \
--benchmark-json="../base-time-$pass.json")
else
echo '{"benchmarks": []}' > "base-time-$pass.json"
fi
done

- name: Run base benchmarks (memory)
working-directory: base
- name: Run benchmarks (memory)
if: steps.select.outputs.names != ''
env:
ACL_BENCH_ONLY: ${{ steps.select.outputs.names }}
run: |
if [ -f benchmarks/memory_bench.py ]; then
python benchmarks/memory_bench.py --output ../base-memory.json
(cd pr && python benchmarks/memory_bench.py --output ../pr-memory.json)
if [ -f base/benchmarks/memory_bench.py ]; then
(cd base && python benchmarks/memory_bench.py --output ../base-memory.json)
else
echo '{"benchmarks": []}' > ../base-memory.json
echo '{"benchmarks": []}' > base-memory.json
fi

- name: Compare results
if: steps.select.outputs.names != ''
run: |
python pr/benchmarks/compare.py \
--base-time base-time.json --pr-time pr-time.json \
--base-time base-time-*.json --pr-time pr-time-*.json \
--base-memory base-memory.json --pr-memory pr-memory.json \
--only "${{ steps.select.outputs.names }}" \
--output report.md

- name: Report that nothing needed benchmarking
if: steps.select.outputs.names == ''
run: |
{
echo '## 📊 Performance Benchmark Results'
echo
echo 'No benchmarked module changed, so no workload was run.'
echo
echo '<sub>Benchmarks run only for the modules a pull request'
echo 'touches. A full sweep of every workload runs on a schedule'
echo '(see `.github/workflows/benchmark-full.yml`).</sub>'
} > report.md

- name: Upload raw results
if: steps.select.outputs.names != ''
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: |
pr-time.json
base-time.json
pr-time-*.json
base-time-*.json
pr-memory.json
base-memory.json
report.md
Expand Down
Loading
Loading