Skip to content

fix(compare): gate regressions on the binary under test #287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BryanFRD wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(compare): gate regressions on the binary under test #287

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
414e467
fix(compare): gate regressions on the binary under test
BryanFRD Oct 25, 2026
58715be
fix(run): follow symlinks when sizing the binary
BryanFRD Oct 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions scripts/compare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ echo "Benchmark regression check"
echo "=========================="
echo ""

# Relative checks (ferrflow benchmarks only)
for key in $(jq -r '.benchmarks | keys[]' "$LATEST" | grep '\-ferrflow-'); do
# Relative checks (ferrflow binary under test only)
for key in $(jq -r '.benchmarks | keys[]' "$LATEST" | grep -- '-ferrflow-binary-'); do
new_median=$(jq -r ".benchmarks[\"$key\"].median_ms" "$LATEST")
old_median=$(jq -r ".benchmarks[\"$key\"].median_ms // empty" "$BASELINE" 2>/dev/null || echo "")

Expand Down
18 changes: 5 additions & 13 deletions scripts/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,22 +186,14 @@ measure_memory() {
fi
}

get_npm_binary_size() {
local pkg="$1"
local tmp_dir bin
tmp_dir=$(mktemp -d)
(
cd "$tmp_dir"
npm init -y &>/dev/null
npm install --save "$pkg" &>/dev/null 2>&1
)
bin=$(find "$tmp_dir/node_modules" -type f -path "*/@$pkg/*/bin/$pkg" 2>/dev/null | head -1)
get_binary_size() {
local bin
bin=$(command -v "$1" 2>/dev/null || true)
if [[ -n "$bin" && -f "$bin" ]]; then
du -m "$bin" | awk '{printf "%.1f", $1}'
stat -L -c %s "$bin" | awk '{printf "%.1f", $1/1048576}'
else
echo "N/A"
fi
rm -rf "$tmp_dir"
}

# Get npm package install size in MB
Expand Down Expand Up @@ -306,7 +298,7 @@ for tool in $TOOLS; do
echo "$npm_size" > "$RAW_DIR/${tool}-npm-size.txt"
fi
if [[ "$method" == "binary" && "$tool" == "ferrflow" ]]; then
bin_size=$(get_npm_binary_size ferrflow)
bin_size=$(get_binary_size "$local_bin")
echo "$bin_size" > "$RAW_DIR/ferrflow-binary-size.txt"
fi

Expand Down
12 changes: 12 additions & 0 deletions tests/compare.bats
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,3 +157,15 @@ EOF
[ "$status" -eq 1 ]
[[ "$output" == *"Usage"* ]]
}

@test "ignores npm and docker ferrflow regressions" {
make_json "$TMPDIR/baseline.json" \
'"single-ferrflow-npm-check": {"median_ms": 100.0, "stddev_ms": 1.0, "memory_mb": "N/A"}, "single-ferrflow-docker-check": {"median_ms": 100.0, "stddev_ms": 1.0, "memory_mb": "N/A"}'
make_json "$TMPDIR/latest.json" \
'"single-ferrflow-npm-check": {"median_ms": 300.0, "stddev_ms": 1.0, "memory_mb": "N/A"}, "single-ferrflow-docker-check": {"median_ms": 300.0, "stddev_ms": 1.0, "memory_mb": "N/A"}'

run bash "$SCRIPT_DIR/compare.sh" "$TMPDIR/baseline.json" "$TMPDIR/latest.json"
[ "$status" -eq 0 ]
[[ "$output" != *"single-ferrflow-npm-check"* ]]
[[ "$output" == *"All benchmarks within thresholds"* ]]
}
Loading