diff --git a/TECHNICAL_REPORTS/2276-portable-ab-output-dirs-20261012.en.md b/TECHNICAL_REPORTS/2276-portable-ab-output-dirs-20261012.en.md new file mode 100644 index 000000000..b015c0f19 --- /dev/null +++ b/TECHNICAL_REPORTS/2276-portable-ab-output-dirs-20261012.en.md @@ -0,0 +1,36 @@ +# Technical Report: PR #2276 - Portable A/B Output Directories + +**Date**: 2026-10-12 +**Status**: Ready for review +**Languages**: Bash +**Risk Level**: Low + +## Executive Summary + +The A/B output-equality and paged-decode gates now create their default artifact directories with templates accepted by both GNU and BSD `mktemp`. If a default or requested output directory cannot be created, each gate prints a clear error and exits 2 before running a model or writing artifacts. + +## 1. Problem Statement + +GNU `mktemp -d -t ` rejects a prefix without `X` placeholders. The previous scripts left `OUT_DIR` empty after that failure and continued, which could misreport the setup failure as a model error and form output paths at the filesystem root. + +## 3. Technical Decisions + +### Use path templates and fail before execution + +Both scripts use `${TMPDIR:-/tmp}/.XXXXXX`, which works on GNU/Linux and macOS. The guards then reject an empty directory value or a failed `mkdir -p` with the scripts' inconclusive exit status, so all later artifact paths remain rooted under a validated directory. + +The regression fixture uses stub binaries and HTTP tools to exercise the full shell flows without a checkpoint or server. Its fake curl consumes piped request data, preserving the production pipeline's `pipefail` behavior without an intermittent SIGPIPE from the fake jq producer. + +## 7. Change Summary + +| Category | Count | Summary | +|---|---:|---| +| Production scripts | 2 | Portable default directories and explicit exit-2 setup failures | +| Regression tests | 1 | Default path, invalid explicit path, direct `mktemp` failure, and no-model-invocation coverage | + +## Appendix: Test Results + +- `bash scripts/ci/ab_scripts_outdir_test.sh` passed, including the direct `TMPDIR` failure paths for both gates. +- Twelve concurrent copies of the focused test passed after the fixture consumed the piped request body. +- The focused test failed against the pre-fix `origin/main` scripts with eight expected assertions. +- `bash -n scripts/ab_output_equality.sh scripts/paged_decode_counter_ab.sh scripts/ci/ab_scripts_outdir_test.sh` and `git diff --check` passed. diff --git a/TECHNICAL_REPORTS/2276-portable-ab-output-dirs-20261012.ko.md b/TECHNICAL_REPORTS/2276-portable-ab-output-dirs-20261012.ko.md new file mode 100644 index 000000000..5b15f8e5e --- /dev/null +++ b/TECHNICAL_REPORTS/2276-portable-ab-output-dirs-20261012.ko.md @@ -0,0 +1,36 @@ +# 기술 보고서: PR #2276 - 이식 가능한 A/B 출력 디렉터리 + +**작성일**: 2026-10-12 +**상태**: 리뷰 준비 완료 +**언어**: Bash +**위험도**: 낮음 + +## 요약 + +A/B 출력 동일성 게이트와 paged decode 게이트가 이제 GNU와 BSD `mktemp` 모두에서 허용되는 템플릿으로 기본 산출물 디렉터리를 만듭니다. 기본 또는 요청한 출력 디렉터리를 만들 수 없으면 각 게이트는 모델을 실행하거나 산출물을 쓰기 전에 명확한 오류를 출력하고 종료 상태 2를 반환합니다. + +## 1. 문제 정의 + +GNU의 `mktemp -d -t `는 `X` 자리표시자가 없는 접두사를 거부합니다. 이전 스크립트는 이 실패 뒤에도 빈 `OUT_DIR`로 계속 실행되어, 설정 실패를 모델 오류로 잘못 보고하거나 파일 시스템 루트에 출력 경로를 만들 수 있었습니다. + +## 3. 기술적 선택과 그 이유 + +### 경로 템플릿 사용 및 실행 전 실패 처리 + +두 스크립트는 GNU/Linux와 macOS에서 모두 동작하는 `${TMPDIR:-/tmp}/.XXXXXX`를 사용합니다. 이후 가드는 빈 디렉터리 값 또는 실패한 `mkdir -p`를 스크립트의 판정 불가 종료 상태로 처리하므로, 이후의 모든 산출물 경로는 검증된 디렉터리 아래에 남습니다. + +회귀 테스트 픽스처는 실제 체크포인트나 서버 없이 전체 셸 흐름을 실행하기 위해 stub 바이너리와 HTTP 도구를 사용합니다. 가짜 curl은 파이프된 요청 본문을 소비하여 가짜 jq 생산자에서 발생하는 간헐적인 SIGPIPE 없이 실제 파이프라인의 `pipefail` 동작을 보존합니다. + +## 7. 변경 요약 + +| 분류 | 수 | 요약 | +|---|---:|---| +| 운영 스크립트 | 2 | 이식 가능한 기본 디렉터리와 명시적인 종료 상태 2 설정 실패 | +| 회귀 테스트 | 1 | 기본 경로, 잘못된 명시 경로, 직접 `mktemp` 실패, 모델 미실행 검증 | + +## 부록: 테스트 결과 + +- `bash scripts/ci/ab_scripts_outdir_test.sh`가 두 게이트의 직접 `TMPDIR` 실패 경로를 포함해 통과했습니다. +- 가짜 curl이 파이프된 요청 본문을 소비하도록 한 뒤 집중 테스트 12개 동시 실행이 모두 통과했습니다. +- 집중 테스트는 수정 전 `origin/main` 스크립트에서 예상한 8개 검증 실패를 보였습니다. +- `bash -n scripts/ab_output_equality.sh scripts/paged_decode_counter_ab.sh scripts/ci/ab_scripts_outdir_test.sh` 및 `git diff --check`가 통과했습니다. diff --git a/scripts/ab_output_equality.sh b/scripts/ab_output_equality.sh index 523e2e02a..33da885fb 100755 --- a/scripts/ab_output_equality.sh +++ b/scripts/ab_output_equality.sh @@ -49,8 +49,8 @@ # /bin/cp target/release/mlxcel target/release/mlxcel.before # git stash pop && cargo build --release --features metal,accelerate --bin mlxcel # -# Exit status: 0 all pairs equal, 1 at least one pair differs, 2 at least one -# pair inconclusive (unstable baseline) and none differ. +# Exit status: 0 all pairs equal, 1 at least one pair differs, 2 at least one pair +# inconclusive (unstable baseline) and none differ, or the output directory cannot be created. set -uo pipefail @@ -126,9 +126,15 @@ if [[ ${#PROMPTS[@]} -eq 0 ]]; then fi if [[ -z "$OUT_DIR" ]]; then - OUT_DIR="$(mktemp -d -t ab_output_equality)" + OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/ab_output_equality.XXXXXX")" || { + echo "error: could not create output directory" >&2 + exit 2 + } +fi +if [[ -z "$OUT_DIR" ]] || ! mkdir -p "$OUT_DIR"; then + echo "error: could not create output directory" >&2 + exit 2 fi -mkdir -p "$OUT_DIR" echo "baseline: $BASELINE_BIN" echo "arm: $ARM_BIN" diff --git a/scripts/ci/ab_scripts_outdir_test.sh b/scripts/ci/ab_scripts_outdir_test.sh new file mode 100644 index 000000000..5790e3076 --- /dev/null +++ b/scripts/ci/ab_scripts_outdir_test.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +# Regression coverage for the A/B script default output directories. + +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(cd "$script_dir/../.." && pwd)" +output_equality="$repo_root/scripts/ab_output_equality.sh" +paged_counter="$repo_root/scripts/paged_decode_counter_ab.sh" + +work="$(mktemp -d "${TMPDIR:-/tmp}/ab-scripts-outdir-test.XXXXXX")" +trap 'rm -rf "$work"' EXIT + +failures=0 + +fail() { + echo "FAIL [$1]: $2" >&2 + failures=$((failures + 1)) +} + +write_fake_mlxcel() { + local path="$1" marker="$2" + printf '%s\n' \ + '#!/usr/bin/env bash' \ + "# $marker" \ + 'if [[ -n "${AB_SCRIPTS_OUTDIR_TEST_MODEL_MARKER:-}" ]]; then' \ + ' : > "$AB_SCRIPTS_OUTDIR_TEST_MODEL_MARKER"' \ + 'fi' \ + 'case "${1:-}" in' \ + ' generate)' \ + ' printf "Generating...\\n"' \ + ' printf "same generated text\\n"' \ + ' printf "[Generated 1 tokens in 0.01s]\\n"' \ + ' ;;' \ + ' serve)' \ + ' while :; do sleep 1; done' \ + ' ;;' \ + ' *)' \ + ' exit 1' \ + ' ;;' \ + 'esac' > "$path" + chmod +x "$path" +} + +write_fake_curl() { + local path="$1" + printf '%s\n' \ + '#!/usr/bin/env bash' \ + 'set -euo pipefail' \ + 'output=""' \ + 'read_stdin=0' \ + 'url=""' \ + 'while [[ $# -gt 0 ]]; do' \ + ' case "$1" in' \ + ' -o) output="$2"; shift 2 ;;' \ + ' -H) shift 2 ;;' \ + ' --data-binary) [[ "$2" == "@-" ]] && read_stdin=1; shift 2 ;;' \ + ' -*) shift ;;' \ + ' *) url="$1"; shift ;;' \ + ' esac' \ + 'done' \ + 'case "$url" in' \ + ' */health) body="{}" ;;' \ + ' */metrics) body=$(printf "mlxcel_paged_decode_launches_total{path=\\\"fused_v2\\\"} 1\\nmlxcel_paged_decode_launches_total{path=\\\"gather\\\"} 0\\nmlxcel_paged_decode_launches_total{path=\\\"cascade\\\"} 0") ;;' \ + ' */v1/completions) body="{\\\"choices\\\":[{\\\"text\\\":\\\"same\\\"}]}" ;;' \ + ' *) printf "unexpected curl URL: %s\\n" "$url" >&2; exit 1 ;;' \ + 'esac' \ + 'if [[ "$read_stdin" -eq 1 ]]; then cat >/dev/null; fi' \ + 'if [[ -n "$output" ]]; then' \ + ' printf "%s\\n" "$body" > "$output"' \ + 'else' \ + ' printf "%s\\n" "$body"' \ + 'fi' > "$path" + chmod +x "$path" +} + +write_fake_jq() { + local path="$1" + printf '%s\n' \ + '#!/usr/bin/env bash' \ + 'if [[ "${1:-}" == "-n" ]]; then' \ + ' printf "{}\\n"' \ + ' exit 0' \ + 'fi' \ + 'if [[ "${1:-}" == "-r" ]]; then' \ + ' printf "same\\n"' \ + ' exit 0' \ + 'fi' \ + 'exit 1' > "$path" + chmod +x "$path" +} + +assert_default_dir() { + local name="$1" prefix="$2" output="$3" + local out_dir + out_dir="$(sed -n 's/^outputs:[[:space:]]*//p' <<< "$output")" + case "$out_dir" in + "$work/tmp/$prefix".??????) ;; + *) + fail "$name" "expected an output directory below $work/tmp, got '${out_dir:-}'" + return + ;; + esac + [[ -d "$out_dir" ]] || fail "$name" "output directory was not created: $out_dir" +} + +assert_create_failure() { + local name="$1" script="$2" + shift 2 + local output="$work/$name.failure.out" + local status + set +e + "$script" "$@" --out-dir "$work/not-a-directory/child" > "$output" 2>&1 + status=$? + set -e + [[ "$status" -eq 2 ]] || fail "$name create failure" "expected exit 2, got $status" + grep -Fq 'error: could not create output directory' "$output" || fail "$name create failure" "missing output-directory error" +} + +assert_mktemp_failure() { + local name="$1" script="$2" + shift 2 + local marker="$work/$name.model-invoked" + local output="$work/$name.mktemp-failure.out" + local status + rm -f "$marker" + set +e + AB_SCRIPTS_OUTDIR_TEST_MODEL_MARKER="$marker" TMPDIR="$work/not-a-directory/missing" "$script" "$@" > "$output" 2>&1 + status=$? + set -e + [[ "$status" -eq 2 ]] || fail "$name mktemp failure" "expected exit 2, got $status" + grep -Fq 'error: could not create output directory' "$output" || fail "$name mktemp failure" "missing output-directory error" + [[ ! -e "$marker" ]] || fail "$name mktemp failure" "model binary was invoked after output-directory creation failed" +} + +mkdir -p "$work/tmp" "$work/bin" +write_fake_mlxcel "$work/baseline" baseline +write_fake_mlxcel "$work/arm" arm +write_fake_curl "$work/bin/curl" +write_fake_jq "$work/bin/jq" +printf 'not a directory\n' > "$work/not-a-directory" + +set +e +output="$(TMPDIR="$work/tmp" "$output_equality" --baseline "$work/baseline" --arm "$work/arm" --model models/fake --prompt test -n 1 2>&1)" +status=$? +set -e +if [[ "$status" -eq 0 ]]; then + assert_default_dir "output equality default" ab_output_equality "$output" +else + fail "output equality default" "expected exit 0, got $status" + sed 's/^/ /' <<< "$output" >&2 +fi +assert_create_failure "output equality" "$output_equality" --baseline "$work/baseline" --arm "$work/arm" --model models/fake --prompt test -n 1 +assert_mktemp_failure "output equality" "$output_equality" --baseline "$work/baseline" --arm "$work/arm" --model models/fake --prompt test -n 1 + +set +e +output="$(PATH="$work/bin:$PATH" TMPDIR="$work/tmp" "$paged_counter" --baseline "$work/baseline" --arm "$work/arm" --model models/fake --requests 1 -n 1 2>&1)" +status=$? +set -e +if [[ "$status" -eq 0 ]]; then + assert_default_dir "paged counter default" paged_decode_counter_ab "$output" +else + fail "paged counter default" "expected exit 0, got $status" + sed 's/^/ /' <<< "$output" >&2 +fi +assert_create_failure "paged counter" "$paged_counter" --baseline "$work/baseline" --arm "$work/arm" --model models/fake --requests 1 -n 1 +assert_mktemp_failure "paged counter" "$paged_counter" --baseline "$work/baseline" --arm "$work/arm" --model models/fake --requests 1 -n 1 + +if [[ "$failures" -ne 0 ]]; then + echo "A/B output-directory regression test: $failures case(s) failed" >&2 + exit 1 +fi + +echo "A/B output-directory regression test: all cases passed" diff --git a/scripts/paged_decode_counter_ab.sh b/scripts/paged_decode_counter_ab.sh index 7d5a907a3..1bf54bc40 100755 --- a/scripts/paged_decode_counter_ab.sh +++ b/scripts/paged_decode_counter_ab.sh @@ -61,7 +61,7 @@ # Exit status: 0 both sides launched the fused path the same number of times # with no fallbacks and produced the same text; 1 the counters disagree, a side # never launched the fused path, or the text differs; 2 the run could not be -# made meaningful (server never came up, metrics unavailable). +# made meaningful (server never came up, metrics unavailable, output directory unavailable). set -uo pipefail BASELINE_BIN="" @@ -107,8 +107,16 @@ if cmp -s "$BASELINE_BIN" "$ARM_BIN"; then exit 1 fi -[[ -n "$OUT_DIR" ]] || OUT_DIR="$(mktemp -d -t paged_decode_counter_ab)" -mkdir -p "$OUT_DIR" +if [[ -z "$OUT_DIR" ]]; then + OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/paged_decode_counter_ab.XXXXXX")" || { + echo "error: could not create output directory" >&2 + exit 2 + } +fi +if [[ -z "$OUT_DIR" ]] || ! mkdir -p "$OUT_DIR"; then + echo "error: could not create output directory" >&2 + exit 2 +fi echo "baseline: $BASELINE_BIN" echo "arm: $ARM_BIN"