feat(e2e_eval): add --build-only mode with per-EP matrix, export dedup, and Azure Artifacts upload

[KayMKM]

Summary

Adds a --build-only mode to scripts/e2e_eval/run_eval.py that generates ModelKit pipeline artifacts (export → optimize → quantize, no compile) across the full EP matrix, without requiring EP hardware, and optionally streams them to the Modelkit Azure Artifacts feed while bounding local disk usage.

Motivation: we need a way to mass-produce per-EP model artifacts for ~200 models on a single (CPU) box and distribute them, but (a) the build normally needs the target EP installed, and (b) writing every stage for every (model × EP) fills the disk fast.

What's included

1. --build-only mode + per-EP matrix

• Runs winml config + winml build --no-compile per model; perf/accuracy are skipped.
• When --ep/--device are omitted, builds the eval EP matrix into <model_dir>/<ep>_<device>/ subdirs: qnn_npu, qnn_gpu, ov_cpu, ov_npu, ov_gpu, mlas_cpu, dml_gpu, vitisai_npu. Pinning --ep/--device writes a single build directly into <model_dir>.
• Precision per combo reuses the existing eval policy (NPU → w8a16, CPU/GPU → auto, native-quant EPs → --no-quant).

2. Cross-EP / cross-host builds (core fix in build.py)

• _run_optimize_stage called resolve_device(device, ep=ep) purely to pick a progress-bar key, which raised when the target EP wasn't installed locally — blocking offline generation of (e.g.) QNN/VitisAI artifacts on a CPU box.
• Now: when the build won't compile (config.compile is None), the missing-EP lookup soft-fails and falls back to the requested device (optimize/quantize only need the EP's static rule data, not a registered EP). When compile will run, it still fails fast. No behavior change for normal compiling builds.

3. Export dedup (disk saver)

• The export.onnx stage is EP/device-independent, so all 8 combos produce an identical export. After each combo builds, its export is hash-compared against a per-model canonical: the first is moved to <model_dir>/_shared/, later identical ones are deleted — one export copy instead of 8 (export is the largest, full-precision artifact).

4. --upload: stream to Azure Artifacts feed, then delete locally

• After a model's combos are built, publishes the whole model dir to the Modelkit feed as a Universal Package, then deletes the local copy — peak disk stays at ~one model's matrix.
• Auth via az login (Entra ID), no PAT. The azure-devops extension is ensured and login verified up front; if not ready, the run aborts (so disk isn't silently filled).
• Package: single name winml-cli-models, one version per model: 0.0.0-<run-stamp>-<model-slug> (valid SemVer 2.0; the 0.0.0- core keeps it legal, the date stamp + slug are the pre-release segment). The shared run-stamp groups a batch.
• Resume: --continue + the same --run-stamp skips already-uploaded models without rebuilding them. Already-uploaded models are detected two ways: a local build_only_uploads.json manifest, and a query against the feed itself at startup (versions matching 0.0.0-<run-stamp>-*). Because the manifest is only written after a successful upload, a fresh --output-dir would otherwise start empty and rebuild everything; seeding from the feed makes it authoritative for what's published, so resume works regardless of local state. A feed-query failure falls back to local-manifest-only behavior.
• Extra flags: --run-stamp, --keep-local, --upload-skip-existing, --feed/--feed-org/--feed-project/--package-name.

Usage

# Build the full EP matrix for P0 models, stream to the feed, delete locals
uv run python scripts/e2e_eval/run_eval.py --build-only --upload --priority P0

# Resume an interrupted batch (same run-stamp; a fresh output dir is fine)
uv run python scripts/e2e_eval/run_eval.py --build-only --upload --continue \
  --run-stamp 20260609 --priority P0

# Local only (no feed), or pin a single EP/device
uv run python scripts/e2e_eval/run_eval.py --build-only --hf-model microsoft/resnet-50

Download a specific model's specific file later:

az artifacts universal download \
  --organization https://dev.azure.com/microsoft --project windows.ai.toolkit \
  --scope project --feed Modelkit --name winml-cli-models \
  --version 0.0.0-20260609-microsoft-resnet-50-image-classification \
  --path ./out --file-filter 'qnn_npu/model.onnx*'

Notes / scope

• src/winml/modelkit/commands/build.py change is gated on config.compile is None, so it only affects no-compile builds; compiling builds are unchanged and still fail fast on a missing EP.
• --continue resumes from the feed (versions matching 0.0.0-<run-stamp>-*) and the local build_only_uploads.json manifest, so a resumed run only needs the same --run-stamp — it no longer has to reuse the same --output-dir. The feed query uses two &-free REST GETs (list packages → resolve the UPack package GUID → list versions) because az resolves to az.cmd and cmd.exe splits query strings on &, dropping every parameter after the first.
• Verified manually end-to-end on a CPU host: full 8-EP matrix build, export dedup, publish to Modelkit, and --file-filter download of individual model.onnx files.
• The feed: https://dev.azure.com/microsoft/windows.ai.toolkit/_artifacts/feed/Modelkit/UPack/winml-cli-models/overview/0.0.0-20260609-prajjwal1-bert-tiny-text-classification

[DingmaomaoBJTU]

Reviewed the full diff. Overall structure is solid and docstrings are thorough. Found a potential data-loss path (false-positive conflict detection triggers local directory deletion), a silent no-op for --continue in the non-upload build-only path, and a few other correctness/usability issues — see inline comments.

[DingmaomaoBJTU]

_hash_files: OSError swallowed with a fixed sentinel — potential dedup false-positive

When a file cannot be opened, the hash is updated with the literal bytes b"<unreadable>" regardless of which file failed or why. Two distinct files that both happen to be unreadable will produce the same hash, which could cause the duplicate-detection path to delete an artifact it never actually verified was identical.

Safer options:

• Re-raise the OSError so the caller can skip dedup for that combo.
• Include the filename in the sentinel so two different unreadable files don't collide: h.update(p.name.encode() + b":unreadable").

[DingmaomaoBJTU]

_is_publish_conflict: substring match on "conflict" is too broad — data-loss risk

Checking for the substring conflict in combined stdout+stderr will produce a false positive any time the Azure CLI emits that word in an unrelated context (e.g. "merge conflict", "address conflict", "proxy conflict"). A false positive here triggers exists-skipped status and deletes the local model directory via shutil.rmtree, making this a potential data-loss path.

Consider narrowing to the specific HTTP 409 indicators or the Azure DevOps error code: StatusCode=409, HttpStatusCode: 409, or PACKAGEVERSIONEXISTS. These are far less likely to appear in unrelated messages.

[DingmaomaoBJTU]

Pinned-EP-only case: _resolve_precision receives device="auto" — w8a16 default silently dropped

When the user sets --ep but omits --device (default is "auto"), use_matrix is False (because args.ep is not None) so combos = [("", args.ep, "auto")]. Then _resolve_precision("auto", ...) returns None since "auto" != "npu", and the config is generated without a precision flag.

This is inconsistent with the matrix path: entry ("qnn_npu", "qnn", "npu") correctly applies w8a16 because device is explicitly npu. Consider resolving "auto" to a concrete device per-EP before calling _resolve_precision, or documenting that precision auto-detection is intentionally delegated to winml config in this pinned path.

[DingmaomaoBJTU]

--continue is silently a no-op in --build-only without --upload

The resume skip is gated on if args.upload and args.continue_run, so --build-only --continue without --upload does nothing — all models are rebuilt from scratch. There is no filesystem-based resume (e.g. checking whether model_dir already contains the expected combo subdirs).

Note: the pre-filtering of entries by --continue in main() lives inside the if args.list_json: branch, and the eval-path filtering is after the if args.build_only: return, so neither runs in build-only mode.

A user resuming an interrupted non-upload batch with --build-only --continue will silently rebuild everything. Consider either:

1. Adding a local-disk skip (check if combo subdir already exists and has artifacts), or
2. Raising an error/warning when --continue is combined with --build-only but not --upload.

[DingmaomaoBJTU]

Misleading --upload-skip-existing help text — the build is NOT skipped

The current text says "skip building+uploading a model whose package version already exists in the feed", but this flag only affects the publish step. By the time a conflict is detected, the model has already been fully built across all EP combos. The flag merely prevents treating the conflict as a failure.

The flag that actually skips the build is --continue (manifest-based resume). Suggested rewrite:

With --upload: treat a version-already-exists publish conflict as success and delete the local copy. To skip rebuilding entirely for already-uploaded models, use --continue.

[DingmaomaoBJTU]

except ValueError is overly broad — may swallow unrelated errors

The intent (per the comment) is to catch "EP not installed on this host", but _resolve_device could also raise ValueError for invalid device names, malformed input, or other programming errors. With config.compile is None, all such errors are silently swallowed and _resolved_device falls back to device or "".

If _resolve_device raises a typed exception for hardware-unavailability (or exposes a separate is_device_available check), prefer that. Otherwise, consider at least checking str(e) for expected keywords (e.g. "not available", "not installed") before swallowing, and re-raising for unexpected messages.