diff --git a/README.md b/README.md index 664d89f..d203d1a 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ inferencekit is a plugin for Claude Code, Codex and OpenCode, backed by reusable inferencekit gives your agent the playbook and the tools of a senior inference engineer. On every run it works the problem the same way. 1. **Understand the task.** The agent first asks about your goal, deadline, budget and how much the results may change, reusing answers you already gave. It identifies the model, hardware and real inputs, then estimates memory, compute and launch costs per request. Text, image, audio and video use their own metrics and quality checks. -2. **Find and tune an engine.** It researches current implementations, then tunes and benchmarks supported engines (vLLM, SGLang, llama.cpp, TensorRT-LLM and others) for your workload on your hardware. The fastest tested configuration that meets your quality requirements can be the result, or the tuned baseline for further work. +2. **Find and tune an engine.** It searches the web for current engines and settings for your model and hardware, then tunes and benchmarks compatible options for your workload. Engines listed here, such as vLLM, SGLang, llama.cpp and TensorRT-LLM, are examples, not a fixed set. The fastest tested configuration that meets your quality requirements can be the result, or the tuned baseline for further work. 3. **Find the bottleneck.** It selects the profiler for your hardware: Nsight on NVIDIA, ROCm tools on AMD, or Instruments and Metal on Apple. It measures the critical path and investigates memory, compute and launch costs, checking current vendor docs when support is uncertain. 4. **Investigate and implement, if requested.** Search current code, papers, compiler docs and engineering talks; inspect dispatch, IR and assembly when useful. Choose graph execution, fusion, matrix kernels, packing, state reuse or other techniques from measured bottlenecks. Independent implementations can run in parallel; one coordinator integrates changes, and measurements stay serialized per device. Separate authorized devices may run concurrently. 5. **Prove every step.** Every change gets a fast correctness check and a timed screen; milestones get the full proof: interleaved rounds against the tuned baseline, the anti-cheat gates and quality against a reference model. Changes that miss the agreed performance or quality requirements are reverted. @@ -128,7 +128,7 @@ Each check ends PASS, FAIL, UNPROVEN (not enough evidence) or NOT RUN. A missing ## Features -- **9 skills, 4 commands and 5 read-only subagents**, backed by [41 executable CLI/MCP operations](https://github.com/KittenML/inferencekit/tree/main/src/inferencekit/ops/). +- **9 skills, 4 commands and 5 read-only subagents**, backed by [47 executable CLI/MCP operations](https://github.com/KittenML/inferencekit/tree/main/src/inferencekit/ops/). - **Reusable run tools:** [ikstep](https://github.com/KittenML/inferencekit/blob/main/plugin/skills/optimizing-inference/references/remote-host.md) manages jobs on local or remote Linux hosts, readiness checks and concise status output. - **Compact output:** CLI summaries and bounded MCP responses keep full evidence records on disk, with paging for large lists. - **Speed-of-light ceilings:** measured memory bandwidth (`ceiling bw`) and exact bytes per token read from the checkpoint's safetensors headers (`roofline`), including NVFP4, FP8 and mixture-of-experts models. diff --git a/docs/audio-quality.md b/docs/audio-quality.md new file mode 100644 index 0000000..dfe2527 --- /dev/null +++ b/docs/audio-quality.md @@ -0,0 +1,126 @@ +# Speech quality evidence + +`quality audio` compares the complete PCM saved by `bench audio` against a named baseline. It produces `quality_audio` evidence for reports. It uses CPU arithmetic and imported observations. Generate those observations with the optional [local speech scorers](audio-scorers.md), or import your own pinned scorer outputs. + +The acceptance policy is explicit: + +| Policy | Required evidence | Meaning of PASS | +|---|---|---| +| `exact` | Identical PCM, frame counts, rate and channels | Preservation of the named baseline outputs | +| `task_noninferiority` | Paired WER/CER and predicted naturalness; optional speaker metric | Noninferiority under the registered task/proxy metrics | +| `perceptual_and_task_noninferiority` | Task metrics plus blinded, paired listening ratings | Noninferiority under the registered task and listening protocol | + +A baseline's intrinsic intelligibility or suitability is not established by preserving it. Predicted MOS, ASR and speaker scores do not substitute for listening. Timing, lifecycle, memory and sustained reliability remain separate requirements. + +## Register before capture + +Create a JSON policy and register it before either benchmark begins. For an exact-output refactor: + +```json +{"acceptance":"exact","min_fixtures":2,"min_rounds":1,"resamples":2000,"seed":0} +``` + +```bash +inferencekit quality audio-policy --policy audio-policy.json --fixtures speech.jsonl --run-dir runs/speech +# Retain the returned POLICY_ID, then capture both benchmark arms with --save-outputs all. +inferencekit quality audio --baseline BASELINE_ID --candidate CANDIDATE_ID \ + --fixtures speech.jsonl --protocol POLICY_ID --run-dir runs/speech +inferencekit report COMPARISON_ID QUALITY_ID --format md +``` + +The registration binds fixture text, reference-voice content, acceptance, margins, scorer configuration and bootstrap plan. It must precede every measured round, including rounds retained by `--extend`. A late policy produces exploratory results with an UNPROVEN acceptance verdict. The original evidence is never rewritten. + +Source arms must have matching fixture IDs, reference content, round indices, seeds, model revision, generation settings, PCM rate and channels. Different numerical formats require `allow_numerics_change: true` in the policy. This permission does not authorize a different model or generation task. Model execution or hardware spending requires its own authorization. + +## Import task observations + +For task noninferiority, supply `metrics` and pinned `scorers` in the policy. These margins are illustrative, not recommended universal tolerances; register values justified for the actual workload. + +```json +{ + "acceptance": "task_noninferiority", + "min_fixtures": 20, + "min_rounds": 1, + "resamples": 2000, + "seed": 0, + "normalization": "nfkc_casefold_punctuation_v1", + "metrics": { + "wer": {"margin": 0.01, "outlier_wer": 1.0}, + "naturalness": {"margin": 0.1} + }, + "scorers": { + "asr": { + "name": "faster-whisper", "version": "PINNED_PACKAGE_VERSION", + "model": "LOCAL_MODEL_ID", "revision": "CHECKPOINT_CONTENT_HASH", + "config": {"device": "cpu", "compute_type": "int8", "language": "en", + "temperature": 0, "beam_size": 5, "vad_filter": true, + "condition_on_previous_text": false} + }, + "naturalness": { + "name": "UTMOSv2", "version": "PINNED_CODE_REVISION", + "model": "fusion_stage3", "revision": "CHECKPOINT_CONTENT_HASH", + "scale": [1, 5], + "config": {"predict_dataset": "somos", "fold": 0, "num_repetitions": 5, + "seed": 0, "preprocessing": "DECLARE_RESAMPLING_DOWNMIX_AND_WINDOW_RULE"} + } + } +} +``` + +Use local, pinned scorer implementations in their own environment. Faster Whisper returns a segment generator: consume it to finish transcription, and retain the concatenated transcript and actual decoder settings. Its official implementation supports VAD and independent-window decoding. UTMOSv2 exposes dataset/domain selection, checkpoint, fold and repeated predictions; retain these settings and any preprocessing in `config`. See [Faster Whisper](https://github.com/SYSTRAN/faster-whisper) and [UTMOSv2 inference](https://github.com/sarulab-speech/UTMOSv2/blob/main/docs/inference.md). Avoid an unpinned `torch.hub` download during scoring. + +Import one observation per arm, fixture and round. `scorers` must exactly equal the registered descriptor object. Use hashes and seeds from the source benchmark's `requests.jsonl`; bind scores to the original PCM even when the scorer resamples it internally. + +```json +{ + "format": "inferencekit.quality.audio.scores.v1", + "protocol": "POLICY_ID", + "baseline": "BASELINE_ID", + "candidate": "CANDIDATE_ID", + "scorers": {"asr": "COPY_REGISTERED_DESCRIPTOR", "naturalness": "COPY_REGISTERED_DESCRIPTOR"}, + "rows": [{ + "arm": "candidate", "id": "one", "round": 0, "seed": 123, + "pcm_sha256": "HASH_FROM_REQUEST_ROW", "text_sha256": "HASH_FROM_REQUEST_ROW", + "asr": {"text": "The meeting starts at ten tomorrow morning."}, + "naturalness": {"windows": [ + {"start_frame": 0, "end_frame": 44100, "score": 4.0}, + {"start_frame": 44100, "end_frame": 66150, "score": 3.8} + ]} + }] +} +``` + +The shown row and descriptor placeholders are a template, not a complete score file. Include both arms and every measured fixture/round. A scorer failure may retain an `error` string and omit the affected metric; missing required observations remain UNPROVEN. Unknown or duplicate identities, different scorer pins, changed PCM/text hashes, nonfinite scores and invalid windows are rejected. + +```bash +inferencekit quality audio --baseline BASELINE_ID --candidate CANDIDATE_ID \ + --fixtures speech.jsonl --protocol POLICY_ID --scores audio-scores.json --run-dir runs/speech +``` + +WER is recomputed as word edit distance divided by reference words, without a cap. `nfkc_casefold_punctuation_v1` applies Unicode NFKC, case folding, punctuation-to-space and whitespace splitting; `whitespace_v1` only splits whitespace. Neither expands numbers or supplies language-specific tokenization. Choose a suitable declared protocol for the fixture language; empty normalized references are rejected. + +CER uses character edit distance divided by reference characters. Add `"cer": {"margin": 0.01, "outlier_cer": 1.0}` under `metrics`, alongside WER or instead of it. Both reuse the same ASR transcript. CER applies the chosen normalization and removes whitespace; its units are Unicode code points, not grapheme clusters. This permits character scoring for languages without whitespace-delimited words. Margins are fractions, and insertions can make either error rate exceed 1. + +Naturalness windows use original PCM frame coordinates, must be contiguous and cover the whole output exactly once, including the final short window. Scores are weighted by window duration within an utterance. The same method and preprocessing apply to both arms. Scorer provenance is an attestation by the importer: hashes establish binding and reproducibility inputs, not that the external model or listening study was honestly run. + +## ASR outliers + +Register `wer.outlier_wer` or `cer.outlier_cer` for each requested error rate before decoding. When either arm exceeds its threshold, the affected pair needs a corroborating decode of both arms. Register an `asr_retry` descriptor with VAD enabled and `condition_on_previous_text: false`; supply each retry as `asr.retry_text`. Retain both transcripts. The policy does not choose whichever transcript has the lower error rate. + +The paired primary result remains visible. The corroborating result substitutes both affected arms for a second calculation; disagreement between verdicts, or a missing corroborating decode, makes that error-rate check UNPROVEN. WER and CER handle outliers independently. A corroborated regression remains FAIL. High waveform SNR never waives task checks. Structural failures and failures of other required quality checks remain FAIL regardless of ASR uncertainty. + +## Listening and speaker scores + +To require listening, use `perceptual_and_task_noninferiority`, register `metrics.listening` with `margin` and `min_raters` (at least two), and add a `scorers.listening` descriptor with `name`, `version`, `protocol_sha256`, `scale: [1,5]` and `config: {"blinded":true}` plus the study's actual configuration. Each observation supplies `listening.ratings` as `[{"rater":"anonymous-id","score":4.0}, ...]`. The same anonymous panel must rate both arms of each pair. The protocol hash binds the externally retained listening study design; blinding remains an attestation. + +Speaker similarity is optional. Register `metrics.speaker.margin` and a scorer with pinned model/version/revision, preprocessing config and increasing score scale (larger means more similar). Each row supplies `speaker: {"score":0.8,"reference_sha256":"FIXTURE_REFERENCE_HASH"}`. A fixture voice reference is required. For an example local scorer, see the publisher's [SpeechBrain ECAPA model card](https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb). + +## Statistics and reports + +The paired bootstrap resamples fixtures, retaining all rounds within each drawn fixture. WER/CER pool edit counts and reference-word/character counts respectively; naturalness, speaker and listening scores average per-utterance values. Listening inference is conditional on the supplied listener panel. The CI is not a power guarantee: register an adequate fixture count for the intended margin. + +For WER/CER the worse-direction difference is candidate minus baseline; for larger-is-better scores it is baseline minus candidate. PASS requires the entire CI inside the registered degradation margin; FAIL requires the entire CI above it; a straddling CI is UNPROVEN. Incomplete coverage or fewer than the registered fixture/round counts cannot PASS. Exact policy also requires bit-identical PCM for every pair; non-exact policies retain parity as diagnostics. + +SNR is `10*log10(sum(baseline²)/sum((baseline-candidate)²))`; maxabs is the largest signed-sample difference divided by 32768. Both use unmodified, aligned PCM. Unequal lengths have no SNR/maxabs value. Identical PCM has `snr_status: "identical"` and a null numeric SNR, avoiding a fabricated finite or nonfinite JSON value. + +The retained `audio_score.json` lists per-pair parity, source IDs, policy, coverage and checks; `imported_scores.json` retains external observations. Report comparison quality can use only evidence bound to its exact baseline/candidate IDs. Proxy acceptance remains labelled as such; missing/changed observations become UNPROVEN, previous FAIL records remain visible, and lifecycle/reliability requirements are not satisfied by quality evidence. diff --git a/docs/audio-scorers.md b/docs/audio-scorers.md new file mode 100644 index 0000000..66f4cf3 --- /dev/null +++ b/docs/audio-scorers.md @@ -0,0 +1,97 @@ +# Local speech scorers + +Use `quality audio-score` to transcribe saved speech with Faster Whisper, predict naturalness with UTMOSv2, and optionally compare a voice with SpeechBrain ECAPA. It writes observations for [`quality audio`](audio-quality.md), which computes WER/CER, paired confidence intervals and the registered acceptance verdict. Scoring runs after the timed benchmark. + +For output-preserving changes, start with the exact PCM policy: no scorer models are needed. When outputs change, register task metrics and margins before capture. Predicted naturalness and speaker similarity are proxies; keep listening comparisons alongside them. + +## Prepare the scorer environment + +Keep scorer dependencies in a separate Python environment. The CLI itself does not depend on PyTorch or any scorer package. Install pinned versions of Faster Whisper, UTMOSv2 and its dependencies, and SpeechBrain only if speaker checks apply. Download the matching checkpoints separately before scoring: + +| Scorer | Local assets | +|---|---| +| Faster Whisper | Converted model directory, tokenizer and configuration | +| UTMOSv2 | Fusion checkpoint, Wav2Vec2 directory and EfficientNet weights | +| SpeechBrain ECAPA | Model directory including hyperparameters, embedding weights and normalization files | + +Use the upstream [Faster Whisper setup](https://github.com/SYSTRAN/faster-whisper), [UTMOSv2 inference guide](https://github.com/sarulab-speech/UTMOSv2/blob/main/docs/inference.md) and [ECAPA model card](https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb). Model licences and scorer applicability remain those of their publishers. Scoring requires local files and does not download checkpoints. + +Example CPU environment used for the Linux smoke checks: + +```bash +uv venv --python 3.11 .venv-speech +uv pip install --python .venv-speech/bin/python \ + --index-url https://download.pytorch.org/whl/cpu \ + torch==2.6.0 torchaudio==2.6.0 torchvision==0.21.0 +uv pip install --python .venv-speech/bin/python \ + faster-whisper==1.2.1 speechbrain==1.0.3 transformers==4.49.0 \ + timm==1.0.15 librosa==0.11.0 numpy==1.26.4 huggingface-hub==0.30.2 \ + 'utmosv2 @ git+https://github.com/sarulab-speech/UTMOSv2.git@cc2700db57bb83ee13dc31ebe1b868c254e15d09' +``` + +Retain your resolved environment lock with the run. The descriptor records installed versions and hashes; a package version alone does not pin its dependencies. Installation and checkpoint download require network access; the scoring phase does not. + +Save a configuration such as `audio-scorers.json`, replacing paths with your installed environment and assets: + +```json +{ + "format": "inferencekit.quality.audio.scorers.v1", + "python": "/path/to/scorers/bin/python", + "threads": 1, + "seed": 0, + "batch_size": 4, + "timeout_s": 3600, + "scorers": { + "asr": { + "backend": "faster-whisper", + "model": "/path/to/faster-whisper-model", + "settings": { + "language": "en", "beam_size": 5, "compute_type": "int8", + "vad_filter": false, "condition_on_previous_text": false + } + }, + "naturalness": { + "backend": "utmosv2", + "model": "/path/to/fold0_s42_best_model.pth", + "assets": { + "facebook/wav2vec2-base": "/path/to/wav2vec2-base", + "tf_efficientnetv2_s.in21k_ft_in1k": "/path/to/efficientnet-model.safetensors" + }, + "settings": { + "model_name": "fusion_stage3", "window_seconds": 3, + "fold": 0, "checkpoint_seed": 42, "predict_dataset": "sarulab", + "num_repetitions": 1 + } + } + } +} +``` + +Choose the ASR language and UTMOS domain for your workload. Both arms use the same configuration. The adapters hash backend code and model assets and record dependency versions for registration; changing these inputs requires a new registered policy and fresh measurements. + +## Register, measure, score + +1. Run `inferencekit quality audio-scorers audio-scorers.json`. Copy its `scorers` object into your [quality policy](audio-quality.md#import-task-observations), then register it with `quality audio-policy` before capturing either arm. +2. Use a fixed representative fixture set and [`bench rounds --mode audio`](media-benchmarks.md#comparisons-and-proof). Retain complete PCM. The benchmark reports streaming onset, completion, realtime factor and available memory measurements separately from quality. +3. Stop the benchmark engines before running the scorers, so quality work cannot compete with timed requests. Use the final baseline and candidate evidence IDs from the rounds plan. + +```bash +inferencekit quality audio-score --config audio-scorers.json \ + --baseline BASELINE_ID --candidate CANDIDATE_ID \ + --fixtures speech.jsonl --protocol POLICY_ID --out audio-scores.json +inferencekit quality audio --baseline BASELINE_ID --candidate CANDIDATE_ID \ + --fixtures speech.jsonl --protocol POLICY_ID --scores audio-scores.json \ + --run-dir runs/speech +inferencekit report COMPARISON_ID QUALITY_ID --format md +``` + +The score file binds each observation to its source PCM, text, seed, fixture and round. Models are reused within a scoring invocation. Missing or failed observations cannot become a quality PASS. Keep the observations with the evidence; they contain transcripts and are not automatically safe to publish. + +## Optional checks + +- **CER:** add `metrics.cer` to the policy, alongside WER or instead of it. It reuses ASR transcripts and counts normalized Unicode code points with whitespace removed. See [normalization and margins](audio-quality.md#import-task-observations). +- **ASR outliers:** register an `asr_retry` scorer using Faster Whisper with VAD enabled and `condition_on_previous_text: false`. Retain both decodes; the quality evaluator does not choose the best transcript. +- **Voice conditioning:** add a `speaker` scorer with `backend: "speechbrain-ecapa"`, the local model directory and `settings: {}`; register a speaker margin. It compares output with each fixture's `reference_audio`. Omit it when no suitable reference exists. A similarity score does not establish voice identity or consent. +- **Listening:** retain blinded, paired listener ratings under the registered listening protocol. Add them to the generated rows using the [listening schema](audio-quality.md#listening-and-speaker-scores), preserving hashes, descriptors and identities. No adapter generates human ratings. + +UTMOS windows cover the complete output, including the tail, in original PCM coordinates. The quality evaluator weights their scores by duration. Native seeded spectrogram views and short-input tiling are retained; identical PCM windows reuse one prediction. Scorer resampling and downmixing apply only to model inputs; PCM preservation always compares the original bytes. These checks assess the registered fixture set, not every possible utterance or voice. diff --git a/docs/cpu-benchmarks.md b/docs/cpu-benchmarks.md new file mode 100644 index 0000000..f297ca0 --- /dev/null +++ b/docs/cpu-benchmarks.md @@ -0,0 +1,126 @@ +# x86 CPUs (no GPU) + +Read [hardware reference](../plugin/skills/inference-triage/references/hardware.md) first (what the CLI does here, what is NOT RUN). Gathered from project docs (sources at the end, read 14 Oct 2026); the installed version's `--help` is the authority and its startup log the proof. + +## Engines + +| Engine | Notes | +|---|---| +| llama.cpp `llama-server` | The default build is CPU-only (`cmake -B build && cmake --build build --config Release -j`). GGUF weights: a converted artifact, so record its sha256 and the conversion command. | +| vLLM CPU | Image `vllm/vllm-openai-cpu:latest-x86_64` (pin a tag). AVX-512 recommended, AVX2 limited; AMD CPUs need Zen 4 or newer. FP32, FP16 and BF16. Building it needs `numpy<2`. | +| OpenVINO Model Server | Serves OpenVINO IR models (`--source_model`, `--model_repository_path`, `--rest_port`). An IR export, and especially an int4/int8 one, is a different artifact and often a different precision: a separate track unless `precision_allowed` names it. `bench serve` always calls `/v1/chat/completions` (or `/v1/completions`): if the pinned version serves only another path, the harness cannot time it, so it is NOT RUN. | +| IPEX-LLM, intel-extension-for-pytorch | Archived on GitHub: do not start new work on them. | +| TensorRT-LLM | NVIDIA-only: excluded for the hardware, not NOT SUPPORTED (nothing was launched). | +| Any other engine (SGLang, LMDeploy, ...) | Not covered by this guide. Run `engine-scout` with the CPU model and ISA (AVX-512, AMX) in the query before excluding it: a launch decides. | + +Wrappers around llama.cpp (Ollama, LM Studio) add a layer, not a candidate. Include one only as the baseline the person already runs. + +## Freeze a tuned baseline + +Before fixing a CPU baseline, screen thread counts × placement on the declared workload and lead metric. Start with physical cores, then screen SMT explicitly; default thread counts and a single placement are not a tuned baseline. `doctor` records visible ISA, core/SMT/L3/NUMA topology, raw hybrid core types when exposed, affinity, governor, perf policy and host load. Missing facts stay unknown; high host load is a contention warning, not per-core utilization. + +Prepare one baseline manifest whose launch uses `{threads}` and `{port}` in its argv, then generate variants beside it: + +```bash +inferencekit cpu screen-manifest baseline.json --out cpu-screen.json \ + --threads 1 --threads 2 --threads 4 --cpus 0-7 +``` + +The helper launches nothing. It sets `OMP_PLACES=cores` and screens no worker binding, compact (`close`) and spread placement, all inside the selected CPU pool. Choose that pool from the actual topology and leave client cores outside it. Use `--thread-env OMP_NUM_THREADS` when the runtime uses that variable instead of an argv thread count; `--include-smt` allows siblings. Run the generated manifest with `bench rounds`, verify the resolved thread/affinity settings, and retain the best screen configuration for independent final comparisons. The helper does not set NUMA memory policy. + +ISA support does not prove library dispatch. Record the loaded BLAS/oneDNN build and runtime dispatch logs where supported; oneMKL's selected path cannot be inferred from the CPU vendor or package presence. Keep dispatch **UNPROVEN** when the engine provides no such evidence. + +## Launching an OpenAI endpoint + +**Core placement decides CPU results.** Give each engine an explicit core and memory placement in its launch line, and keep the benchmark client off those cores: +- Pin `bench rounds` to spare cores: `taskset -c inferencekit bench rounds ...`. +- Engines inherit that affinity from the runner, so every launch line must set its own: `numactl --physcpubind= --membind= `, or the engine's own binding (below). +- Start with one logical CPU per physical core, then use the screen to test SMT siblings for this workload. + +```bash +# llama.cpp on NUMA node 0, physical cores 0-29 (leave the rest for the client and the OS) +numactl --physcpubind=0-29 --membind=0 llama-server -m -t 30 -tb 30 -c -np 1 \ + --port 8003 --seed 0 --metrics +``` +Pass `--port` explicitly: the default has changed between releases. With more than one NUMA node, see `--numa distribute|isolate|numactl` in the server's help, and drop the page cache before switching modes. + +```bash +# vLLM CPU: tensor parallel = number of NUMA nodes; leave 1-2 cores per node for the server +docker run --rm --network host --security-opt seccomp=unconfined --cap-add SYS_NICE \ + -v :/models -e VLLM_CPU_KVCACHE_SPACE= -e VLLM_CPU_OMP_THREADS_BIND= \ + vllm/vllm-openai-cpu: --model /models/ --served-model-name --port 8001 \ + --dtype bfloat16 --seed 0 --no-enable-prefix-caching +``` +Under `bench rounds`, launch the container through a wrapper that stops it on exit ([container cleanup](../plugin/skills/inference-triage/references/amd-rocm.md), "Containers under `bench rounds`"). A venv install needs `LD_PRELOAD` of tcmalloc and libiomp5, as the vLLM CPU docs describe. + +- **Identity:** use `hardware.backend: cpu` and `hardware.cpu.model` in the run manifest. CPU rounds record a CPU identity, not a GPU name; direct media clients use `--device cpu --hardware-name` with the server CPU model. +- **Price:** `hardware.price_usd_h` is the whole host's price. CPU load/capacity mode remains unsupported; do not label CPU throughput per GPU. + +## Proving settings in effect + +- **llama.cpp:** `system_info` (the SIMD flags it was built with, e.g. AVX512, AMX), the thread counts, `n_ctx`, `type_k`/`type_v`, and the slot count. +- **Expected HARD GATE lines.** On a CPU host, `bench rounds` tags rows `[HARD GATE]` for GPU settings that cannot apply: llama.cpp's `no GPU backend in this llama.cpp build (system_info)` and `only 0/N layers offloaded to the GPU`, vLLM CPU's `CUDA graphs off (...)`, and `fallback attention backend (TORCH_SDPA)` if its log names that backend. Report each as "not applicable on x86 CPU" next to the row. Any other HARD GATE is still a FAIL ([hardware reference](../plugin/skills/inference-triage/references/hardware.md), note 3). +- **vLLM CPU:** the engine config line (dtype, KV cache space) and the thread-binding lines it prints for `VLLM_CPU_OMP_THREADS_BIND`. If the binding is not printed, check the threads' placement while it runs: `ps -L -o pid,lwp,psr,comm -p ` (`psr` is the core each thread is on). +- **Placement:** record the `numactl`/`taskset` line and `numactl -H` in `runs//hw/`. + +## Profilers and how to read them + +Run them under `env -i` with an allowlist ([hardware reference](../plugin/skills/inference-triage/references/hardware.md), section 5). `doctor` reads `perf_event_paranoid` and capabilities without probing counters: a policy that allows user-space events (commonly paranoid ≤ 2) does not prove container access. Use `perf` when permitted; otherwise use [engine timers or CPU-only torch.profiler](#cpu-profiling-fallback). Do not change host policy as part of profiling. + +| Question | Tool | Command | +|---|---|---| +| How close to memory bandwidth is decode? | likwid-perfctr, MEM group | `likwid-perfctr -c -g MEM -S 30s` while steady decode runs: memory bandwidth in MBytes/s for those cores | +| Memory-bound or compute-bound? | perf stat | `perf stat -e cycles,instructions,cache-misses -p -- sleep 30` | +| Where does the time go (functions)? | perf record | `perf record -F 99 -g -p -- sleep 30`, then `perf report --no-children` or `perf script report flamegraph` | +| DRAM-bound share, bandwidth histogram, remote NUMA traffic | VTune memory-access | `vtune -collect memory-access -r vt -- ` | + +Reading them with the profiling skill's six questions: +- **Bandwidth used.** likwid's MEM bandwidth during decode ÷ the measured peak (below) is the CPU's "% of peak" (derived). Near the peak, only fewer bytes per token help, as on a GPU. +- **Compute-bound decode.** Low memory bandwidth together with high instructions per cycle (`instructions / cycles`, roughly above 2) means compute-bound: dequantising Q4/Q8 blocks, or narrow vector units (AVX2 rather than AVX-512/AMX). The bandwidth ceiling is then not the binding one: say so, inferred. +- **Threading problems.** Large shares in OpenMP spin or barrier functions (`__kmp_*`, `gomp_*`) in `perf report` mean too many threads, SMT siblings, or uneven work. +- **NUMA.** VTune's "NUMA: % of Remote Accesses" should be near zero when each rank and its memory sit on one node. +- **Page faults** in the first requests mean the weights are memory-mapped and not resident yet. llama.cpp's `--mlock` keeps them in RAM; the harness's warm-up requests are not timed. + +## Bandwidth and the ceiling + +Measure all the memory the engine uses: every socket the engine runs on, with one thread per physical core. +- **Read-only first** (decode is read-dominant): + - `likwid-bench -a` lists the kernels; use the widest read-only one (`load_avx512`, `load_avx` or `load`), one workgroup per socket: `likwid-bench -t load_avx512 -w S0:4GB -w S1:4GB`. + - Intel MLC (Intel CPUs, needs root and the `msr` module): `mlc --peak_injection_bandwidth`, then quote the "ALL Reads" line. `mlc --bandwidth_matrix` shows bandwidth between NUMA nodes. +- **STREAM** when those are unavailable: + ```bash + gcc -O3 -fopenmp -mcmodel=medium -DSTREAM_ARRAY_SIZE= stream.c -o stream + OMP_NUM_THREADS= OMP_PROC_BIND=spread OMP_PLACES=cores ./stream + ``` + Each array must be at least 4x the sum of all last-level caches. STREAM counts Triad as 24 bytes per iteration and has no read-only kernel, so quote Triad and label it "read-only bandwidth not measured; % of ceiling may be overstated". +- **Datasheet fallback:** channels × transfer rate × 8 bytes per socket, e.g. 8 channels of DDR5-4800 = 307.2 GB/s (derived, labelled "datasheet-based; measured ceiling NOT RUN"). +- Then `inferencekit roofline --model --dtype --gpu-bw --context ` per bucket. `--dtype` covers bf16, fp8 and q8_0 only. For other GGUF types (Q4_K and so on), derive bytes per token by hand from the tensor sizes that `gguf-dump` (llama.cpp's `gguf` Python package) lists: every weight tensor once, the token-embedding table as one row per token, plus the KV term. Show the arithmetic (derived). + +## Pitfalls + +- **Each vLLM rank must fit on one NUMA node,** or it is killed with `exitcode 9`. Set tensor parallel to the number of NUMA nodes. +- **In Docker, `get_mempolicy` errors** mean the container needs `--cap-add SYS_NICE`. +- **Frequency and power.** Record the governor and boost state (`cpupower frequency-info`). Ask before changing them. A long run on a hot or power-capped host drifts, and the balanced round order spreads that drift over every engine. +- **Nothing else on the engine's cores,** the benchmark client included (above). +- **Same precision:** BF16 in vLLM CPU and a BF16 GGUF compare; Q8_0 and an OpenVINO int8 export do not (rule 8). + +## Sources (read 14 Oct 2026) + +- llama.cpp server: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md; build: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md +- vLLM CPU: https://docs.vllm.ai/en/latest/getting_started/installation/cpu.html +- OpenVINO Model Server LLM quickstart: https://github.com/openvinotoolkit/model_server/blob/main/docs/llm/quickstart.md +- IPEX-LLM: https://github.com/intel/ipex-llm; intel-extension-for-pytorch: https://github.com/intel/intel-extension-for-pytorch +- perf: https://perfwiki.github.io/main/tutorial/; VTune memory access: https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2025-0/memory-access-analysis.html +- likwid: https://github.com/RRZE-HPC/likwid/wiki/Likwid-Bench; STREAM: https://www.cs.virginia.edu/stream/ref.html; Intel MLC: https://www.intel.com/content/www/us/en/support/articles/000055898.html + +# CPU profiling fallback + +Use the [CPU thread and placement screen](#freeze-a-tuned-baseline) before fixing the baseline. Keep the same model, inputs, thread count, CPU affinity and NUMA placement for the diagnostic run. `doctor` supplies OS facts and a route; it does not test profiler access or establish library dispatch. + +When Linux permits process-level events, use a short steady-state `perf record` for function attribution and `perf stat` for supported counters. Request user-space events when policy excludes kernel activity. `perf_event_paranoid ≤ 2` or CAP_PERFMON suggests permission; container/security policy can still refuse it. Check installed help, preserve the actual error, and use the fallback below if unavailable. Do not change sysctls or capabilities to make a capture work. [Kernel permission rules](https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html). + +When perf is blocked or missing, use existing engine per-layer timers or `torch.profiler` with `activities=[ProfilerActivity.CPU]`. Warm up first, bound the capture to representative steady requests, and keep stack/shape recording off unless needed. Timers cover only instrumented regions; tensor operation totals can overlap across threads, so they are not wall-clock critical-path shares. [PyTorch profiler](https://docs.pytorch.org/docs/stable/profiler.html). + +Capture with downloaded offline weights and an allowlisted environment, retaining only the CPU runtime settings needed to match the benchmark. Logs/traces may include paths, commands and input data; keep them local under the run directory. Inspect thread placement, barriers/spin time, memory allocation and the loaded library/kernel dispatch. Missing counter or dispatch evidence remains NOT RUN or UNPROVEN. + +Profiles explain a mechanism. Turn instrumentation off and use independent interleaved client-timed measurements for any speedup claim. diff --git a/docs/media-benchmarks.md b/docs/media-benchmarks.md index ebdafef..662cc0c 100644 --- a/docs/media-benchmarks.md +++ b/docs/media-benchmarks.md @@ -64,7 +64,18 @@ The client reports first PCM arrival, first nonzero signal arrival, simulated no ## Comparisons and proof -These commands do not restart engines. For a published comparison, an external driver must start each engine fresh per round, retain process/startup records and run balanced interleaved A/B rounds on an otherwise idle device. Run one round per invocation and append each engine’s record with `--extend EVIDENCE_ID`; continuation requires identical configuration and intact prior artifacts. Use the same fixture set, precision, generation settings and round seeds for both engines. Keep each engine's endpoint stable between restarts. +Use `bench rounds` to launch each engine fresh per round, check readiness, warm up, measure and stop its process group in balanced order. Set manifest `workload.modes` to `["audio"]` or `["image"]`; `--mode auto` also recognizes modality and endpoint declarations. Engines use `audio_http` or `image_http` and default to `/health` readiness. Keep the same fixture set, precision, generation settings and round seeds for both engines. + +```bash +inferencekit bench rounds --manifest manifest.json --prompts speech.jsonl --mode audio --rounds 4 +# Use the two final arm IDs from plan.audio_evidence_final for compare and quality audio. +``` + +The manifest supplies `model.id`, `model.revision`, `model.precision`, each engine's version/launch/environment/port, and `workload.quant` (`none` for unquantized). Audio also accepts `workload.sample_rate`, `channels`, `seed` and `settings`; image settings stay in each fixture. `--serve-args` can supply client timeouts and SLOs but cannot override the scheduler's identity, seeds or round indices. `--mode clip` uses the video client. + +For CPU servers, set `hardware` to `{"backend":"cpu","cpu":{"model":"Exact CPU model"},"count":1}`. The runner uses a CPU-exclusive advisory lock and records sampled process-group RSS, OS thread counts, affinity and selected runtime environment controls. Requested thread settings do not prove a runtime used that many workers. RSS is host RAM, may miss brief peaks and counts shared pages per process; it is not VRAM or swap. CPU load/capacity mode is not supported. Direct CPU media calls use `--device cpu --hardware-name 'Exact CPU model'`. + +Manual drivers can still use one `bench audio`/`bench image` round per invocation and `--extend EVIDENCE_ID`. Continuation requires identical configuration and intact artifacts. For audio ports that change on restart, set the same `--endpoint-id` from the first round; otherwise keep the endpoint fixed. Scheduler-final records are immutable; start a new scheduler run instead of extending them. ```bash inferencekit compare BASELINE_ID CANDIDATE_ID --metric image_ms @@ -72,7 +83,9 @@ inferencekit compare BASELINE_ID CANDIDATE_ID --metric first_audio_p95_ms inferencekit report BASELINE_ID CANDIDATE_ID COMPARISON_ID --format md ``` -`compare` requires matching request hashes and pinned model identities, at least three complete matching pairs, interleaved non-overlapping timing windows, and no structural failures. Use at least four rounds for a balanced two-engine final run. The paired bootstrap CI measures timing only. Fresh-process lifecycle, memory fit, semantic quality and repeated-run reliability require separate evidence; the report preserves these as UNPROVEN and propagates failed speech SLOs. Quality gates must use the exact timed configuration and the pre-registered modality protocol. Cross-precision results are format trade-offs and require separate quality reporting. +`compare` requires matching request hashes and pinned model identities, at least three complete matching pairs, interleaved non-overlapping timing windows, and no structural failures. Use at least four rounds for a balanced two-engine final run. Missing scheduled rounds prevent a conclusive comparison; manual chains can declare `--expected-rounds 4` to check missing tail rounds. Known memory measurements appear beside timing, with their method and coverage. + +The paired bootstrap CI measures timing only. Scheduler-final records can establish fresh-process lifecycle. Memory fit, semantic quality and sustained reliability need their own evidence; failed speech SLOs remain failures. Register speech acceptance before capture with `quality audio-policy`, then evaluate saved PCM using [`quality audio`](audio-quality.md). Cross-precision results remain format trade-offs requiring separate quality reporting. Keep output artifacts for quality evaluation (`--save-outputs all`, the default). Retain cold-start/compile timing, whole-device and process memory, pressure/swap, sustained long outputs, task quality and mechanism profiles separately. Do not infer a smaller-memory-device result from a run on larger hardware. diff --git a/plugin/skills/benchmarking/SKILL.md b/plugin/skills/benchmarking/SKILL.md index 1aa7406..b366197 100644 --- a/plugin/skills/benchmarking/SKILL.md +++ b/plugin/skills/benchmarking/SKILL.md @@ -11,6 +11,8 @@ metadata: Choose the workload before the metric: complete-request latency, sustained throughput, peak memory or cost at the agreed quality. Use the image/audio route in [affordability](../optimizing-inference/references/affordability.md) and [video checks](references/video-checks.md) for those modalities. Token metrics, text load grids and GSM8K below apply to text; other tasks need a registered adapter, with unsupported checks NOT RUN. +`bench rounds --mode audio|image|clip` runs fresh-process media comparisons; CPU manifests use `hardware.backend: cpu` for a CPU lock and process-group telemetry. Compare the final evidence ID for each engine in the run plan; [media contracts and round scheduling](https://github.com/KittenML/inferencekit/blob/main/docs/media-benchmarks.md). + Speed numbers that survive review: a timer outside the engine, one precision for every engine, settings proven in effect, warm engines, fresh-process rounds in a balanced order, confidence intervals, a speed-of-light ceiling and anti-cheat gates. [optimizing-inference](../optimizing-inference/SKILL.md) calls this skill. **Use** for latency and tok/s per input bucket, capacity (`N*` within an SLO, goodput, $/1M tokens), comparisons of engines, configs or builds, and any reported number. A change-loop step needs only a screen; the full protocol is for milestones (fixed rounds: [references/statistics.md](references/statistics.md)) and anything published ([optimizing-kernels](../optimizing-kernels/SKILL.md), "Checks"). **Not** for *why* it is slow ([profiling](../profiling/SKILL.md)), nor while builds' outputs differ unchecked ([proving-correctness](../proving-correctness/SKILL.md) first). diff --git a/plugin/skills/inference-triage/references/cpu.md b/plugin/skills/inference-triage/references/cpu.md index b8c5384..77d5f3f 100644 --- a/plugin/skills/inference-triage/references/cpu.md +++ b/plugin/skills/inference-triage/references/cpu.md @@ -1,99 +1,45 @@ -# x86 CPUs (no GPU) +# CPU inference -Read [hardware.md](hardware.md) first (what the CLI does here, what is NOT RUN). Gathered from project docs (sources at the end, read 14 Oct 2026); the installed version's `--help` is the authority and its startup log the proof. +Read [hardware.md](hardware.md) first. Use `doctor` for CPU ISA, physical/SMT cores, L3 groups, raw hybrid core types, NUMA, affinity, governor, perf policy and host load. Unknown facts stay unknown. Host load is a contention warning, not per-core utilization. ISA support and package presence do not prove BLAS/oneDNN dispatch: retain loaded-library and runtime dispatch logs, otherwise mark it UNPROVEN. -## Engines +## Freeze a tuned baseline -| Engine | Notes | +Screen thread counts × placement for the declared workload and lead metric before fixing a CPU baseline. Start with physical cores, then test SMT; defaults are not a tuned baseline. Prepare one manifest whose engine argv uses `{threads}` and `{port}`: + +```bash +inferencekit cpu screen-manifest baseline.json --out cpu-screen.json \ + --threads 1 --threads 2 --threads 4 --cpus 0-7 +``` + +Choose CPUs from actual topology and reserve others for the client. The helper launches nothing: it generates no-binding, compact and spread variants inside the selected CPU pool, with `OMP_PLACES=cores`. Use `--thread-env OMP_NUM_THREADS` instead of `{threads}` when appropriate; `--include-smt` permits siblings. Run the generated manifest with `bench rounds`, verify effective settings, select the best screen configuration, then collect independent final comparisons. Pinning controls affinity, not NUMA memory policy. + +Use manifest `hardware.backend: cpu` and `hardware.cpu.model`. CPU rounds use an advisory CPU lock, record requested runtime controls plus observed OS thread counts/affinity, and sample process-group RSS. Requested counts do not prove effective runtime pools. RSS may miss peaks and counts shared pages per process; it is not VRAM, swap or a leak test. CPU load/capacity mode is unsupported. + +## Engine selection and launch + +| Candidate | What to verify | |---|---| -| llama.cpp `llama-server` | The default build is CPU-only (`cmake -B build && cmake --build build --config Release -j`). GGUF weights: a converted artifact, so record its sha256 and the conversion command. | -| vLLM CPU | Image `vllm/vllm-openai-cpu:latest-x86_64` (pin a tag). AVX-512 recommended, AVX2 limited; AMD CPUs need Zen 4 or newer. FP32, FP16 and BF16. Building it needs `numpy<2`. | -| OpenVINO Model Server | Serves OpenVINO IR models (`--source_model`, `--model_repository_path`, `--rest_port`). An IR export, and especially an int4/int8 one, is a different artifact and often a different precision: a separate track unless `precision_allowed` names it. `bench serve` always calls `/v1/chat/completions` (or `/v1/completions`): if the pinned version serves only another path, the harness cannot time it, so it is NOT RUN. | -| IPEX-LLM, intel-extension-for-pytorch | Archived on GitHub: do not start new work on them. | -| TensorRT-LLM | NVIDIA-only: excluded for the hardware, not NOT SUPPORTED (nothing was launched). | -| Any other engine (SGLang, LMDeploy, ...) | Not covered by this guide. Run `engine-scout` with the CPU model and ISA (AVX-512, AMX) in the query before excluding it: a launch decides. | +| llama.cpp | CPU build, GGUF artifact hash and conversion command; installed server help and startup log. | +| vLLM CPU | Pinned build/image, supported ISA/precision, KV cache and actual thread binding. | +| OpenVINO Model Server | IR export/quantization identity and endpoint compatibility: `bench serve` needs `/v1/chat/completions` or `/v1/completions`. | +| Other engines | Research current CPU support with the exact CPU and ISA before exclusion; a launch decides support. | -Wrappers around llama.cpp (Ollama, LM Studio) add a layer, not a candidate. Include one only as the baseline the person already runs. +Ollama/LM Studio wrappers matter when they are the user's baseline. Do not start new work on archived IPEX projects; TensorRT-LLM is excluded on CPU hardware. Format changes are separate tracks unless explicitly permitted: a BF16 GGUF can match BF16 weights, while Q8_0 and an int8 IR export are not the same format. -## Launching an OpenAI endpoint +Keep the client off the engine cores. Engines inherit runner affinity, so launch them with explicit `taskset`, `numactl` or verified runtime binding. Record CPU and NUMA memory placement. Start with one logical CPU per core and measure SMT, rather than assuming it helps. Explicit ports and clean environment are required; container launch wrappers must stop the container on exit. -**Core placement decides CPU results.** Give each engine an explicit core and memory placement in its launch line, and keep the benchmark client off those cores: -- Pin `bench rounds` to spare cores: `taskset -c inferencekit bench rounds ...`. -- Engines inherit that affinity from the runner, so every launch line must set its own: `numactl --physcpubind= --membind= `, or the engine's own binding (below). -- Use one logical CPU per physical core. SMT siblings slow inference down. +For llama.cpp retain build SIMD flags, thread counts, context, KV types and slots. For vLLM retain dtype, KV space and binding logs; inspect thread placement when logs omit it. Existing GPU-specific log gates may flag missing CUDA/offload on CPU: identify those as not applicable, preserve every other failure. Do not dismiss actual correctness failures. -```bash -# llama.cpp on NUMA node 0, physical cores 0-29 (leave the rest for the client and the OS) -numactl --physcpubind=0-29 --membind=0 llama-server -m -t 30 -tb 30 -c -np 1 \ - --port 8003 --seed 0 --metrics -``` -Pass `--port` explicitly: the default has changed between releases. With more than one NUMA node, see `--numa distribute|isolate|numactl` in the server's help, and drop the page cache before switching modes. +## Profiling -```bash -# vLLM CPU: tensor parallel = number of NUMA nodes; leave 1-2 cores per node for the server -docker run --rm --network host --security-opt seccomp=unconfined --cap-add SYS_NICE \ - -v :/models -e VLLM_CPU_KVCACHE_SPACE= -e VLLM_CPU_OMP_THREADS_BIND= \ - vllm/vllm-openai-cpu: --model /models/ --served-model-name --port 8001 \ - --dtype bfloat16 --seed 0 --no-enable-prefix-caching -``` -Under `bench rounds`, launch the container through a wrapper that stops it on exit ([amd-rocm.md](amd-rocm.md), "Containers under `bench rounds`"). A venv install needs `LD_PRELOAD` of tcmalloc and libiomp5, as the vLLM CPU docs describe. - -- **Identity:** `--serve-args "--gpu-name 'CPU 2x Xeon 6980P' --gpu-count 1"`. `hardware.gpu` in the intake is free text: name the CPU there too, with the socket count. -- **Price:** `hardware.price_usd_h` is the whole host's price, so $/1M tokens stays comparable with GPU hosts. - -## Proving settings in effect - -- **llama.cpp:** `system_info` (the SIMD flags it was built with, e.g. AVX512, AMX), the thread counts, `n_ctx`, `type_k`/`type_v`, and the slot count. -- **Expected HARD GATE lines.** On a CPU host, `bench rounds` tags rows `[HARD GATE]` for GPU settings that cannot apply: llama.cpp's `no GPU backend in this llama.cpp build (system_info)` and `only 0/N layers offloaded to the GPU`, vLLM CPU's `CUDA graphs off (...)`, and `fallback attention backend (TORCH_SDPA)` if its log names that backend. Report each as "not applicable on x86 CPU" next to the row. Any other HARD GATE is still a FAIL ([hardware.md](hardware.md), note 3). -- **vLLM CPU:** the engine config line (dtype, KV cache space) and the thread-binding lines it prints for `VLLM_CPU_OMP_THREADS_BIND`. If the binding is not printed, check the threads' placement while it runs: `ps -L -o pid,lwp,psr,comm -p ` (`psr` is the core each thread is on). -- **Placement:** record the `numactl`/`taskset` line and `numactl -H` in `runs//hw/`. - -## Profilers and how to read them - -Run them under `env -i` with an allowlist ([hardware.md](hardware.md), section 5). `perf` needs `kernel.perf_event_paranoid` at 1 or lower, a host setting: ask before changing it. - -| Question | Tool | Command | -|---|---|---| -| How close to memory bandwidth is decode? | likwid-perfctr, MEM group | `likwid-perfctr -c -g MEM -S 30s` while steady decode runs: memory bandwidth in MBytes/s for those cores | -| Memory-bound or compute-bound? | perf stat | `perf stat -e cycles,instructions,cache-misses -p -- sleep 30` | -| Where does the time go (functions)? | perf record | `perf record -F 99 -g -p -- sleep 30`, then `perf report --no-children` or `perf script report flamegraph` | -| DRAM-bound share, bandwidth histogram, remote NUMA traffic | VTune memory-access | `vtune -collect memory-access -r vt -- ` | - -Reading them with the profiling skill's six questions: -- **Bandwidth used.** likwid's MEM bandwidth during decode ÷ the measured peak (below) is the CPU's "% of peak" (derived). Near the peak, only fewer bytes per token help, as on a GPU. -- **Compute-bound decode.** Low memory bandwidth together with high instructions per cycle (`instructions / cycles`, roughly above 2) means compute-bound: dequantising Q4/Q8 blocks, or narrow vector units (AVX2 rather than AVX-512/AMX). The bandwidth ceiling is then not the binding one: say so, inferred. -- **Threading problems.** Large shares in OpenMP spin or barrier functions (`__kmp_*`, `gomp_*`) in `perf report` mean too many threads, SMT siblings, or uneven work. -- **NUMA.** VTune's "NUMA: % of Remote Accesses" should be near zero when each rank and its memory sit on one node. -- **Page faults** in the first requests mean the weights are memory-mapped and not resident yet. llama.cpp's `--mlock` keeps them in RAM; the harness's warm-up requests are not timed. - -## Bandwidth and the ceiling - -Measure all the memory the engine uses: every socket the engine runs on, with one thread per physical core. -- **Read-only first** (decode is read-dominant): - - `likwid-bench -a` lists the kernels; use the widest read-only one (`load_avx512`, `load_avx` or `load`), one workgroup per socket: `likwid-bench -t load_avx512 -w S0:4GB -w S1:4GB`. - - Intel MLC (Intel CPUs, needs root and the `msr` module): `mlc --peak_injection_bandwidth`, then quote the "ALL Reads" line. `mlc --bandwidth_matrix` shows bandwidth between NUMA nodes. -- **STREAM** when those are unavailable: - ```bash - gcc -O3 -fopenmp -mcmodel=medium -DSTREAM_ARRAY_SIZE= stream.c -o stream - OMP_NUM_THREADS= OMP_PROC_BIND=spread OMP_PLACES=cores ./stream - ``` - Each array must be at least 4x the sum of all last-level caches. STREAM counts Triad as 24 bytes per iteration and has no read-only kernel, so quote Triad and label it "read-only bandwidth not measured; % of ceiling may be overstated". -- **Datasheet fallback:** channels × transfer rate × 8 bytes per socket, e.g. 8 channels of DDR5-4800 = 307.2 GB/s (derived, labelled "datasheet-based; measured ceiling NOT RUN"). -- Then `inferencekit roofline --model --dtype --gpu-bw --context ` per bucket. `--dtype` covers bf16, fp8 and q8_0 only. For other GGUF types (Q4_K and so on), derive bytes per token by hand from the tensor sizes that `gguf-dump` (llama.cpp's `gguf` Python package) lists: every weight tensor once, the token-embedding table as one row per token, plus the KV term. Show the arithmetic (derived). - -## Pitfalls - -- **Each vLLM rank must fit on one NUMA node,** or it is killed with `exitcode 9`. Set tensor parallel to the number of NUMA nodes. -- **In Docker, `get_mempolicy` errors** mean the container needs `--cap-add SYS_NICE`. -- **Frequency and power.** Record the governor and boost state (`cpupower frequency-info`). Ask before changing them. A long run on a hot or power-capped host drifts, and the balanced round order spreads that drift over every engine. -- **Nothing else on the engine's cores,** the benchmark client included (above). -- **Same precision:** BF16 in vLLM CPU and a BF16 GGUF compare; Q8_0 and an OpenVINO int8 export do not (rule 8). - -## Sources (read 14 Oct 2026) - -- llama.cpp server: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md; build: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md -- vLLM CPU: https://docs.vllm.ai/en/latest/getting_started/installation/cpu.html -- OpenVINO Model Server LLM quickstart: https://github.com/openvinotoolkit/model_server/blob/main/docs/llm/quickstart.md -- IPEX-LLM: https://github.com/intel/ipex-llm; intel-extension-for-pytorch: https://github.com/intel/intel-extension-for-pytorch -- perf: https://perfwiki.github.io/main/tutorial/; VTune memory access: https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2025-0/memory-access-analysis.html -- likwid: https://github.com/RRZE-HPC/likwid/wiki/Likwid-Bench; STREAM: https://www.cs.virginia.edu/stream/ref.html; Intel MLC: https://www.intel.com/content/www/us/en/support/articles/000055898.html +Match the timed model, inputs, threads, affinity and NUMA placement. `doctor` reads policy without testing counter access. Use user-space `perf record`/`perf stat` when permitted; paranoid ≤2 or CAP_PERFMON suggests access but container policy may still block it. Preserve errors and fall back to engine per-layer timers or `torch.profiler` with `ProfilerActivity.CPU`. Do not change sysctls/capabilities to obtain access. + +Use downloaded offline weights and an allowlisted environment. Warm up, bound the capture, keep stack/shape collection off unless needed, and retain traces locally. Inspect allocation, barriers/spin, placement, library dispatch and page faults. Summed operation time can overlap across threads; it is not wall-clock critical-path time. Profiles explain mechanisms; turn instrumentation off for independent timed proof. + +## Bandwidth and host controls + +Measure read bandwidth across the sockets used by the engine, preferably with a supported read-only LIKWID or MLC kernel. STREAM is a fallback: arrays exceed 4× total LLC; Triad counts 24 bytes/iteration and is not read-only bandwidth, so qualify the ceiling. Datasheet bandwidth remains an estimate. Feed measured GB/s to `roofline`; unsupported GGUF formats need explicit tensor/KV byte accounting. + +Record governor/boost and ask before changes. Keep other work off engine cores. Respect NUMA capacity and runtime constraints; container permission changes need their own authorization. + +Detailed launch commands, profiler examples, bandwidth arithmetic and primary sources: [CPU benchmark guide](https://github.com/KittenML/inferencekit/blob/main/docs/cpu-benchmarks.md). Check installed help and current official engine documentation before using version-specific flags. diff --git a/plugin/skills/optimizing-inference/SKILL.md b/plugin/skills/optimizing-inference/SKILL.md index d9c472e..c079f49 100644 --- a/plugin/skills/optimizing-inference/SKILL.md +++ b/plugin/skills/optimizing-inference/SKILL.md @@ -31,6 +31,10 @@ No plan or plan approval is required by default. Show or create one only when re ## Start or resume +Existing run: first call `inferencekit intake resume runs/`; inspect files for reported issues or required details ([state.md](references/state.md)). + +Fresh run: + ```bash inferencekit intake template > intake.yaml # only when no intake was supplied inferencekit intake validate intake.yaml --allow-placeholders @@ -39,7 +43,7 @@ export INFERENCEKIT_RUNS_DIR="$PWD/runs//evidence" inferencekit intake budget runs/ ``` -Use the derived run id; `--run-dir` starts the clock and pins the intake. Resume from [state.md](references/state.md). With SSH, runtime and client run on the device host ([remote-host.md](references/remote-host.md)). +Use the derived run id; `--run-dir` pins the intake and starts the clock. SSH runtime and client share the [device host](references/remote-host.md). ## 1. Define the actual task @@ -59,7 +63,7 @@ Estimate critical-path share, attainable efficiency, exact byte savings and perm ## 3. Establish the strongest baseline -Use [candidates.md](references/candidates.md) and the research queue. Launch compatible engines in isolated environments; record versions, commands, resolved configs and smoke results. Text engines, image pipelines, speech runtimes and video pipelines have different candidates. NVIDIA graphs are not a generic launch flag for AMD or Apple. +Search the web at the start of each run for current engines and recipes for the exact model, hardware and goal; refresh when these or the runtime change. [candidates.md](references/candidates.md) gives examples, not a fixed list or count. Launch compatible engines in isolated environments; record sources, versions, commands, resolved configs and smoke results. NVIDIA graphs are not a generic launch flag for AMD or Apple. [Tune](references/tuning.md) supported engines for the goal. A task-fixed baseline is labelled fixed, never automatically best in the field. Reconsider numerical-only baseline rejects under a newly authorized lossy policy. diff --git a/plugin/skills/optimizing-inference/references/candidates.md b/plugin/skills/optimizing-inference/references/candidates.md index 33405e3..96c9480 100644 --- a/plugin/skills/optimizing-inference/references/candidates.md +++ b/plugin/skills/optimizing-inference/references/candidates.md @@ -1,6 +1,10 @@ # Candidate engines -Pick about 5 engines for the modality and GPU, then **verify each by launching it** on the pinned local checkpoint, at its precision, with a smoke request. Everything below is a starting point: flags move between releases, **the installed version's `--help` is the authority**, and the startup log is the proof. +Search the web before selecting baselines for each run: use the exact model/architecture, modality, precision, workload and hardware. Check current official releases, support tables, repositories and vendor/model recipes; follow benchmark claims to their configurations and measurement conditions. Discover applicable engines beyond the examples below. There is no fixed engine list or count. + +Choose the strongest applicable candidates within the approved scope and budget. Record source URLs/access dates and reasons for exclusions or untested candidates in the existing research note. Recheck freshness and compatibility before reusing a previous survey. If web access is unavailable, state that current coverage is unverified; do not claim the latest or best current baseline. + +**Verify each candidate by launching it** on the pinned local checkpoint, at its precision, with a smoke request, then tune and measure it. Web rankings do not prove which engine wins locally. Keep versions, workloads and quality checks pinned during each comparison. Flags move between releases: **the installed version's `--help` is the authority**, and the startup log is the proof. ## Common procedure (every engine) @@ -84,7 +88,7 @@ Pick about 5 engines for the modality and GPU, then **verify each by launching i - Block reuse (prefix caching) may default to on; turn it off. - Support for a new GPU architecture or a new quantization format often lands in release candidates first. That is the "one documented fix". -### Fifth candidate (one of these, when it applies) +### Additional examples, when applicable - **LMDeploy:** `lmdeploy serve api_server --server-port 8005 --backend turbomind|pytorch`. The resolved config is logged at startup. KV quantization (`--quant-policy`) is a precision change. - **ExLlamaV3 via TabbyAPI:** EXL3 weights are a different precision, so it is a separate track only. - **A custom engine** (`candidates.include`) enters like the others: a reproducible install (git URL, commit, build commands) and the [custom-engine contract](#custom-engine-contract) below, so it can be gated. @@ -106,16 +110,16 @@ A custom or patched engine serves all of this, or the harness cannot time, gate ## Other hardware (AMD ROCm, x86 CPU, Apple Silicon) -NVIDIA is fully supported; on other hardware the skills guide the flow and the CLI does less ([hardware.md](../../inference-triage/references/hardware.md)). The common procedure above holds. The candidates: +NVIDIA is fully supported; on other hardware the skills guide the flow and the CLI does less ([hardware.md](../../inference-triage/references/hardware.md)). The common procedure above holds. Recheck these examples and exclusions against current support: -| Platform | Candidates to try | Excluded for the hardware | +| Platform | Candidate examples | Exclusions to recheck | |---|---|---| | AMD ROCm (MI300X, MI325X, MI350 series, Radeon) | vLLM (ROCm image, AITER on), SGLang (ROCm image), llama.cpp (HIP build), a custom engine | TensorRT-LLM | | x86 CPU | llama.cpp `llama-server`, vLLM CPU, OpenVINO Model Server, a custom engine | TensorRT-LLM | | Apple Silicon | mlx-lm `mlx_lm.server`, llama.cpp (Metal), a custom engine | TensorRT-LLM | | Intel GPU, Google TPU | [other-accelerators.md](../../inference-triage/references/other-accelerators.md) | TensorRT-LLM | -- **Fewer than about 5 is fine here.** Record each excluded engine with the reason "not available on ". That is not NOT SUPPORTED, because nothing was launched. Any engine not in the table gets an `engine-scout` run with the platform in the query, and a launch decides. +- Record each exclusion with its source and hardware-specific reason. An engine that was not launched is untested, not NOT SUPPORTED. Research candidates outside this table with `engine-scout`, including the hardware in the query; a launch decides support. - **Launch lines, the settings to prove and the pitfalls** per platform: [amd-rocm.md](../../inference-triage/references/amd-rocm.md), [cpu.md](../../inference-triage/references/cpu.md), [apple-silicon.md](../../inference-triage/references/apple-silicon.md). - **Identity in every `bench rounds` call:** `--serve-args "--gpu-name --gpu-count "`. - **Precision names hide formats:** FP8 on MI300X (FNUZ) is not NVIDIA's e4m3fn; MLX 8-bit is not Q8_0; an OpenVINO int8 export is not either of them. Rank only rows whose resolved formats match. diff --git a/plugin/skills/optimizing-inference/references/intake.md b/plugin/skills/optimizing-inference/references/intake.md index f352e65..b4b148b 100644 --- a/plugin/skills/optimizing-inference/references/intake.md +++ b/plugin/skills/optimizing-inference/references/intake.md @@ -39,8 +39,9 @@ inferencekit intake validate intake.yaml --run-dir runs/ # start or resu | `goals.traffic` | — | `arrival_rate_rps` (the **peak**: stage 8 sizes replicas for it), `burstiness`, `session_len`. | | `goals.after_target` | `keep_climbing` when `non_interactive` or a budget is fixed; else `stop` | At the target, `keep_climbing` continues while headroom and budget remain ([stop-point.md](stop-point.md)); set it for benchmark and comparison runs too. | | `io_modes` | input `complete`, output `streaming` | Streaming input warns: engines without incremental input are NOT SUPPORTED for it; any emulation is labelled. | -| `buckets` | all four | `very_short` 1–5 words, `short` ~20–50 tokens, `medium` ~300 tokens, `long` 2k–4k tokens. Choose them from the input-length p50 and p90 ([workload.md](workload.md)). | -| `workload` | `max_tokens` 256, `rounds` 5, `prompt_set` `dev` | `max_tokens` from the output-length p50 (p90 for capacity when p90 > 2 × p50). `concurrency` defaults to `[1,2,4,8,16,32,64]` in load mode; it is a floor, extended until the SLO fails ([capacity.md](capacity.md)). | +| `buckets` | all four; TTS declared text buckets when supplied | LLM: `very_short` 1–5 words, `short` ~20–50 tokens, `medium` ~300 tokens, `long` 2k–4k tokens. Choose them from the input-length p50 and p90 ([workload.md](workload.md)). TTS character bounds live in `workload.tts.text_length_buckets`. | +| `workload` | `rounds` 5; LLM only: `max_tokens` 256, `prompt_set` `dev` | LLM `max_tokens` comes from the output-length p50 (p90 for capacity when p90 > 2 × p50); omit it for media. `concurrency` defaults to `[1,2,4,8,16,32,64]` in load mode; it is a floor, extended until the SLO fails ([capacity.md](capacity.md)). | +| `workload.client`, `workload.tts` | unset | Timing client (`bench_audio` for TTS, `external` for another adapter); TTS voice, sample rate, character buckets and lead metric. `io_modes.output` selects streaming or complete utterance; see below. | | `budget` | `approved: false` | `gpu_hours` (host hours × count), `usd`, `wall_clock_h` (the deadline; see "Time budget" below). Reuse explicit session/intake authorization; no separate plan approval ([starting-questions.md](starting-questions.md)). | | `quality.reference` | `none` | `{kind: endpoint, base_url}` for a higher-precision reference server. With `none`, the reference is the baseline engine at the same precision. | | `quality.noise_floor` | `measure` | `measure` = an A/A run of the reference; or a fixed agreement floor in [0, 1]. | @@ -63,8 +64,8 @@ inferencekit intake validate intake.yaml --run-dir runs/ # start or resu | `decode_ceiling_fraction` | higher | fraction | llm | decode tok/s ÷ ceiling tok/s at that bucket's context (derived; ceilings.md) | | `n_star` | higher | streams/GPU | not video | `bench serve` load mode, against `goals.slo` | | `usd_per_1m` | lower | $/1M units | not video | `price_usd_h` ÷ (units/s at the lead operating point × 3600) × 1e6 | -| `ttfa_ms` | lower | ms | tts | a modality client (UNPROVEN without one) | -| `rtf` | lower | × real time | tts, stt | a modality client (UNPROVEN without one) | +| `ttfa_ms` | lower | ms | tts | `bench audio` or a matching modality client (UNPROVEN without one) | +| `rtf` | lower | × real time | tts, stt | `bench audio` for TTS; an external matching client for STT | | `clip_ms`, `clips_per_hour` | lower, higher | ms, clips/h | video | `bench clip` | A target that does not apply to the modality is reported NOT RUN. @@ -163,6 +164,24 @@ constraints: `modality: image` adds `image_ms`/`images_per_hour`; `tts` adds `ttfa_p95_ms` (first playable audio). Both need a real modality adapter, not a token endpoint. [Affordability](affordability.md) defines memory/latency/quality acceptance: `goals.affordability` requires `memory_gb_max`/`memory_kind`, optional smaller stretch cap, host cap and `offload_allowed`; `lead: affordability` additionally requires a latency target and `quality.protocol`. Image quality budgets use PSNR/SSIM/LPIPS, speech WER/CER/speaker similarity; protocols also cover reference/A/A, datasets/seeds, margins/CIs and human review. +For TTS, omit `max_tokens` and declare the speech workload: + +```yaml +modality: tts +io_modes: {input: complete, output: streaming} +workload: + client: bench_audio + tts: + voice: speaker-a + sample_rate: 24000 + lead_metric: first_audio_ms # or rtf / complete_audio_ms + text_length_buckets: + - {name: short, min_chars: 1, max_chars: 80} + - {name: long, min_chars: 81, max_chars: 800} +``` + +These pins describe the intended run; pass them to the adapter and verify effective settings. `bench_audio` removes the missing-client warning, but timing does not prove quality or fresh-process interleaving. [Audio contracts and metric definitions](https://github.com/KittenML/inferencekit/blob/main/docs/media-benchmarks.md) apply; STT needs `client: external`. + ### Relative modality budgets Use `fidelity: lossy` and a typed `quality.budget.relative` when the agreed criterion is degradation relative to a baseline or A/A floor. Do not encode an LPIPS excess as `lpips_max`, or a WER increase as `wer_max`: those existing keys are absolute bounds. Absolute and relative branches may coexist; every declared bound still needs proof. diff --git a/plugin/skills/optimizing-inference/references/state.md b/plugin/skills/optimizing-inference/references/state.md index 26f1a20..8e695c9 100644 --- a/plugin/skills/optimizing-inference/references/state.md +++ b/plugin/skills/optimizing-inference/references/state.md @@ -5,7 +5,7 @@ - **Run id:** `run_id` from the intake if it has one; otherwise `derived.run_id` from the last dry check of the intake *file* before `--run-dir`. It contains the time, so a later dry check prints a different id: reuse the one you copied everywhere. - **Fresh run:** `--run-dir` writes a redacted intake copy, `intake.normalized.json`, `state.json`, `experiments.jsonl` and `QUESTIONS.md`. It only creates them, never overwrites. - **Evidence root, one per run: `runs//evidence/`,** as `INFERENCEKIT_RUNS_DIR` (absolute) for every inferencekit call, here and on the GPU host ([remote-host.md](remote-host.md)). Shells forget exports: prefix each call or pass `inferencekit --runs-dir `. Without it, records land in `./runs/ev_*`, outside the sync, and citations (`evidence show`, `--gpu-bw`, `ledger trailers`, `review build`) break. MCP tools see only the MCP server's start environment: without the variable there, use the CLI. -- **Resume:** re-run `intake validate --run-dir runs/` (`state.json` `intake_source` names it; the redacted copy is refused), set the evidence root, read `state.json`, trust `current_stage`, `stages.*` and `next_action`, check the cited evidence exists (`inferencekit evidence show `), and continue. Never redo a `done` stage whose evidence exists. A changed intake is refused: start a new run id, never edit the copy, `intake_sha256` or a target ([intake.md](intake.md)). +- **Resume:** `intake resume` checks pinned policy and cited evidence. Use `--file ` if `state.intake_source` moved; `--now ` fixes budget time. Set `evidence_root`; integrity is not proof. Never redo a `done` stage whose evidence exists. Redacted/changed intakes are refused: changes need a new run id; never edit the copy, `intake_sha256` or targets ([intake.md](intake.md)). ## state.json diff --git a/plugin/skills/optimizing-inference/references/tuning.md b/plugin/skills/optimizing-inference/references/tuning.md index 0d0f09b..b2c0648 100644 --- a/plugin/skills/optimizing-inference/references/tuning.md +++ b/plugin/skills/optimizing-inference/references/tuning.md @@ -2,8 +2,8 @@ ## Procedure (per supported engine, inside its share of the stage 4 box) -1. **Baseline recipe.** Use the engine's documented low-latency configuration for this model class and GPU, with prefix caching off and CUDA graphs on. Measure it, label it `-recipe` (manifest role `baseline-recipe`), and keep it. If quoted numbers for this engine beat the recipe (e.g. a lower TTFT), find and measure their config first. -2. **Candidate list.** Pick up to ~12 configs from the knob table below, most promising first, at most 2 knobs changed per config. Order them by the lever ladder ([stop-point.md](stop-point.md)): overhead first, then batching and scheduler (capacity), then precision only within the intake. A vendor autotuner may propose some of them ("Vendor autotuners", below). +1. **Baseline recipe.** Recheck current official recipes for this model, hardware and goal; the knob tables below are examples. Use the documented configuration with prefix caching off and CUDA graphs on where applicable. Measure it, label it `-recipe` (manifest role `baseline-recipe`), and keep it. If quoted numbers for this engine beat the recipe (e.g. a lower TTFT), find and measure their config first. +2. **Candidate list.** Pick up to ~12 configs from current official recipes and applicable examples below, most promising first, at most 2 knobs changed per config. Order them by the lever ladder ([stop-point.md](stop-point.md)): overhead first, then batching and scheduler (capacity), then precision only within the intake. A vendor autotuner may propose some of them ("Vendor autotuners", below). 3. **Screening runs:** two buckets, `short` and `long` (or the intake's target buckets), one round, the same prompts and `max_tokens`: `bench serve --buckets short --buckets long --rounds 1 --engine-kind ` ([harness-commands.md](harness-commands.md); never the engine's own benchmark). Why two: a knob that cuts launch overhead can help short prompts and hurt long ones (prefill, KV), and one bucket hides that. A single-round point estimate only **ranks** configs to pick the top 3: never keep or drop, never a reported result. 4. **Successive halving:** the top 3 by the lead metric plus the recipe, each its own manifest entry (label, port, launch line), go to the full protocol (all buckets, interleaved rounds; ``, its floor and fitting the stage box: [harness-commands.md](harness-commands.md), section 2): one `bench rounds --engines ,,,` over pairwise calls, dropping candidates before rounds. - **Also promote** a screened config that beats the pre-registered effect `e` on another metric the person cares about (the intake's SLO or goals, e.g. TTFT or TPOT) in either bucket, without screening worse than the recipe on the lead metric. It joins the same call. If `bench rounds` shows that metric `improved` and the lead metric not `regressed`, report it as an option for that metric: it does not replace the headline config, and the `REPORT.md` summary lists it on its own line ([stop-point.md](stop-point.md), "Summary"). diff --git a/plugin/skills/optimizing-inference/references/workload.md b/plugin/skills/optimizing-inference/references/workload.md index 869e81a..6e7c6df 100644 --- a/plugin/skills/optimizing-inference/references/workload.md +++ b/plugin/skills/optimizing-inference/references/workload.md @@ -10,6 +10,8 @@ Experienced inference engineers start from the workload, not the engine: prompt ## Technical mapping (derive first; translate necessary questions into ordinary language) +The token-length mapping below is for LLMs. For TTS, use [speech workload fields](intake.md): text character buckets, voice, sample rate, `io_modes.output` and the first-audio/RTF/completion lead metric. Do not invent a `max_tokens` value for audio, image or video. + | # | Ask | Intake field | If unknown | |---|---|---|---| | 1 | **SLO:** the p90 or p95 time to first token (TTFT) and time per output token (TPOT) a user must get, and the failures allowed | `goals.slo.ttft_p95_ms`, `tpot_p95_ms`, `max_fail` | Interactive: ask. There is no default SLO. You may offer a starting point for a chat UI, e.g. p95 TTFT ≤ 1,000 ms and TPOT ≤ 50 ms (≥ 20 tok/s, faster than reading speed) (inferred). It is never the person's SLO until they say yes. | diff --git a/plugin/skills/profiling/SKILL.md b/plugin/skills/profiling/SKILL.md index 87c2dd0..e11a1bb 100644 --- a/plugin/skills/profiling/SKILL.md +++ b/plugin/skills/profiling/SKILL.md @@ -17,7 +17,7 @@ metadata: ## Route before capturing -Read `doctor`'s `data.hardware.platform`/`data.profiling_route` and the relevant [hardware reference](../inference-triage/references/hardware.md). NVIDIA: nsys → ncu. AMD: rocprofv3 HIP/kernel/copy timeline → architecture-supported counters (Instinct support does not establish Radeon support). Apple: Instruments CPU/Metal → Xcode/MLX capture. CPU: platform profiler. nsys/ncu capture is NOT RUN elsewhere; profiling itself still applies. +Read `doctor`'s `data.hardware.platform`/`data.profiling_route` and the relevant [hardware reference](../inference-triage/references/hardware.md). NVIDIA: nsys → ncu. AMD: rocprofv3 HIP/kernel/copy timeline → architecture-supported counters (Instinct support does not establish Radeon support). Apple: Instruments CPU/Metal → Xcode/MLX capture. CPU: [perf when permitted, otherwise engine timers or CPU-only torch.profiler](../inference-triage/references/cpu.md#profiling). nsys/ncu capture is NOT RUN elsewhere; profiling itself still applies. **Retrieve current official docs online** for unknown architectures, runtime/tool support, flags or errors before choosing a path; record URL/date/version and check local help. Use AMD ROCm, Apple Developer/MLX and NVIDIA primary sources. Retain timeline/op attribution when counters fail, marking unsupported checks NOT RUN. All platforms follow critical path → Amdahl → change → correctness → unprofiled interleaved A/B, with the same [environment hygiene](references/capture-hygiene.md). Include text encoding, image VAE, audio codec/vocoder and buffering where applicable. diff --git a/plugin/skills/proving-correctness/SKILL.md b/plugin/skills/proving-correctness/SKILL.md index b64426c..cf67e3e 100644 --- a/plugin/skills/proving-correctness/SKILL.md +++ b/plugin/skills/proving-correctness/SKILL.md @@ -30,6 +30,8 @@ Read the run's [quality choice](../optimizing-inference/references/starting-ques Use modality-appropriate structural integrity and task quality: text accuracy/distributions, image semantics/readability, speech content/completion/speaker, video temporal consistency. Other tasks need a registered adapter; do not substitute GSM8K or token identity. Missing listening/viewing remains UNPROVEN. The text procedures below apply only where their inputs and claims match. +For TTS, register the acceptance policy before capture with `quality audio-policy`, then run `quality audio` on final configuration records and saved PCM ([protocol](https://github.com/KittenML/inferencekit/blob/main/docs/audio-quality.md)). For changed outputs, `quality audio-scorers` pins local models and `quality audio-score` produces ASR/naturalness/voice observations ([setup](https://github.com/KittenML/inferencekit/blob/main/docs/audio-scorers.md)). Score after timing; WER/CER and MOS predictors do not replace required listening. + ## Which rule applies (decide before measuring) | Kind of change | Examples | Fidelity | Pass rule | diff --git a/schemas/evidence.v2.json b/schemas/evidence.v2.json index 6c5f9d9..6d3be1a 100644 --- a/schemas/evidence.v2.json +++ b/schemas/evidence.v2.json @@ -51,6 +51,7 @@ "clip_steps", "image_benchmark", "audio_benchmark", + "quality_audio", "rocprof_capture", "rocprof_summary" ] @@ -77,6 +78,17 @@ "workload" ], "properties": { + "cpu": { + "type": "object", "additionalProperties": false, + "properties": { + "model": {"type": ["string", "null"]}, + "architecture": {"type": ["string", "null"]}, + "logical_cores": {"type": ["integer", "null"], "minimum": 1}, + "physical_cores": {"type": ["integer", "null"], "minimum": 1}, + "allowed_cpus": {"type": ["array", "null"], "items": {"type": "integer", "minimum": 0}, "uniqueItems": true}, + "topology_sha256": {"type": ["string", "null"], "pattern": "^[0-9a-f]{64}$"} + } + }, "gpu": { "type": "object", "additionalProperties": false, diff --git a/schemas/intake.v1.json b/schemas/intake.v1.json index 5d39248..0909155 100644 --- a/schemas/intake.v1.json +++ b/schemas/intake.v1.json @@ -6,6 +6,8 @@ "type": "object", "required": ["model", "modality", "hardware"], "additionalProperties": false, + "allOf": [{"if": {"properties": {"modality": {"const": "llm"}}, "required": ["modality"]}, + "then": {"properties": {"workload": {"properties": {"max_tokens": {"minimum": 2}}}}}}], "properties": { "intake_version": {"const": 1}, "run_id": {"type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]{0,79}$", @@ -135,10 +137,30 @@ "type": "object", "additionalProperties": false, "properties": { - "max_tokens": {"type": "integer", "minimum": 2, "default": 256}, + "max_tokens": {"type": "integer", "minimum": 1, "description": "LLM output length (minimum 2, default 256). Optional and unused for media workloads."}, "rounds": {"type": "integer", "minimum": 3, "default": 5}, "concurrency": {"type": "array", "minItems": 1, "items": {"type": "integer", "minimum": 1}}, - "prompt_set": {"type": "string", "default": "dev"}, + "prompt_set": {"type": "string", "description": "LLM prompt set (default dev)."}, + "client": {"enum": ["bench_serve", "bench_audio", "bench_image", "bench_clip", "external"], "description": "Configured timing client; selecting one declares the route, not successful measurement or quality proof."}, + "tts": { + "type": "object", "additionalProperties": false, + "properties": { + "voice": {"type": "string", "minLength": 1}, + "sample_rate": {"type": "integer", "minimum": 1}, + "lead_metric": {"enum": ["rtf", "first_audio_ms", "complete_audio_ms"]}, + "text_length_buckets": { + "type": "array", "minItems": 1, + "items": {"type": "object", "additionalProperties": false, + "required": ["name", "min_chars", "max_chars"], + "properties": { + "name": {"enum": ["very_short", "short", "medium", "long"]}, + "min_chars": {"type": "integer", "minimum": 1}, + "max_chars": {"type": "integer", "minimum": 1} + } + } + } + } + }, "chat_template_kwargs": {"type": "object", "description": "Pre-registered template settings, e.g. {\"enable_thinking\": false} for a reasoning model. The intake itself sends nothing: intake validate records the pin in intake.normalized.json and state.json; manifest validate and bench rounds refuse a run manifest whose workload.chat_template_kwargs leaves out or changes a pinned key; bench serve, gates run and quality trajectory/gsm8k/niah with --run-dir add the pinned keys to every chat request and refuse a different value."} } }, diff --git a/schemas/run-manifest.v1.json b/schemas/run-manifest.v1.json index 01dbead..6c7d0b6 100644 --- a/schemas/run-manifest.v1.json +++ b/schemas/run-manifest.v1.json @@ -23,9 +23,14 @@ }, "hardware": { "type": "object", - "required": ["gpu", "count"], + "required": ["count"], + "if": {"properties": {"backend": {"const": "cpu"}}, "required": ["backend"]}, + "then": {"required": ["cpu"], "not": {"required": ["gpu"]}}, + "else": {"required": ["gpu"]}, "additionalProperties": false, "properties": { + "backend": {"enum": ["cpu", "cuda", "rocm", "metal"], "description": "CPU hosts use an exclusive CPU lock and process-group RSS; other backends retain the GPU route."}, + "cpu": {"type": "object", "required": ["model"], "properties": {"model": {"type": "string", "minLength": 1}}, "description": "CPU identity from doctor or cpu screen-manifest; optional topology and allowed CPU facts."}, "gpu": {"type": "string", "minLength": 1, "description": "GPU model as nvidia-smi names it, or a substring of that name (e.g. 'RTX 5090')."}, "count": {"type": "integer", "minimum": 1}, "price_usd_h": {"type": ["number", "null"], "minimum": 0} @@ -51,7 +56,7 @@ "ready_check": {"type": "string", "description": "HTTP path polled until it returns 200, e.g. /v1/models or /health."}, "resolved_config": {"type": "object", "description": "What the engine logged it actually used."}, "role": {"enum": ["baseline-recipe", "tuned", "custom"]}, - "endpoint_kind": {"enum": ["clip_http", "comfyui"], "description": "bench rounds --mode clip: how bench clip reaches the engine (default clip_http)."}, + "endpoint_kind": {"enum": ["clip_http", "comfyui", "audio_http", "image_http", "openai_chat"], "description": "Client contract for bench rounds. Audio and image use the contracts in docs/media-benchmarks.md."}, "workflow": {"type": "string", "description": "comfyui: the API-format workflow JSON (a path relative to the manifest is resolved there)."} } } @@ -59,12 +64,18 @@ "workload": { "type": "object", "required": ["modes"], - "if": {"properties": {"modes": {"contains": {"const": "clip"}}}, "required": ["modes"]}, - "then": {"required": ["clips"]}, - "else": {"required": ["buckets", "max_tokens"]}, + "allOf": [ + {"if": {"properties": {"modes": {"contains": {"const": "clip"}}}}, "then": {"required": ["clips"]}}, + {"if": {"properties": {"modes": {"contains": {"enum": ["single", "load"]}}}}, "then": {"required": ["buckets", "max_tokens"]}} + ], "additionalProperties": false, "properties": { - "modes": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"enum": ["single", "load", "clip"]}}, + "modes": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"enum": ["single", "load", "clip", "audio", "image"]}}, + "modality": {"enum": ["llm", "text", "tts", "audio", "image", "video"]}, + "sample_rate": {"type": "integer", "minimum": 1}, + "channels": {"type": "integer", "minimum": 1, "maximum": 8}, + "settings": {"type": "object", "description": "Audio/image generation settings passed unchanged to every arm."}, + "quant": {"type": "string", "description": "Declare none for unquantized weights."}, "clips": {"type": "string", "description": "Clip set JSONL for --mode clip (sha256 pinned by bench clip records)."}, "buckets": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"enum": ["very_short", "short", "medium", "long"]}}, "max_tokens": {"type": "integer", "minimum": 1}, diff --git a/src/inferencekit/cli.py b/src/inferencekit/cli.py index 085396e..1f5b7d0 100644 --- a/src/inferencekit/cli.py +++ b/src/inferencekit/cli.py @@ -116,8 +116,8 @@ def output_mode(flag: bool) -> str: def is_record(result) -> bool: """An evidence record (it has an id, a kind and a summary), as opposed to a listing or a check result.""" - return isinstance(result, dict) and all(isinstance(result.get(k), str) and result.get(k) - for k in ("evidence_id", "kind", "summary")) + return isinstance(result, dict) and result.get("view") != "summary" and \ + all(isinstance(result.get(k), str) and result.get(k) for k in ("evidence_id", "kind", "summary")) def _one_line(text, limit: int | None = None) -> str: diff --git a/src/inferencekit/cpuinfo.py b/src/inferencekit/cpuinfo.py new file mode 100644 index 0000000..272a8b2 --- /dev/null +++ b/src/inferencekit/cpuinfo.py @@ -0,0 +1,166 @@ +"""Selected CPU facts; no framework imports, counter probes, or workloads. + +Do not retain raw /proc output, process names, host names, serials, or paths. +Visible ISA support and configuration are not proof of an engine's dispatch. +""" +from __future__ import annotations + +import hashlib +import json +import os +import platform +import re +from pathlib import Path + +from inferencekit.proc import which + +CPU_ENV_RULES = { + "OMP_NUM_THREADS": r"[1-9][0-9]*", "MKL_NUM_THREADS": r"[1-9][0-9]*", + "OPENBLAS_NUM_THREADS": r"[1-9][0-9]*", "BLIS_NUM_THREADS": r"[1-9][0-9]*", + "NUMEXPR_NUM_THREADS": r"[1-9][0-9]*", "OMP_PLACES": r"(?:threads|cores|sockets)", + "OMP_PROC_BIND": r"(?:true|false|master|primary|close|spread)", + "OMP_DYNAMIC": r"(?:true|false)", "MKL_DYNAMIC": r"(?:true|false)", + "MKL_ENABLE_INSTRUCTIONS": r"(?:SSE4_2|AVX|AVX2|AVX512|AVX512_MIC|AVX512_MIC_E1|AVX512_E1)", + "DNNL_MAX_CPU_ISA": r"(?:DEFAULT|SSE41|AVX|AVX2|AVX2_VNNI|AVX512_CORE|AVX512_CORE_VNNI|AVX512_CORE_BF16|AVX512_CORE_AMX|AVX512_CORE_AMX_FP16)", + "ONEDNN_MAX_CPU_ISA": r"(?:DEFAULT|SSE41|AVX|AVX2|AVX2_VNNI|AVX512_CORE|AVX512_CORE_VNNI|AVX512_CORE_BF16|AVX512_CORE_AMX|AVX512_CORE_AMX_FP16)", + "MKL_VERBOSE": r"[01]", "DNNL_VERBOSE": r"[012]", "ONEDNN_VERBOSE": r"[012]", +} +ISA_FLAGS = {"avx2": ("avx2",), "avx512": ("avx512f",), "avx512_vnni": ("avx512_vnni",), + "avx_vnni": ("avx_vnni",), "avx512_bf16": ("avx512_bf16",), + "amx_bf16": ("amx_bf16",), "amx_int8": ("amx_int8",), "neon": ("asimd", "neon")} + + +def _read(path: Path) -> str | None: + try: + return path.read_text(errors="replace").strip() or None + except OSError: + return None + + +def _integer(text) -> int | None: + try: + return int(text) if text is not None else None + except ValueError: + return None + + +def _topology_id(text) -> int | None: + value = _integer(text) + return value if value is not None and value >= 0 else None + + +def cpu_list(text: str | None) -> list[int]: + """Parse a Linux CPU list; malformed/absurd masks are unknown, not partial.""" + if not text: + return [] + result = set() + for part in text.split(","): + if not re.fullmatch(r"\d+(?:-\d+)?", part): + return [] + lo, _, hi = part.partition("-") + a, b = int(lo), int(hi or lo) + if a > b or b > 1_000_000 or b - a > 100_000: + return [] + result.update(range(a, b + 1)) + return sorted(result) + + +def cpu_tuning_env(environ=None) -> dict: + env = os.environ if environ is None else environ + return {key: value for key, pattern in CPU_ENV_RULES.items() + if isinstance(value := env.get(key), str) and re.fullmatch(pattern, value, re.I)} + + +def cpu_facts(*, proc_root=Path("/proc"), cpu_root=Path("/sys/devices/system/cpu"), + node_root=Path("/sys/devices/system/node"), allowed_cpus=None) -> dict: + """Read visible host topology and process affinity separately (containers may differ).""" + blocks = (_read(proc_root / "cpuinfo") or "").split("\n\n") + info = [] + for block in blocks: + fields = {k.strip(): v.strip() for line in block.splitlines() if ":" in line + for k, v in [line.split(":", 1)]} + if fields: + info.append(fields) + model = next((r.get("model name") or r.get("Processor") for r in info + if r.get("model name") or r.get("Processor")), None) + if allowed_cpus is None: + try: + allowed_cpus = sorted(os.sched_getaffinity(0)) + except (AttributeError, OSError): + allowed_cpus = None + online = cpu_list(_read(cpu_root / "online")) + if not online: + online = sorted(n for r in info if (n := _integer(r.get("processor"))) is not None) + topology, l3 = [], {} + for n in online: + root = cpu_root / f"cpu{n}" + topology.append({"cpu": n, "package": _topology_id(_read(root / "topology/physical_package_id")), + "core": _topology_id(_read(root / "topology/core_id")), + "siblings": cpu_list(_read(root / "topology/thread_siblings_list")), + "core_type": _integer(_read(root / "topology/core_type"))}) + for cache in root.glob("cache/index*"): + if _read(cache / "level") != "3": + continue + shared = cpu_list(_read(cache / "shared_cpu_list")) + if shared: + size = _read(cache / "size") + l3[tuple(shared)] = {"cpus": shared, "size": size if size and re.fullmatch(r"\d+[KMG]?", size) else None} + cores = {(r["package"], r["core"]) for r in topology if r["package"] is not None and r["core"] is not None} + topology_complete = bool(topology) and all(r["package"] is not None and r["core"] is not None for r in topology) + nodes = [{"node": int(p.name[4:]), "cpus": cpu_list(_read(p / "cpulist"))} + for p in sorted(node_root.glob("node[0-9]*")) if re.fullmatch(r"node\d+", p.name)] + flag_sets = [set(r.get("flags", r.get("Features", "")).split()) for r in info + if "flags" in r or "Features" in r] + common = set.intersection(*flag_sets) if flag_sets else None + present = set.union(*flag_sets) if flag_sets else None + governors = sorted({v for p in cpu_root.glob("cpufreq/policy*/scaling_governor") + if (v := _read(p)) and re.fullmatch(r"[a-zA-Z0-9_-]{1,40}", v)}) + status = _read(proc_root / "self/status") or "" + cap_match = re.search(r"^CapEff:\s*([0-9a-fA-F]+)\s*$", status, re.M) + caps = int(cap_match.group(1), 16) if cap_match else None + paranoid = _integer(_read(proc_root / "sys/kernel/perf_event_paranoid")) + cap_perf = bool(caps & (1 << 38)) if caps is not None else None + cap_admin = bool(caps & (1 << 21)) if caps is not None else None + perf_present = which("perf") is not None + access = ("policy_allows_user" if cap_perf or cap_admin or (paranoid is not None and paranoid <= 2) + else "restricted" if paranoid is not None and paranoid > 2 and caps is not None else "unknown") + load_fields = (_read(proc_root / "loadavg") or "").split() + try: + loads = [float(v) for v in load_fields[:3]] if len(load_fields) >= 3 else None + if loads is not None and any(not (0 <= v < float("inf")) for v in loads): + loads = None + except ValueError: + loads = None + logical = len(online) or os.cpu_count() + warnings = [] + capacity = len(allowed_cpus) if allowed_cpus else logical + if loads and capacity and loads[0] >= max(1, capacity / 2): + warnings.append("Host one-minute load is at least half the available logical CPU count; contention may skew timings. " + "Load is host-wide, may include tasks outside this affinity, and includes uninterruptible tasks; " + "it is not per-core utilization.") + if not topology_complete: + warnings.append("Physical-core topology is incomplete; do not infer SMT or placement from CPU numbering.") + if not any(r["core_type"] is not None for r in topology): + warnings.append("Hybrid core types are unavailable; no P/E assignment is inferred from model or frequency.") + return {"model": model[:200] if model else None, "architecture": platform.machine(), + "logical_cores": logical, "physical_cores": len(cores) if topology_complete else None, + "allowed_cpus": sorted(set(allowed_cpus)) if allowed_cpus is not None else None, + "topology": topology, "l3_groups": list(l3.values()), "numa_nodes": nodes, + "isa": {k: any(f in common for f in fs) if common is not None else None for k, fs in ISA_FLAGS.items()}, + "isa_any_cpu": {k: any(f in present for f in fs) if present is not None else None for k, fs in ISA_FLAGS.items()}, + "governors": governors, "load_average": loads, + "perf": {"installed": perf_present, "perf_event_paranoid": paranoid, + "cap_perfmon": cap_perf, "cap_sys_admin": cap_admin, "access": access, "verified": False}, + "tuning_env": cpu_tuning_env(), + "library_dispatch": {"status": "unverified", "reason": "ISA flags and environment pins do not establish " + "loaded BLAS/oneDNN libraries or selected kernels; retain engine runtime dispatch logs."}, + "limitations": warnings} + + +def cpu_identity(facts=None) -> dict: + facts = cpu_facts() if facts is None else facts + result = {k: facts.get(k) for k in ("model", "architecture", "logical_cores", "physical_cores", "allowed_cpus")} + topology = {k: facts.get(k) for k in ("topology", "l3_groups", "numa_nodes", "isa")} + result["topology_sha256"] = (hashlib.sha256(json.dumps(topology, sort_keys=True).encode()).hexdigest() + if facts.get("topology") else None) + return result diff --git a/src/inferencekit/evidence.py b/src/inferencekit/evidence.py index 50cdd83..cacb6cb 100644 --- a/src/inferencekit/evidence.py +++ b/src/inferencekit/evidence.py @@ -53,6 +53,7 @@ KINDS += ("clip_benchmark", "clip_gates", "clip_quality", "clip_steps") # Client-timed images and speech. Structural validity never establishes semantic quality. KINDS += ("image_benchmark", "audio_benchmark", "rocprof_capture", "rocprof_summary") +KINDS += ("quality_audio",) # identity.workload fields: the v1 four are always present (null when unknown); the v2 serving fields # are written only when the operation knows them. WORKLOAD_V1 = ("prompt_len", "gen_len", "batch", "seed") @@ -90,7 +91,7 @@ def metric(name: str, value: float | int | None, unit: str, stat: str, n: int | def identity(gpu: dict | None = None, engine: dict | None = None, model: dict | None = None, - workload: dict | None = None) -> dict: + workload: dict | None = None, cpu: dict | None = None) -> dict: """The four things a number depends on. Unknown fields are null, never guessed.""" g = {"name": None, "count": None, "driver": None, "cuda": None, "clocks": None} e = {"name": None, "version": None, "commit": None, "flags": {}} @@ -103,7 +104,11 @@ def identity(gpu: dict | None = None, engine: dict | None = None, model: dict | for k in WORKLOAD_V2: # serving workloads (v2): mode, concurrency grid, prompt set, SLO... if (workload or {}).get(k) is not None: w[k] = workload[k] - return {"gpu": g, "engine": e, "model": m, "workload": w} + result = {"gpu": g, "engine": e, "model": m, "workload": w} + if cpu is not None: + result["cpu"] = {k: cpu.get(k) for k in ("model", "architecture", "logical_cores", "physical_cores", + "allowed_cpus", "topology_sha256")} + return result def build(call: Call, *, kind: str, claim_class: str, summary: str, identity_: dict | None = None, @@ -299,11 +304,14 @@ def validate(record: dict) -> list[str]: return errors -def verify(record: dict) -> dict: +def verify(record: dict, *, include_error_count: bool = False) -> dict: """Integrity check: does the id match the content, and does the record fit the schema?""" errors = validate(record) - return {"id_ok": compute_id(record) == record.get("evidence_id"), "schema_ok": not errors, - "schema_errors": errors[:20]} + result = {"id_ok": compute_id(record) == record.get("evidence_id"), "schema_ok": not errors, + "schema_errors": errors[:20]} + if include_error_count and errors: + result["schema_error_count"] = len(errors) + return result def integrity_problem(record: dict) -> str | None: diff --git a/src/inferencekit/hardware.py b/src/inferencekit/hardware.py index b8030d9..8b2a490 100644 --- a/src/inferencekit/hardware.py +++ b/src/inferencekit/hardware.py @@ -14,6 +14,7 @@ from pathlib import Path from inferencekit.proc import run, which +from inferencekit.cpuinfo import cpu_facts def _read(path: Path) -> str | None: @@ -114,7 +115,7 @@ def discover_non_nvidia() -> dict: accelerator_interface = any(Path(p).exists() for p in ("/dev/dxg", "/dev/kfd", "/dev/dri")) backend = devices[0]["backend"] if devices else "unknown" if accelerator_interface else "cpu" return {"platform": backend, "devices": devices, - "cpu": {"architecture": platform.machine(), "logical_cores": os.cpu_count()}, + "cpu": cpu_facts(), "limitations": (["Device identification does not prove runtime or profiler support; verify the exact " "GPU, OS and installed runtime against current official docs and a smoke request."] if devices else ["Accelerator identity is unknown; a device interface exists but vendor " @@ -123,13 +124,19 @@ def discover_non_nvidia() -> dict: "device can also produce this result."])} -def profiler_route(backend: str) -> dict: +def profiler_route(backend: str, cpu: dict | None = None) -> dict: + if backend == "cpu": + perf = (cpu or {}).get("perf", {}) + usable = perf.get("installed") and perf.get("access") == "policy_allows_user" + return {"timeline": "perf (user-space; verify access)" if usable else "engine per-layer timers / torch.profiler (CPU)", + "kernel": "perf stat/record" if usable else "engine per-layer timers / torch.profiler (CPU)", + "fallback": "engine per-layer timers / torch.profiler (CPU)", + "permission_verified": False, + "documentation": "https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html"} return { "nvidia": {"timeline": "nsys", "kernel": "ncu", "documentation": "https://docs.nvidia.com/nsight-systems/"}, "rocm": {"timeline": "rocprofv3", "kernel": "rocprofv3 counters (verify GPU support)", "documentation": "https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/latest/how-to/using-rocprofv3.html"}, "metal": {"timeline": "Instruments Metal System Trace", "kernel": "Xcode Metal debugger / MLX capture", "documentation": "https://developer.apple.com/documentation/xcode/optimizing-gpu-performance"}, - "cpu": {"timeline": "perf / Instruments Time Profiler", "kernel": "platform CPU profiler", - "documentation": "https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html"}, }.get(backend, {"timeline": None, "kernel": None, "documentation": None}) diff --git a/src/inferencekit/mcp_server.py b/src/inferencekit/mcp_server.py index 818b231..2ab6e4f 100644 --- a/src/inferencekit/mcp_server.py +++ b/src/inferencekit/mcp_server.py @@ -42,7 +42,9 @@ "build_review for the run's review page; ledger_trailers / history for Perf-* commit trailers. " "Pass chat_template_kwargs (e.g. {\"enable_thinking\": false}) identically to bench_serve, bench_rounds, " "run_gates, quality_trajectory, quality_gsm8k and quality_niah (an intake pin in the run dir is applied and " - "enforced when run_dir is given). Long lists in results are truncated: page them with " + "enforced when run_dir is given). Start evidence inspection with inspect_evidence(id, view='summary') " + "for integrity, metrics, limitations and stored decisions (failures first); follow path for details. " + "Long lists in results are truncated: page them with " "inspect_evidence(id, path, offset, limit)." ) @@ -51,17 +53,17 @@ def _size(obj) -> int: return len(json.dumps(obj, ensure_ascii=False, default=str)) -def _lists(node, path=""): +def _lists(node, path=()): """Yield (path, parent, key, list) for every list inside node, largest first by caller.""" if isinstance(node, dict): for k, v in node.items(): - p = f"{path}.{k}" if path else k + p = (*path, k) if isinstance(v, list): yield p, node, k, v yield from _lists(v, p) elif isinstance(node, list): for i, v in enumerate(node): - yield from _lists(v, f"{path}.{i}") + yield from _lists(v, (*path, i)) def _trim_strings(node, limit: int) -> None: @@ -83,17 +85,119 @@ def _fit_page(view: dict, max_chars: int) -> dict: """Keep the first k items of a page that fit, and point next_offset / the hint at item offset+k.""" items = view["items"] k = len(items) - while k > 1 and _size({**view, "items": items[:k]}) > max_chars: + # Continuation hints also consume the response budget. Recheck the complete page + # after each reduction so generic trimming cannot hide rows behind next_offset. + while k > 1 and _size(view) > max_chars: k = max(1, k * 3 // 4) - if k == len(items): + nxt = view["offset"] + k + view["items"], view["next_offset"] = items[:k], nxt + pagination = view.get("pagination") + if isinstance(pagination, dict) and isinstance(pagination.get("tool"), str) and \ + isinstance(pagination.get("args"), dict): + view["next_call"] = {"tool": pagination["tool"], + "arguments": {**pagination["args"], "offset": nxt, "limit": k}} + view["more"] = f"page shortened to {k} items; continue with next_call (original arguments preserved)" + elif view.get("view") == "summary": + view["more"] = (f"inspect_evidence(id={view['source']!r}, view='summary', " + f"offset={nxt}, limit={k})") + elif "path" in view: # inspect_evidence: repeat the same path, from the first item not shown + view["more"] = (f"page shortened to {k} items: inspect_evidence(id='{view.get('evidence_id')}', " + f"path='{view['path']}', offset={nxt}, limit={k})") + else: # list_evidence + view["more"] = f"page shortened to {k} items: call again with offset={nxt}, limit={k}" + return view + + +def _fit_resume_page(view: dict, max_chars: int) -> dict: + """Keep resume checks visible when checkpoint prose, rather than page length, is oversized.""" + from inferencekit.ops.store import join_path + + # The first shortening adds continuation metadata. Account for those bytes before accepting the page; + # generic list previews must never replace items while next_offset still skips the unshown rows. + while len(view["items"]) > 1 and _size(view) > max_chars: + view = _fit_page(view, max_chars) + if _size(view) <= max_chars: return view - nxt = view["offset"] + k - view["items"], view["next_offset"] = items[:k], nxt - if "path" in view: # inspect_evidence: repeat the same path, from the first item not shown - view["more"] = (f"page shortened to {k} items: inspect_evidence(id='{view.get('evidence_id')}', " - f"path='{view['path']}', offset={nxt}, limit={k})") - else: # list_evidence - view["more"] = f"page shortened to {k} items: call again with offset={nxt}, limit={k}" + + state_path = view.get("state_path") + normalized_path = str(state_path).rsplit("/", 1)[0] + "/intake.normalized.json" if state_path else None + omitted = {} + + def brief(value, source, path, allowance=800): + if _size(value) <= allowance: + return value + pointer = {"file": source, "path": path} + omitted[path] = pointer + result = {"truncated": True, "source": pointer} + if isinstance(value, str): + result.update(text=value[:allowance // 2], omitted_characters=len(value) - allowance // 2) + elif isinstance(value, (dict, list)): + result["total"] = len(value) + return result + + # Keep policy/cap/authorization facts. Free-form checkpoint prose and full evidence qualifiers remain + # accessible by their exact file paths; an omitted qualifier is never a successful proof. + for key in ("next_action", "best"): + if key in view: + view[key] = brief(view[key], state_path, key) + if isinstance(view.get("stages"), dict): + for name, stage in view["stages"].items(): + if isinstance(stage, dict) and "notes" in stage: + stage["notes"] = brief(stage["notes"], state_path, join_path(join_path("stages", name), "notes"), 400) + view["stages"] = brief(view["stages"], state_path, "stages", 3000) + if isinstance(view.get("policy"), dict): + for key in ("quality", "architecture_changes"): + if key in view["policy"]: + source_key = "intake.quality" if key == "quality" else "derived.architecture_changes" + view["policy"][key] = brief(view["policy"][key], normalized_path, source_key) + if isinstance(view.get("intake"), dict) and "warnings" in view["intake"]: + view["intake"]["warnings"] = brief(view["intake"]["warnings"], normalized_path, "warnings") + for row in view["items"]: + source = row.get("path") + for key in ("summary", "limitations"): + if key in row: + row[key] = brief(row[key], source, key) + if "cited_by" in row: + row["cited_by"] = brief(row["cited_by"], state_path, "") + if isinstance(row.get("coverage"), dict) and "missing" in row["coverage"]: + row["coverage"]["missing"] = brief(row["coverage"]["missing"], source, "coverage.missing") + if "reason" in row: + row["reason"] = brief(row["reason"], source, "") + if isinstance(row.get("integrity"), dict) and "schema_errors" in row["integrity"]: + row["integrity"]["schema_errors"] = brief(row["integrity"]["schema_errors"], source, + "") + view["truncated"] = True + view["detail_sources"] = list(omitted.values()) + view["note"] = ("Resume details shortened; read the named files/paths for omitted facts. " + "Omissions establish neither proof nor authorization. next_call advances past every shown item.") + # Sources can be long paths. Avoid repeating them in this optional index: each shortened field still + # carries its source and the original operation arguments remain in pagination/next_call. + if _size(view) > max_chars: + view.pop("detail_sources", None) + if _size(view) > max_chars: + # Unusually large identifiers or source arguments cannot safely be truncated into executable + # calls. Stop MCP pagination explicitly; the CLI returns the original, untrimmed operation result. + optional = {"budget": view.get("budget"), "policy": view.get("policy"), + "next_action": view.get("next_action"), "state_path": state_path, + "intake": view.get("intake"), "evidence_root": view.get("evidence_root"), + "cli_arguments": view["pagination"]["args"], "items": view["items"]} + stopped = {"truncated": True, "status": "needs_attention", "resume_context_status": view.get("status"), + "trust": view.get("trust"), "evidence_summary": view.get("evidence_summary"), + "error": "Resume metadata exceeds the response limit; no MCP continuation is provided.", + "recovery": "Use inferencekit intake resume with the original run_dir/file/now arguments in " + "the CLI for untrimmed output. Read state.json, intake.normalized.json and evidence/ " + "under the original run_dir. Do not repeat this MCP page; omitted facts establish " + "neither proof nor authorization.", + "omitted_fields": list(optional)} + if _size(stopped) > max_chars: + raise OpError("MCP response limit is too small to retain resume status, trust and evidence counts; " + "use inferencekit intake resume with the original arguments in the CLI") + for key, value in optional.items(): + candidate = {**stopped, key: value, + "omitted_fields": [name for name in stopped["omitted_fields"] if name != key]} + if _size(candidate) <= max_chars: + stopped = candidate + return stopped return view @@ -106,22 +210,37 @@ def fit(result: dict, max_chars: int = MAX_CHARS) -> dict: view = _fit_page(view, max_chars) if _size(view) <= max_chars: return view + if isinstance(view.get("pagination"), dict) and view["pagination"].get("tool") == "intake_resume": + return _fit_resume_page(view, max_chars) record = view.get("record", view) # inspect_evidence wraps the record eid = record.get("evidence_id") if isinstance(record, dict) else None - prefix = "record." if "record" in view else "" candidates = sorted(_lists(view), key=lambda t: -_size(t[3])) - for path, parent, key, items in candidates: + for parts, parent, key, items in candidates: if _size(view) <= max_chars: break if len(items) <= LIST_PREVIEW: continue - inner = path[len(prefix):] if path.startswith(prefix) else path - if _is_page(view) and "path" in view and inner.startswith("items."): + from inferencekit.ops.store import join_path + + parts = parts[1:] if "record" in view and parts[0] == "record" else parts + inner = "" + if view.get("view") == "summary" and parts[0] == "items" and len(parts) > 1: + # Summary findings are projections: their qualifier paths live beside the source decision. + source_path = view["items"][parts[1]]["path"] + inner = source_path.rsplit("/" if source_path.startswith("/") else ".", 1)[0] + parts = parts[2:] + elif _is_page(view) and "path" in view and parts[0] == "items" and len(parts) > 1: # A list inside one item of a page: "items.0.x" is item offset+0 of the paged path. - i, _, rest = inner[len("items."):].partition(".") - inner = f"{view['path']}.{view['offset'] + int(i)}" + (f".{rest}" if rest else "") + inner = join_path(view["path"], view["offset"] + parts[1]) + parts = parts[2:] + elif "path" in view and "value" in view and parts[0] == "value": + # inspect_evidence(path=...) wraps a non-list value; the wrapper is not a source key. + inner, parts = view["path"], parts[1:] + for part in parts: + inner = join_path(inner, part) parent[key] = {"truncated": True, "total": len(items), "first": items[:LIST_PREVIEW], - "more": f"inspect_evidence(id='{eid}', path='{inner}', offset={LIST_PREVIEW}, limit=50)" + "more": f"inspect_evidence(id={view.get('source', eid)!r}, path={inner!r}, " + f"offset={LIST_PREVIEW}, limit=50)" if eid else "call again with offset/limit"} if _size(view) > max_chars: # long text, e.g. a report's markdown (the result also gives its file path) _trim_strings(view, max_chars // 2) @@ -129,9 +248,36 @@ def fit(result: dict, max_chars: int = MAX_CHARS) -> dict: record["data"] = {"truncated": True, "more": f"inspect_evidence(id='{eid}', path='data.') for one part at a time", "keys": sorted(result.get("record", result).get("data", {}).keys())} - if _size(view) > max_chars: # last resort: a summary that always fits - view = {"truncated": True, "evidence_id": eid, "summary": record.get("summary") if isinstance(record, dict) - else None, "more": "use inspect_evidence with a path"} + if _size(view) > max_chars: # last resort: preserve integrity and decision counts for summary inspections + if result.get("view") == "summary": + integrity = result["integrity"] + # Validation errors can contain arbitrarily long invalid values. Keep the verdicts and exact count, + # but do not restore the unbounded messages after trimming the view. + compact_integrity = {"id_ok": integrity["id_ok"], "schema_ok": integrity["schema_ok"], + "schema_error_count": integrity.get("schema_error_count", 0), + "schema_errors_omitted": bool(integrity.get("schema_errors"))} + def bounded_label(value): + if not isinstance(value, str): + return None + if len(value) > 512: + return value[:512] + "... [truncated; use the original input]" + return value + limitations = result.get("limitations") + coverage = result.get("coverage") + missing = coverage.get("missing") if isinstance(coverage, dict) else None + view = {"view": "summary", "truncated": True, "evidence_id": bounded_label(eid), + "source": bounded_label(result["source"]), "integrity": compact_integrity, + "finding_counts": result["finding_counts"], + "total": result["total"], + "limitation_count": len(limitations) if isinstance(limitations, list) else None, + "missing_count": len(missing) if isinstance(missing, list) else None, + "more": "Summary exceeds response budget. Inspect source paths (metrics, coverage, limitations, " + "data) separately; omitted details cannot establish PASS. For schema errors, run " + "inferencekit evidence show with the original id or path; use inferencekit.evidence.validate " + "on the stored record for every validation error."} + else: + view = {"truncated": True, "evidence_id": eid, "summary": record.get("summary") if isinstance(record, dict) + else None, "more": "use inspect_evidence with a path"} return view diff --git a/src/inferencekit/ops/__init__.py b/src/inferencekit/ops/__init__.py index 40ae0c9..8296420 100644 --- a/src/inferencekit/ops/__init__.py +++ b/src/inferencekit/ops/__init__.py @@ -17,6 +17,7 @@ manifest, ) from inferencekit.ops import intake # noqa: F401,E402 +from inferencekit.ops import resume # noqa: F401,E402 from inferencekit.ops import quality # noqa: F401,E402 from inferencekit.ops import bench_serve # noqa: E402,F401 from inferencekit.ops import bench_rounds # noqa: F401,E402 @@ -28,3 +29,6 @@ from inferencekit.ops import gates_clip # noqa: F401,E402 (video: gates clip; needs bench_clip) from inferencekit.ops import bench_image # noqa: F401,E402 from inferencekit.ops import bench_audio # noqa: F401,E402 +from inferencekit.ops import quality_audio # noqa: F401,E402 +from inferencekit.ops import audio_scorers # noqa: F401,E402 +from inferencekit.ops import cpu # noqa: F401,E402 diff --git a/src/inferencekit/ops/audio_scorers.py b/src/inferencekit/ops/audio_scorers.py new file mode 100644 index 0000000..49a957c --- /dev/null +++ b/src/inferencekit/ops/audio_scorers.py @@ -0,0 +1,117 @@ +"""CLI/MCP operations for local speech scorer descriptors and execution.""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +import tempfile + +from inferencekit import evidence as ev +from inferencekit.ops.bench_audio import load_fixtures +from inferencekit.ops.quality_audio import _source_compatibility +from inferencekit.quality import audio as qa, audio_scorers as scorers +from inferencekit.registry import Arg, OpError, operation + + +@operation("quality_audio_scorers", ("quality", "audio-scorers"), + "Describe actual local CPU scorer packages and checkpoint hashes before policy registration.", + [Arg("config", "string", "Local scorer configuration JSON, including optional Python executable.", required=True, positional=True)]) +def quality_audio_scorers(call, config): + return scorers.describe(scorers.load_config(config)) + + +@operation("quality_audio_score", ("quality", "audio-score"), + "Run local offline CPU ASR, UTMOSv2 and ECAPA scorers on retained speech PCM; never human listening.", + [Arg("baseline", "string", "Baseline bench audio evidence ID or path.", required=True), + Arg("candidate", "string", "Candidate bench audio evidence ID or path.", required=True), + Arg("fixtures", "string", "Registered speech fixture JSONL.", required=True), + Arg("protocol", "string", "Registered quality audio-policy evidence ID or path.", required=True), + Arg("config", "string", "Local scorer configuration JSON.", required=True), + Arg("out", "string", "New hash-bound audio scores JSON; refuses overwrite.", required=True)]) +def quality_audio_score(call, baseline, candidate, fixtures, protocol, config, out): + output = Path(out) + if output.exists() or output.is_symlink(): + raise OpError("audio scores output already exists; choose a new file") + settings = scorers.load_config(config) + store = ev.Store() + registered, root = qa.load_record(store, protocol, "quality_audio") + if registered["data"].get("format") != qa.POLICY_FORMAT: + raise OpError("protocol must be a quality audio-policy record") + qa.artifact_bytes(root, qa.one_artifact(registered, "audio_quality_policy")) + policy = qa.validate_policy(registered["data"]["policy"]) + fixture_rows, fixture_sha = load_fixtures(fixtures) + if fixture_sha != registered["data"]["fixture_sha256"]: + raise OpError("audio fixtures differ from the registered policy") + a, arows = qa.load_benchmark(store, baseline) + b, brows = qa.load_benchmark(store, candidate) + _source_compatibility(a, b, policy) + if a["data"]["fixture_set"]["sha256"] != fixture_sha or set(arows) != set(brows): + raise OpError("audio benchmark fixtures or paired requests differ") + roles = set(settings["scorers"]) + expected = set(policy["scorers"]) - {"listening"} + if roles != expected: + raise OpError("configured scorer roles must match every registered model scorer") + if ("asr" in roles and not {"wer", "cer"}.intersection(policy["metrics"])) or any( + role not in policy["metrics"] for role in roles & {"naturalness", "speaker"}): + raise OpError("scorers require corresponding registered quality metrics") + descriptors = scorers.describe(settings) + if descriptors["scorers"] != {k: v for k, v in policy["scorers"].items() if k != "listening"}: + raise OpError("actual scorer packages, checkpoints or settings differ from the registered policy") + references = {f.id: f for f in fixture_rows} + with tempfile.TemporaryDirectory(prefix="inferencekit-audio-inputs-") as directory: + temporary = Path(directory) + jobs = [] + for arm, rows in (("baseline", arows), ("candidate", brows)): + for (fid, round_index), row in sorted(rows.items()): + if not row["ok"] or row["pcm"] is None: + raise OpError("audio scoring requires complete successful saved PCM") + pcm = temporary / f"{len(jobs)}.pcm" + pcm.write_bytes(row["pcm"]) + job = {key: row[key] for key in ("seed", "pcm_sha256", "text_sha256", "sample_rate", "channels", "sample_count", "reference_sha256")} + job.update(arm=arm, id=fid, round=round_index, pcm_path=str(pcm)) + fixture = references[fid] + if fixture.reference_audio is not None and "speaker" in roles: + reference = temporary / f"reference-{fixture.reference_sha256}" + if not reference.exists(): + reference.write_bytes(fixture.reference_audio) + job["reference_path"] = str(reference) + jobs.append(job) + result = scorers.run_worker(settings, "score", {"descriptors": descriptors, "jobs": jobs}) + if not isinstance(result, dict) or set(result) != {"descriptors", "rows"} or result["descriptors"] != descriptors: + raise OpError("audio scoring returned mismatched scorer identities") + rows = result["rows"] + if not isinstance(rows, list) or len(rows) != len(jobs): + raise OpError("audio scorer returned incomplete observations") + required = ({"asr"} if "asr" in roles else set()) | ({"naturalness"} if "naturalness" in roles else set()) + for row in rows: + if not isinstance(row, dict) or "error" in row or "listening" in row or not required.issubset(row): + raise OpError("audio scorer returned failed or incomplete observations") + if "asr_retry" in roles and (not isinstance(row.get("asr"), dict) or "retry_text" not in row["asr"]): + raise OpError("audio scorer omitted registered ASR retry") + if "speaker" in roles and references.get(row.get("id")) and references[row["id"]].reference_audio is not None and "speaker" not in row: + raise OpError("audio scorer omitted fixture voice reference observation") + doc = {"format": qa.SCORES_FORMAT, "protocol": registered["evidence_id"], "baseline": a["evidence_id"], + "candidate": b["evidence_id"], "scorers": policy["scorers"], "rows": rows} + validate_path = temporary / "scores.json" + validate_path.write_text(json.dumps(doc, indent=2, allow_nan=False) + "\n") + qa.load_scores(validate_path, registered["evidence_id"], a["evidence_id"], b["evidence_id"], policy, + {"baseline": arows, "candidate": brows}, {f.id: f.text for f in fixture_rows}) + # Publish a complete validated file using an exclusive hard link; neither + # an existing target nor a concurrent writer can be overwritten. + output.parent.mkdir(parents=True, exist_ok=True) + fd, staging_name = tempfile.mkstemp(prefix=".audio-scores-", dir=output.parent) + try: + with os.fdopen(fd, "wb") as stream: + raw = validate_path.read_bytes() + stream.write(raw) + os.link(staging_name, output) + except FileExistsError: + raise OpError("audio scores output already exists; choose a new file") from None + except OSError: + raise OpError("cannot publish audio scores output") from None + finally: + Path(staging_name).unlink(missing_ok=True) + return {"format": qa.SCORES_FORMAT, "sha256": qa.sha(raw), "rows": len(rows), + "model_scorers": sorted(roles), "listening": "NOT RUN", "quality_verdict": "NOT RUN", + "next": "Import this file with quality audio --scores; model proxies do not establish listening quality."} diff --git a/src/inferencekit/ops/bench_audio.py b/src/inferencekit/ops/bench_audio.py index 3db772a..028717c 100644 --- a/src/inferencekit/ops/bench_audio.py +++ b/src/inferencekit/ops/bench_audio.py @@ -29,6 +29,8 @@ _ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$") ARGS = [ Arg("base_url", "string", "Speech server root; stored only as a keyed endpoint tag.", required=True, sensitive=True), + Arg("endpoint_id", "string", "Stable server identity for --extend across port changes. Set the same ID from " + "the first round; model, engine version and all configuration checks still apply."), Arg("model", "string", "Pinned model identifier sent to the server.", required=True), Arg("model_revision", "string", "Checkpoint revision, sent and required in final effective settings."), Arg("model_dtype", "string", "Checkpoint dtype, sent as dtype and checked in effective settings."), @@ -45,6 +47,8 @@ Arg("api_key_env", "string", "Environment variable containing the key; never a key value."), Arg("engine_version", "string", "Pinned engine version, sent and checked in effective settings."), Arg("hardware_name", "string", "Measured hardware label; never inferred from a remote client."), + Arg("device", "string", "Whether the named server hardware is a GPU or CPU.", default="gpu", choices=("gpu", "cpu")), + Arg("gpu_count", "integer", "Number of GPUs serving audio (GPU mode only).", default=1), Arg("run_dir", "string", "Also copy request/output artifacts and metrics here."), Arg("timeout_s", "number", "Whole-response deadline including DNS, connect, headers and streamed body.", default=300.0), Arg("resamples", "integer", "Round-cluster bootstrap resamples.", default=stats.DEFAULT_RESAMPLES), @@ -175,11 +179,17 @@ def _previous(store, reference, config_sha, call): raise OpError("--extend requires a bench audio record") if ev.integrity_problem(record): raise OpError("--extend evidence integrity check failed") + if record.get("data", {}).get("scheduler_final"): + raise OpError("--extend cannot append to a finalized scheduler record; start a new bench rounds run") if record["data"].get("configuration_sha256") != config_sha: raise OpError("--extend requires the same workload, model, version, hardware and seed policy") old_args = record["data"].get("request", {}).get("args", {}) - if old_args.get("base_url") != call.request()["args"].get("base_url"): - raise OpError("--extend requires the same endpoint") + new_args = call.request()["args"] + if old_args.get("endpoint_id") != new_args.get("endpoint_id"): + raise OpError("--extend requires the same endpoint-id, set from the first round") + if not new_args.get("endpoint_id") and old_args.get("base_url") != new_args.get("base_url"): + raise OpError("--extend endpoint changed (including its port); use a fixed port per label or set " + "--endpoint-id consistently from the first round. Endpoint addresses are kept private.") artifact = next((a for a in record["artifacts"] if a.get("role") == "requests"), None) if artifact is None: raise OpError("--extend per-request artifact missing") @@ -201,13 +211,19 @@ def _previous(store, reference, config_sha, call): "first-audio p95, complete audio, RTF and buffer underruns. Audio quality remains UNPROVEN.", ARGS) def bench_audio(call, base_url, model, model_revision, model_dtype, model_quant, fixtures, label, rounds, first_round, warmup, seed_base, sample_rate, channels, settings, api_key_env, engine_version, hardware_name, run_dir, timeout_s, - resamples, save_outputs, slo_first_audio_p95_ms, slo_rtf, extend): + resamples, save_outputs, slo_first_audio_p95_ms, slo_rtf, extend, endpoint_id=None, device="gpu", gpu_count=1): if rounds < 1 or (first_round is not None and first_round < 0) or warmup < 0 or resamples < 1: raise OpError("rounds/resamples must be positive; first_round/warmup nonnegative") if not _ID.fullmatch(label): raise OpError("invalid audio label") + if endpoint_id is not None and not _ID.fullmatch(endpoint_id): + raise OpError("endpoint-id must contain 1-64 filename-safe characters") if not model.strip(): raise OpError("audio model identifier must be nonempty") + if not 1 <= gpu_count <= 1024 or device == "cpu" and gpu_count != 1: + raise OpError("gpu_count must be 1..1024; do not set it for a CPU server") + if device == "cpu" and not (hardware_name or "").strip(): + raise OpError("CPU audio requires --hardware-name identifying the server CPU") if not 8000 <= sample_rate <= 192000 or not 1 <= channels <= 8: raise OpError("PCM rate must be 8000..192000 Hz and channels 1..8") if any(not math.isfinite(v) or v <= 0 for v in (timeout_s, slo_first_audio_p95_ms, slo_rtf)): @@ -247,6 +263,8 @@ def bench_audio(call, base_url, model, model_revision, model_dtype, model_quant, "seed_policy": {"name": "per_round", "base": seed_base}} request_sha = _sha(canonical_json(workload_config).encode()) config_sha = _sha(canonical_json({**workload_config, "label": label, "hardware": hardware_name, + **({"device": "cpu"} if device == "cpu" else {}), + **({"gpu_count": gpu_count} if gpu_count != 1 else {}), "engine_version": engine_version, "effective_settings": setting_doc, "save_outputs": save_outputs, "slo_first_audio_p95_ms": slo_first_audio_p95_ms, "slo_rtf": slo_rtf}).encode()) @@ -303,7 +321,8 @@ def request(fixture, seed, text=None): if result.ok and save_outputs == "all": rel = f"outputs/{label}/r{round_index}/{fixture.id}.pcm" _write(roots, rel, result.pcm) - outputs.append({"path": rel, "sha256": row["pcm_sha256"], "bytes": row["pcm_bytes"], + outputs.append({"id": fixture.id, "round": round_index, + "path": rel, "sha256": row["pcm_sha256"], "bytes": row["pcm_bytes"], "sample_rate": sample_rate, "channels": channels, "encoding": ENCODING}) valid = [row for row in current_rows if row["ok"]] round_rows.append({"index": round_index, "t_start": start, "t_end": time.time(), "seed": seed, @@ -318,7 +337,8 @@ def request(fixture, seed, text=None): table["rtf"]["max"] > slo_rtf or any(row["underruns"] for row in rows if row["ok"]) else "PASS" verdict = stats.combine_claims([structure, timing, "UNPROVEN"]) data = {"source": "bench audio", "timer": "client perf_counter_ns", "label": label, - "endpoint_kind": "audio_http", "base_url_kind": "local" if endpoint.is_local() else "remote", + "endpoint_kind": "audio_http", "endpoint_id": endpoint_id, + "base_url_kind": "local" if endpoint.is_local() else "remote", "fixture_set": {"sha256": fixture_sha, "fixtures": [f.identity() for f in fixture_rows]}, "configuration_sha256": config_sha, "extends": previous["evidence_id"] if previous else None, "rounds": round_rows, "warmup": {"excluded": True, "requests": warm_rows}, @@ -365,7 +385,9 @@ def request(fixture, seed, text=None): ev.artifact(staging / f"outputs/{label}/index.json", staging, "outputs_index")] for output in outputs: artifacts.append(ev.artifact(staging / output["path"], staging, "audio_pcm")) - identity = ev.identity(gpu={"name": hardware_name}, engine={"name": label, "version": engine_version}, + identity = ev.identity(gpu={"name": hardware_name, "count": gpu_count} if device == "gpu" else None, + cpu={"model": hardware_name} if device == "cpu" else None, + engine={"name": label, "version": engine_version}, model={"id": model, "revision": model_revision, "dtype": model_dtype, "quant": model_quant}, workload={"mode": "single", "batch": 1, "endpoint_kind": "audio_http", "request_set": {"sha256": request_sha, "n": len(fixture_rows)}, diff --git a/src/inferencekit/ops/bench_clip.py b/src/inferencekit/ops/bench_clip.py index 8c397c3..c4654df 100644 --- a/src/inferencekit/ops/bench_clip.py +++ b/src/inferencekit/ops/bench_clip.py @@ -74,6 +74,7 @@ from inferencekit import prompts as sets from inferencekit import stats from inferencekit.canonical import canonical_json, file_sha256 +from inferencekit.cpuinfo import cpu_identity from inferencekit.gpuinfo import gpu_identity from inferencekit.ops.bench_serve import _record_call from inferencekit.registry import Arg, OpError, operation @@ -623,6 +624,8 @@ def bench_clip(call, base_url, clips, endpoint_kind, label, model, rounds, first ep = Endpoint.parse(base_url, api_key_from_env(api_key_env)) except ValueError as e: raise OpError(str(e)) from None + if ep.is_local() and os.environ.get("INFERENCEKIT_CPU_LOCK_HELD") and (gpu_name or gpu_count != 1): + raise OpError("CPU rounds cannot use --gpu-name or a non-default --gpu-count") rcall, key_persistent = _record_call(call, ep) store = ev.Store() prev, prev_rows = (None, []) @@ -762,14 +765,18 @@ def _record(*, rcall, store, staging, run, backend, ep, clipset, order, rounds, "seed_policy": policy, "seeds": [round_seeds[r] for r in sorted(round_seeds)], "audio": audio, "request_set": request_set(clipset, policy, seeds), **{k: clipset.uniform(k) for k in SHAPE_KEYS}} - if gpu_name: + cpu = cpu_identity() if ep.is_local() and os.environ.get("INFERENCEKIT_CPU_LOCK_HELD") else None + if cpu is not None: + gpu = {} + elif gpu_name: gpu = {"name": gpu_name, "count": gpu_count} elif ep.is_local(): gpu = {**(gpu_identity() or {}), "count": gpu_count} else: gpu = {"count": gpu_count} - gpu_missing = [] if gpu.get("name") else ["gpu identity (remote engine or no nvidia-smi: pass --gpu-name)"] - ident = ev.identity(gpu=gpu, engine={"name": label, "version": engine_version}, model={"id": model}, + gpu_missing = (["cpu identity"] if not cpu.get("model") else []) if cpu is not None else \ + ([] if gpu.get("name") else ["gpu identity (remote engine or no nvidia-smi: pass --gpu-name)"]) + ident = ev.identity(gpu=gpu, cpu=cpu, engine={"name": label, "version": engine_version}, model={"id": model}, workload=workload) a = table["all"] cm, cph = a.get("clip_ms") or {}, a.get("clips_per_hour") or {} diff --git a/src/inferencekit/ops/bench_image.py b/src/inferencekit/ops/bench_image.py index a1c9fe8..f99dd5c 100644 --- a/src/inferencekit/ops/bench_image.py +++ b/src/inferencekit/ops/bench_image.py @@ -44,7 +44,9 @@ Arg("quant", "string", "Quantization label, sent and required in effective settings."), Arg("label", "string", "Engine configuration label.", default="image_http"), Arg("engine_version", "string", "Pinned engine version or commit."), - Arg("gpu_name", "string", "Exact GPU model (required; image service may be remote).", required=True), + Arg("gpu_name", "string", "Exact GPU model (required for GPU mode; image service may be remote)."), + Arg("hardware_name", "string", "CPU model for --device cpu; never inferred from the remote client."), + Arg("device", "string", "Server hardware class.", default="gpu", choices=("gpu", "cpu")), Arg("gpu_count", "integer", "Number of GPUs serving images.", default=1), Arg("rounds", "integer", "Timed passes over the image fixtures.", default=3), Arg("first_round", "integer", "First round index; defaults to zero, or previous last + 1 with extend."), @@ -188,6 +190,8 @@ def _previous(store, ref, rcall, ident): previous = store.load(ref) if previous.get("kind") != KIND or ev.integrity_problem(previous): raise OpError("extend needs an intact image benchmark record") + if previous.get("data", {}).get("scheduler_final"): + raise OpError("extend cannot append to a finalized scheduler record; start a new bench rounds run") if previous["identity"] != ident: raise OpError("extend needs identical workload, model, engine version and hardware") old_args = previous.get("data", {}).get("request", {}).get("args", {}) @@ -208,11 +212,15 @@ def _previous(store, ref, rcall, ident): "seeds, effective-setting/cache gates and image artifacts; semantic quality remains UNPROVEN.", ARGS) def bench_image(call, base_url, prompts, model, model_revision, dtype, quant, label, engine_version, gpu_name, gpu_count, rounds, first_round, warmup, seed_policy, seed_base, seeds, timeout_s, resamples, - save_outputs, api_key_env, run_dir, extend, allow_cache="none"): + save_outputs, api_key_env, run_dir, extend, allow_cache="none", hardware_name=None, device="gpu"): if not _LABEL.fullmatch(label or ""): raise OpError("image label must be a safe 1-64 character filename") - if not model.strip() or not gpu_name.strip(): - raise OpError("model and gpu_name must be non-empty") + if not model.strip() or not ((hardware_name if device == "cpu" else gpu_name) or "").strip(): + raise OpError("model and server hardware must be non-empty: --hardware-name for CPU or --gpu-name for GPU") + if device == "cpu" and gpu_name: + raise OpError("--gpu-name cannot describe a CPU server; use --hardware-name") + if device == "cpu" and gpu_count != 1: + raise OpError("do not set --gpu-count for a CPU server") if not 1 <= rounds <= 10000 or not 0 <= warmup <= 100 or not 1 <= gpu_count <= 1024: raise OpError("rounds, warmup or gpu_count is outside its finite bound") if not 1 <= resamples <= 100000 or first_round is not None and not 0 <= first_round <= 1000000: @@ -237,7 +245,8 @@ def bench_image(call, base_url, prompts, model, model_revision, dtype, quant, la rset = request_set(fixtures, model, pins, seed_policy, seed_base, seeds, allow_cache) uniform = {key: fixtures[0][key] if all(row[key] == fixtures[0][key] for row in fixtures) else None for key in ("width", "height", "steps", "guidance_scale")} - ident = ev.identity(gpu={"name": gpu_name, "count": gpu_count}, + ident = ev.identity(gpu={"name": gpu_name, "count": gpu_count} if device == "gpu" else None, + cpu={"model": hardware_name} if device == "cpu" else None, engine={"name": label, "version": engine_version}, model={"id": model, "revision": model_revision, "dtype": dtype, "quant": quant}, workload={"mode": "single", "batch": 1, "task": "t2i", "endpoint_kind": "image_http", diff --git a/src/inferencekit/ops/bench_rounds.py b/src/inferencekit/ops/bench_rounds.py index c464267..5245927 100644 --- a/src/inferencekit/ops/bench_rounds.py +++ b/src/inferencekit/ops/bench_rounds.py @@ -39,6 +39,7 @@ from inferencekit.ops.doctor import gpu_processes from inferencekit.registry import Arg, OpError, operation from inferencekit.serve import rounds as rr +from inferencekit.serve import media_rounds, round_evidence KIND = "serve_benchmark" @@ -132,17 +133,19 @@ def _limitations(result: dict) -> tuple[list[str], list[str]]: return lim, missing -@operation("bench_rounds", ("bench", "rounds"), "Run every engine of a run manifest for R balanced (Williams) " - "rounds, one server at a time: clean launch, readiness, warm-up, bench serve, /metrics, group stop, " - "GPU memory check, resolved-config capture; medians with CIs and ratio to best.", +@operation("bench_rounds", ("bench", "rounds"), "Run manifest engines for balanced fresh-process rounds: " + "text, audio, image or video clients, CPU/GPU locking, cleanup and per-round evidence with CIs.", [Arg("manifest", "string", "Run manifest (runs//manifest.json); its 'engines' are launched.", required=True), - Arg("prompts", "string", "Prompt set (JSONL) passed to every bench serve call (--mode clip: the clip " - "set, passed to every bench clip call).", required=True), + Arg("prompts", "string", "Shared JSONL prompt, speech, image or clip fixture set.", required=True), Arg("rounds", "integer", "Rounds; each runs every engine once. Default: the smallest multiple of the " "Williams period that is at least 3, so the order is balanced (2 engines: 4, 3: 6, 4: 4, 5: 10)."), - Arg("mode", "string", "bench serve mode (single, load), or clip: video clip engines measured with bench " - "clip, --prompts is the clip set (JSONL).", default="single", choices=("single", "load", "clip")), + Arg("mode", "string", "Client: single/load text, audio, image or clip. Auto reads manifest modes, " + "modality, intake modality or endpoint kind; ambiguous modes require a selection.", + default="auto", choices=("auto", "single", "load", "clip", "audio", "image")), + Arg("device", "string", "Hardware route; auto reads hardware.backend from the manifest.", default="auto", + choices=("auto", "cpu", "gpu")), + Arg("cpu_lock", "string", "CPU-exclusive advisory lock (default $INFERENCEKIT_CPU_LOCK or ~/.cache/inferencekit/cpu.lock)."), Arg("engines", "string", "Comma-separated engine labels to run (default: every engine in the manifest)."), Arg("run_dir", "string", "Run directory for logs and results (default: the manifest's directory)."), Arg("gpu_lock", "string", "Lock file held for the whole run (default: $INFERENCEKIT_GPU_LOCK or " @@ -185,27 +188,42 @@ def _limitations(result: dict) -> tuple[list[str], list[str]]: "--rounds, the smallest whole number of Williams periods that is at least 3). Needs --ci-target.")]) def bench_rounds(call, manifest, prompts, rounds, mode, engines, run_dir, gpu_lock, startup_timeout_s, stop_timeout_s, serve_timeout_s, warmup, ref_e2e_ms, serve_args, serve_cmd, gpu_mem_tolerance_mib, - chat_template_kwargs=None, extra_body=None, ci_target=None, min_rounds=None): - if mode == "clip": # video clips: bench clip per cell (serve/clip_rounds.py); --prompts is the clip set + chat_template_kwargs=None, extra_body=None, ci_target=None, min_rounds=None, device="auto", cpu_lock=None): + declared = rr.load_manifest(manifest) + device = round_evidence.resource_mode(declared, device) + mode = media_rounds.mode_for(declared, mode, manifest) + if device == "cpu" and mode == "load": + raise OpError("CPU load mode is not supported yet: capacity metrics use GPU denominators; use --mode single") + if mode in ("clip", "audio", "image"): unsupported = [flag for flag, value in (("--ci-target", ci_target), ("--min-rounds", min_rounds), ("--ref-e2e-ms", ref_e2e_ms), ("--chat-template-kwargs", chat_template_kwargs), ("--extra-body", extra_body)) if value is not None] if unsupported: - raise OpError(f"{', '.join(unsupported)} not supported with --mode clip (bench clip times whole clips; " + raise OpError(f"{', '.join(unsupported)} not supported with --mode {mode} (media clients time complete outputs; " "adaptive stop and text request settings do not apply)") + if mode == "clip": from inferencekit.serve.clip_rounds import bench_rounds_clip return bench_rounds_clip(call, manifest=manifest, clips=prompts, rounds=rounds, engines=engines, run_dir=run_dir, gpu_lock=gpu_lock, startup_timeout_s=startup_timeout_s, stop_timeout_s=stop_timeout_s, clip_timeout_s=serve_timeout_s, warmup=warmup, - clip_args=serve_args, clip_cmd=serve_cmd, gpu_mem_tolerance_mib=gpu_mem_tolerance_mib) - others = gpu_processes() - result = rr.run_rounds(manifest, prompts, rounds, mode, engines=engines, run_dir=run_dir, + clip_args=serve_args, clip_cmd=serve_cmd, gpu_mem_tolerance_mib=gpu_mem_tolerance_mib, + resource_mode=device, cpu_lock_path=cpu_lock) + others = [] if device == "cpu" else gpu_processes() + media = mode in ("audio", "image") + if media: + result = media_rounds.run_media_rounds(manifest, prompts, mode, rounds=rounds, engines=engines, + run_dir=run_dir, device=device, cpu_lock=cpu_lock, gpu_lock=gpu_lock, + startup_timeout_s=startup_timeout_s, stop_timeout_s=stop_timeout_s, + serve_timeout_s=serve_timeout_s, warmup=warmup, serve_cmd=serve_cmd, + serve_args=serve_args, gpu_mem_tolerance_mib=gpu_mem_tolerance_mib) + else: + result = rr.run_rounds(manifest, prompts, rounds, mode, engines=engines, run_dir=run_dir, gpu_lock_path=gpu_lock, startup_timeout_s=startup_timeout_s, stop_timeout_s=stop_timeout_s, serve_timeout_s=serve_timeout_s, warmup=warmup, serve_cmd=serve_cmd, serve_args=serve_args, gpu_mem_tolerance_mib=gpu_mem_tolerance_mib, ref_e2e_ms=ref_e2e_ms, chat_template_kwargs=chat_template_kwargs, extra_body=extra_body, - ci_target=ci_target, min_rounds=min_rounds) + ci_target=ci_target, min_rounds=min_rounds, resource_mode=device, cpu_lock_path=cpu_lock) plan, summary, rdir = result["plan"], result["summary"], Path(result["run_dir"]) labels = [e["label"] for e in plan["engines"]] limitations, missing = _limitations(result) @@ -224,10 +242,10 @@ def bench_rounds(call, manifest, prompts, rounds, mode, engines, run_dir, gpu_lo flags = {"engines": labels} if plan.get("extra_body"): flags["extra_body"] = plan["extra_body"]["value"] # same body for every engine: part of what was measured - ident = ev.identity(gpu=gpu_identity(), + ident = ev.identity(gpu={} if device == "cpu" else gpu_identity(), cpu=plan.get("cpu"), engine={"name": "+".join(labels), "flags": flags}, model={"id": m.get("id"), "revision": m.get("revision"), "dtype": m.get("precision")}, - workload={"mode": mode, "endpoint_kind": "openai_chat", + workload={"mode": "single" if media else mode, "endpoint_kind": f"{mode}_http" if media else "openai_chat", "prompt_set": {"name": plan["prompts"]["name"], "sha256": plan["prompts"]["sha256"]}}) n_ok = sum(1 for c in result["cells"] if c.get("status") == "ok") @@ -247,23 +265,25 @@ def bench_rounds(call, manifest, prompts, rounds, mode, engines, run_dir, gpu_lo if heads: h = heads[0] summary_line += f"; best {h['metric']} [{h['bucket']}]: {h['engine']} {h['median']:.4g} [{cls}]" - artifacts = [ev.artifact(rdir / "rounds" / name, rdir, role) + output_dir = f"{mode}_rounds" if media else "rounds" + artifacts = [ev.artifact(rdir / output_dir / name, rdir, role) for name, role in (("summary.md", "summary"), ("summary.json", "summary"), ("plan.json", "plan")) - if (rdir / "rounds" / name).is_file()] + if (rdir / output_dir / name).is_file()] for c in result["cells"]: - log = rdir / c["log"] + log = rdir / c.get("log", "") if log.is_file(): artifacts.append(ev.artifact(log, rdir, "server_log")) rc = (c.get("resolved_config") or {}).get("file") if rc and (rdir / rc).is_file(): artifacts.append(ev.artifact(rdir / rc, rdir, "resolved_config")) - v2 = KIND in ev.KINDS + kind = f"{mode}_benchmark" if media else KIND + v2 = kind in ev.KINDS if not v2: limitations.append("evidence v2 (kind serve_benchmark) not available in this build: record written " "without schema validation") cells = [{k: c.get(k) for k in ("round", "position", "label", "kind", "status", "served_model", "ready", "warmup", "serve", "stop", "gpu_memory", "resolved_config", "env", "log", - "metrics_source", "load_schedule", "t_start", "t_end", "detail")} + "metrics_source", "load_schedule", "t_start", "t_end", "detail", "resources", "server_env", "audio", "image")} for c in result["cells"]] if load: claim_classes = {"": "derived (median over rounds of bench serve's capacity record: N* and the " @@ -272,7 +292,11 @@ def bench_rounds(call, manifest, prompts, rounds, mode, engines, run_dir, gpu_lo else: claim_classes = {"": "observed (median over rounds of bench serve's client-side timings)", "_ratio_to_best": "derived"} - record = ev.build(call, kind=KIND if v2 else "benchmark", claim_class=cls, summary=summary_line, + if media: + claim_classes = {"": "observed client timer, except simulated first-audio onset (derived)", + ", rtf, _ratio_to_best": "derived", + "peak_rss_mib": "observed sampled server process-group RSS"} + record = ev.build(call, kind=kind if v2 else "benchmark", claim_class=cls, summary=summary_line, identity_=ident, metrics=metrics, artifacts=artifacts, missing=missing, limitations=limitations, data={"run_dir": str(rdir), "plan": plan, "cells": cells, "rows": summary["rows"], @@ -281,7 +305,9 @@ def bench_rounds(call, manifest, prompts, rounds, mode, engines, run_dir, gpu_lo "resolved_config": result["resolved_config"], "bench_serve_evidence": [c["serve"]["evidence_id"] for c in result["cells"] if (c.get("serve") or {}).get("evidence_id")], - "gpu_processes_at_start": len(others), "markdown": result["markdown"]}) + **({f"bench_{mode}_evidence": list(plan.get(f"{mode}_evidence_final", {}).values())} + if media else {}), + "gpu_processes_at_start": None if device == "cpu" else len(others), "markdown": result["markdown"]}) store = ev.Store() if v2: rec = store.save(record) diff --git a/src/inferencekit/ops/bench_serve.py b/src/inferencekit/ops/bench_serve.py index 9ab0bb1..f2ff221 100644 --- a/src/inferencekit/ops/bench_serve.py +++ b/src/inferencekit/ops/bench_serve.py @@ -50,6 +50,7 @@ from inferencekit import evidence as ev from inferencekit import stats from inferencekit.canonical import canonical_json, file_sha256 +from inferencekit.cpuinfo import cpu_identity from inferencekit.gpuinfo import gpu_identity from inferencekit.ops.intake import apply_pin, intake_pin from inferencekit.registry import Arg, Call, OpError, operation @@ -369,15 +370,20 @@ async def _check_server(ep, model) -> tuple[list[str], dict]: def _identity(ep, gpu_name, gpu_count, engine_kind, engine_version, model, workload): missing = [] - if gpu_name: + cpu = cpu_identity() if ep.is_local() and os.environ.get("INFERENCEKIT_CPU_LOCK_HELD") else None + if cpu is not None: + gpu = {} + elif gpu_name: gpu = {"name": gpu_name, "count": gpu_count} elif ep.is_local(): gpu = gpu_identity() or {} else: gpu = {"count": gpu_count} - if not gpu.get("name"): + if cpu is not None and not cpu.get("model"): + missing.append("cpu identity") + elif cpu is None and not gpu.get("name"): missing.append("gpu identity (remote server or no nvidia-smi: pass --gpu-name)") - ident = ev.identity(gpu=gpu, engine={"name": engine_kind, "version": engine_version}, model={"id": model}, + ident = ev.identity(gpu=gpu, cpu=cpu, engine={"name": engine_kind, "version": engine_version}, model={"id": model}, workload=workload) return ident, missing @@ -447,6 +453,11 @@ def bench_serve(call, base_url, model, prompts, api, engine_kind, mode, max_toke ep = cl.Endpoint.parse(base_url, cl.api_key_from_env(api_key_env)) except ValueError as e: raise OpError(str(e)) from None + if ep.is_local() and os.environ.get("INFERENCEKIT_CPU_LOCK_HELD"): + if mode == "load": + raise OpError("CPU load mode is not supported: capacity metrics currently use GPU normalization; use --mode single") + if gpu_name or gpu_count != 1: + raise OpError("CPU rounds cannot use --gpu-name or a non-default --gpu-count") if warmup < 0: raise OpError("warmup must be >= 0") pset = prompts_io.load(prompts, buckets or None, limit_per_bucket or None) @@ -600,7 +611,7 @@ async def run(): summary += f"; {len(failed)} failed" data = {"label": label, "base_url_kind": "local" if ep.is_local() else "remote", "server": info, "prompt_set_sha256": pset.sha256, "prompt_set_name": pset.name, "buckets": table, - "rounds": round_rows, "request_profile": {"engine_kind": runner.engine_kind, "api": runner.api, + "rounds": round_rows, "request_accounting": _request_accounting(all_rows), "request_profile": {"engine_kind": runner.engine_kind, "api": runner.api, "asked": runner.asked, "note": profiles.PROFILES[runner.engine_kind].note}, "prefix_cache": {"mode": "cold" if cache_salt else "warm", @@ -622,6 +633,23 @@ async def run(): return store.save(record, staging) +def _request_accounting(rows): + """Summarize timed rows after collection; token totals cover successful requests only.""" + ok = [r for r in rows if r["ok"]] + counts = [n for r in ok if type(n := r.get("n_tokens")) is int and n >= 0] + return {"timed_requests": len(rows), "successful_requests": len(ok), + "failed_requests": len(rows) - len(ok), "selected_token_counts": len(counts), + "unknown_token_counts": len(ok) - len(counts), + "potentially_estimated_counts": sum( + 1 for r in ok if type(r.get("n_tokens")) is int and r["n_tokens"] >= 0 + and not (r.get("token_count_method") == "usage" + or str(r.get("token_count_method") or "").startswith("stream_token_ids:"))), + "selected_output_tokens_total": sum(counts) if counts else None, + "selected_output_tokens_min": min(counts) if counts else None, + "selected_output_tokens_max": max(counts) if counts else None, + "successful_count_mismatches": sum(bool(r.get("count_mismatch")) for r in ok)} + + def _method_counts(rows): out = {} for r in rows: @@ -778,7 +806,7 @@ async def run(): "levels": level_data, "salted_requests": pool.salted, "timer": "client perf_counter_ns", "request_profile": {"engine_kind": runner.engine_kind, "api": runner.api, "asked": runner.asked, "note": profiles.PROFILES[runner.engine_kind].note}, - "token_count_methods": _method_counts(all_rows)} + "token_count_methods": _method_counts(all_rows), "request_accounting": _request_accounting(all_rows)} if not [r for r in all_rows if r["ok"]]: missing.append("every request failed") kind_word = "closed-loop stress" if closed_loop else "open-loop" diff --git a/src/inferencekit/ops/compare.py b/src/inferencekit/ops/compare.py index 4adb916..8d67e11 100644 --- a/src/inferencekit/ops/compare.py +++ b/src/inferencekit/ops/compare.py @@ -38,11 +38,13 @@ from __future__ import annotations import math +import re from inferencekit import evidence as ev from inferencekit import stats from inferencekit.registry import Arg, OpError, operation from inferencekit.serve.clip_client import TASKS as CLIP_TASKS +from inferencekit.serve.round_evidence import MAX_ROUNDS, MAX_ROUND_INDEX, finite, memory_of, round_coverage # bench clip's rounds: every clip, then one task's clips (bench_clip.round_key). CLIP_METRICS = ("clip_ms", "clips_per_hour") + tuple(f"{m}_{t}" for t in CLIP_TASKS @@ -135,7 +137,7 @@ def media_workload_reasons(a: dict, b: dict, image_tradeoff: bool = False) -> li workload = rec["identity"].get("workload") or {} if not (workload.get("request_set") or {}).get("sha256"): out.append(f"{label} missing request_set hash") - if not rec["identity"]["gpu"].get("name"): + if not rec["identity"]["gpu"].get("name") and not (rec["identity"].get("cpu") or {}).get("model"): out.append(f"{label} missing hardware identity") if not rec["identity"]["model"].get("id"): out.append(f"{label} missing model identity") @@ -163,19 +165,85 @@ def media_workload_reasons(a: dict, b: dict, image_tradeoff: bool = False) -> li def round_values(record: dict, metric: str) -> dict[int, dict]: """{round index: round} for rounds that have the metric.""" out = {} - for r in record.get("data", {}).get("rounds", []) or []: - if r.get(metric) is not None: - out[int(r["index"])] = r + rows = record.get("data", {}).get("rounds") or [] + for r in rows[:MAX_ROUNDS] if isinstance(rows, list) else []: + if isinstance(r, dict) and type(r.get("index")) is int and 0 <= r["index"] <= MAX_ROUND_INDEX and finite(r.get(metric)): + out[r["index"]] = r return out +def cpu_hardware_reasons(a: dict, b: dict) -> list[str]: + ca, cb = (r["identity"].get("cpu") for r in (a, b)) + if ca is None and cb is None: + return [] + if ca is None or cb is None: + return ["hardware backends differ or CPU identity is missing from one arm"] + reasons = [] + for label, rec, cpu in (("baseline", a, ca), ("candidate", b, cb)): + if rec["identity"]["gpu"].get("name"): + reasons.append(f"{label} mixes CPU and GPU hardware identities") + if not cpu.get("model"): + reasons.append(f"{label} missing CPU model identity") + # Affinity and threads are tuning knobs; topology is the host's identity. + for key in ("model", "architecture", "logical_cores", "physical_cores", "topology_sha256"): + if ca.get(key) is not None and cb.get(key) is not None and ca[key] != cb[key]: + reasons.append(f"different CPU hardware ({key})") + return reasons + + +def lifecycle_verdict(record: dict, coverage: dict) -> str: + data = record.get("data") or {} + proof = data.get("lifecycle") or {} + contexts = data.get("runtime_rounds") or [] + expected = coverage.get("expected_indices") + if not (data.get("scheduler_final") is True and proof.get("source") == "bench rounds" + and proof.get("fresh_process_per_cell") is True and proof.get("valid") is True + and proof.get("verdict") == "PASS" and coverage.get("complete") is True + and expected and len(contexts) == len(expected)): + return "UNPROVEN" + indices = [r.get("index") for r in contexts if isinstance(r, dict)] + if (len(indices) != len(contexts) or any(type(i) is not int for i in indices) + or len(set(indices)) != len(indices) or set(indices) != set(expected)): + return "UNPROVEN" + if not all(r.get("status") == "ok" and (r.get("ready") or {}).get("status") == "ready" + and (r.get("stop") or {}).get("group_gone") is True for r in contexts): + return "UNPROVEN" + return "PASS" + + +def arm_environment(record: dict) -> dict | None: + """Selected scheduler-recorded environment facts; no arbitrary engine flags.""" + env = (record.get("identity", {}).get("engine", {}).get("flags") or {}).get("server_env") + if not isinstance(env, dict): + return None + return {key: env[key] for key in ("controls", "ld_preload", "omitted", "varied_between_rounds") if key in env} + + +def environment_differences(environments: dict) -> list[str]: + a, b = environments["baseline"], environments["candidate"] + if a is None or b is None: + return ["availability"] if a != b else [] + ca, cb = a.get("controls") or {}, b.get("controls") or {} + if not isinstance(ca, dict) or not isinstance(cb, dict): + return ["unverified controls"] + names = [k for k in sorted(set(ca) | set(cb)) if isinstance(k, str) + and re.fullmatch(r"[A-Z][A-Z0-9_]{0,63}", k) and ca.get(k) != cb.get(k)] + if a.get("ld_preload") != b.get("ld_preload"): + names.append("LD_PRELOAD") + if a.get("varied_between_rounds") != b.get("varied_between_rounds"): + names.append("round stability") + if a.get("omitted") != b.get("omitted"): + names.append("omitted controls") + return names + + def interleaved(base: dict[int, dict], cand: dict[int, dict], pairs: list[int]) -> bool | None: """True if each pair's rounds are adjacent in time; None if timestamps are missing.""" events = [] for arm, rounds in (("b", base), ("c", cand)): for i in pairs: t = rounds[i].get("t_start") - if t is None: + if not finite(t): return None events.append((t, arm, i)) events.sort() @@ -192,19 +260,82 @@ def interleaved(base: dict[int, dict], cand: dict[int, dict], pairs: list[int]) Arg("metric", "string", "Per-round metric to compare.", default="decode_tok_s", choices=tuple(HIGHER_IS_BETTER)), Arg("threshold", "number", "Practical threshold as a fraction (0.02 = 2%).", default=0.02), + Arg("expected_rounds", "integer", "Intended total rounds per arm, indexed from zero; checks missing tail rounds when no plan was recorded."), Arg("image_tradeoff", "boolean", "Compare image timing with permitted step/precision differences; " "quality and lifecycle require separate proof. Requires image task identities.", default=False), Arg("resamples", "integer", "Bootstrap resamples.", default=stats.DEFAULT_RESAMPLES), Arg("seed", "integer", "Bootstrap seed.", default=0)]) -def compare(call, baseline, candidate, metric, threshold, resamples, seed, image_tradeoff=False): +def compare(call, baseline, candidate, metric, threshold, resamples, seed, image_tradeoff=False, expected_rounds=None): + if expected_rounds is not None and not 1 <= expected_rounds <= MAX_ROUNDS: + raise OpError(f"expected_rounds must be positive and <= {MAX_ROUNDS}") store = ev.Store() a, b = store.load(baseline), store.load(candidate) higher = HIGHER_IS_BETTER[metric] ra, rb = round_values(a, metric), round_values(b, metric) limitations, missing = [], [] reasons_inconclusive = [] + arm_coverage = {label: round_coverage(rec, expected_rounds) for label, rec in (("baseline", a), ("candidate", b))} + memory = {label: memory_of(rec) for label, rec in (("baseline", a), ("candidate", b))} + environments = {label: arm_environment(rec) for label, rec in (("baseline", a), ("candidate", b))} for rec, label in ((a, "baseline"), (b, "candidate")): + coverage = arm_coverage[label] + for key, what in (("invalid_indices", "invalid or oversized round indices"), + ("invalid_plan", "invalid round plan"), ("oversized_plan", "oversized round plan"), + ("oversized_rows", "too many recorded rounds")): + if coverage[key]: + reasons_inconclusive.append(f"{label} {what}") + plan = rec.get("data", {}).get("round_plan") or {} + recorded_expected = plan.get("expected_indices") if isinstance(plan, dict) else None + if expected_rounds is not None and recorded_expected is not None and recorded_expected != list(range(expected_rounds)): + reasons_inconclusive.append(f"{label} --expected-rounds conflicts with its recorded round plan") + raw_rows = rec.get("data", {}).get("rounds") or [] + rows = raw_rows[:MAX_ROUNDS] if isinstance(raw_rows, list) else [] + if any(not isinstance(r, dict) or type(r.get("index")) is not int or r["index"] < 0 for r in rows): + reasons_inconclusive.append(f"{label} has invalid round indices") + coverage["complete"] = False + for key, what in (("missing_indices", "missing requested rounds"), ("unexpected_indices", "rounds outside the requested plan"), + ("gaps", "gaps in observed rounds"), + ("duplicate_indices", "duplicate round indices")): + if coverage[key]: + reasons_inconclusive.append(f"{label} {what}: {coverage[key]}") + coverage["complete"] = False + metric_indices = set((ra if label == "baseline" else rb)) + metric_missing = sorted(set(coverage["actual_indices"]) - metric_indices) + coverage["metric_indices"] = sorted(metric_indices) + coverage["missing_metric_indices"] = metric_missing + if metric_missing: + reasons_inconclusive.append(f"{label} missing finite {metric} in rounds: {metric_missing}") + coverage["complete"] = False + if coverage["expected_indices"] is None: + limitations.append(f"{label} planned round count unknown; missing tail rounds cannot be checked " + "without --expected-rounds or a recorded round plan") + if not memory[label]: + limitations.append(f"{label} peak RSS/VRAM was not recorded; memory use is unknown") + for key, measurement in memory[label].items(): + if measurement.get("complete") is not True: + qualifier = "partial" if measurement.get("complete") is False else "unknown" + limitations.append(f"{label} {key} coverage is {qualifier}; the recorded value does not establish " + "the peak over every requested round") + if measurement.get("verdict") not in (None, "PASS"): + limitations.append(f"{label} {key} source memory verdict: {measurement['verdict']}") + limitations.extend(f"{label} {key}: {note}" for note in measurement.get("limitations", []) + if isinstance(note, str)) + if measurement.get("method") == "sampled_process_group_rss": + limitations.append(f"{label} peak RSS is sampled process-group RSS: a lower bound on the true peak, " + "with shared pages counted per process") + server_env = (rec.get("identity", {}).get("engine", {}).get("flags") or {}).get("server_env") + envs = [server_env] if isinstance(server_env, dict) else [] + if isinstance(server_env, dict) and "varied_between_rounds" in server_env: + reasons_inconclusive.append(f"{label} server tuning environment varied or was unavailable between rounds") + envs = server_env["varied_between_rounds"] if isinstance(server_env["varied_between_rounds"], list) else [] + if any(isinstance(env, dict) and (env.get("omitted") or any( + value is None for value in (env.get("controls") or {}).values())) for env in envs): + limitations.append(f"{label} requested CPU controls were omitted or invalid; the effective tuning " + "configuration is unverified") + if any(isinstance(env, dict) and any(isinstance(preload, dict) and preload.get("status") != "hashed" + for preload in env.get("ld_preload", [])) for env in envs): + limitations.append(f"{label} preload library identity was not fully verified") problem = ev.integrity_problem(rec) if problem: # an edited input must not turn into a fresh, verifying comparison reasons_inconclusive.append(f"{label} {problem}") @@ -215,7 +346,7 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image "mode") == "load": reasons_inconclusive.append(f"{label} is a load-mode serve_benchmark (one level, no rounds); compare " "capacity records' N* instead") - failed = sum(int(r.get("n_failed") or 0) for r in rec.get("data", {}).get("rounds", []) or []) + failed = sum(int(r.get("n_failed") or 0) for r in rows if isinstance(r, dict)) if failed and rec.get("kind") in ("serve_benchmark", "clip_benchmark", *MEDIA_METRICS): # Round medians drop failed requests: an arm that fails the slow prompts would look faster. reasons_inconclusive.append(f"{label} had {failed} failed request(s); its round medians exclude them, " @@ -250,8 +381,8 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image reasons_inconclusive += media_workload_reasons(a, b, image_tradeoff=image_tradeoff) if metric not in MEDIA_METRICS[a["kind"]]: reasons_inconclusive.append(f"{metric} is not measured by {a['kind']}") - limitations.append("This compares client-timed performance only; image/speech quality, fresh-process " - "lifecycle and repeated-run reliability require their own evidence.") + limitations.append("This compares client-timed performance only; image/speech quality and repeated-run " + "reliability require their own evidence. Fresh-process lifecycle is checked separately.") if image_tradeoff: limitations.append("Image format/work tradeoff: steps and numerical formats may differ. " "Timing is not a quality or same-work engine comparison.") @@ -277,6 +408,10 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image ga, gb = a["identity"]["gpu"].get("name"), b["identity"]["gpu"].get("name") if ga and gb and ga != gb: reasons_inconclusive.append(f"different GPUs ({ga} vs {gb})") + reasons_inconclusive += cpu_hardware_reasons(a, b) + if a["identity"].get("cpu") is not None and b["identity"].get("cpu") is not None: + if any(not (r["identity"]["cpu"] or {}).get("topology_sha256") for r in (a, b)): + limitations.append("CPU topology identity is incomplete; equal model labels alone do not verify the same topology") unavailable = [label for rec, rounds, label in ((a, ra, "baseline"), (b, rb, "candidate")) if not rounds and rec.get("data", {}).get("rounds")] @@ -289,6 +424,9 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image why = f"{metric} is absent from every round of the {' and '.join(unavailable)} record" reasons_inconclusive.append(why + "; more rounds will not help, pick another --metric") pairs = sorted(set(ra) & set(rb)) + if set(ra) != set(rb): + reasons_inconclusive.append(f"unmatched metric rounds (baseline only {sorted(set(ra) - set(rb))}; " + f"candidate only {sorted(set(rb) - set(ra))})") if clips or media: reasons_inconclusive += clip_seed_reasons(ra, rb, pairs) paired = len(pairs) >= 2 @@ -298,10 +436,6 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image reasons_inconclusive.append("media comparisons need at least three complete matching round pairs") if inter is not True: reasons_inconclusive.append("media comparison requires verified interleaving") - for rec, label in ((a, "baseline"), (b, "candidate")): - indices = [r.get("index") for r in rec.get("data", {}).get("rounds", [])] - if len(indices) != len(set(indices)): - reasons_inconclusive.append(f"{label} has duplicate round indices") windows = [] for label, rows in (("baseline", ra), ("candidate", rb)): for index, row in rows.items(): @@ -354,12 +488,40 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image if ratio else f"{metric}: no ratio ({decision['reason']})") if image_tradeoff: summary = "image format/work tradeoff timing; quality requires separate evidence: " + summary + memory_summary = [] + for key, name in (("peak_rss_mib", "peak RSS"), ("peak_vram_mib", "peak VRAM")): + if any(key in row for row in memory.values()): + values = [] + for label in ("baseline", "candidate"): + measurement = memory[label].get(key) + if measurement is None: + values.append(f"{label} unknown") + continue + qualifiers = [] + if measurement.get("method") == "sampled_process_group_rss": + qualifiers.append("sampled") + if measurement.get("complete") is not True: + qualifiers.append("partial" if measurement.get("complete") is False else "coverage unknown") + suffix = " (" + ", ".join(qualifiers) + ")" if qualifiers else "" + values.append(f"{label} {measurement['value']:.1f} MiB{suffix}") + memory_summary.append(name + " " + ", ".join(values)) + if memory_summary: + summary += "; " + "; ".join(memory_summary) + env_differences = environment_differences(environments) + if env_differences: + note = ", ".join(env_differences[:4]) + (", ..." if len(env_differences) > 4 else "") + summary += "; server env differs: " + note + limitations.append("Requested server environment differs between arms; controls and preload identities " + "are retained in data.arm_environment. Requested settings do not prove runtime dispatch.") metrics = [ev.metric(f"{metric}_ratio", ratio, "x", "ratio_of_medians", n=n, ci95=[lo, hi])] for rec, rounds, label in ((a, ra, "baseline"), (b, rb, "candidate")): vals = [r[metric] for r in rounds.values()] s = stats.summarize(vals) metrics.append(ev.metric(metric, s.get("median"), UNITS.get(metric) or ("tok/s" if higher else "ms"), "median", n=len(vals), subject=label)) + for key, measurement in memory[label].items(): + metrics.append(ev.metric(key, measurement["value"], "MiB", "max", subject=label)) + lifecycle = {label: lifecycle_verdict(rec, arm_coverage[label]) for label, rec in (("baseline", a), ("candidate", b))} record = ev.build(call, kind="comparison", claim_class="derived", summary=summary, identity_=b["identity"], metrics=metrics, missing=missing, limitations=limitations, data={"verdict": decision["verdict"], "decision": decision, "metric": metric, @@ -369,9 +531,12 @@ def compare(call, baseline, candidate, metric, threshold, resamples, seed, image "resamples": resamples, "seed": seed, "baseline": a["evidence_id"], "candidate": b["evidence_id"], "baseline_identity": a["identity"], + "arm_round_coverage": arm_coverage, "expected_rounds": expected_rounds, + "memory": memory, "arm_environment": environments, **({"source_kind": a["kind"], "required_proofs": { - "media quality": "UNPROVEN", "fresh-process lifecycle": "UNPROVEN", + "media quality": "UNPROVEN", "fresh-process lifecycle": "PASS" if all(v == "PASS" for v in lifecycle.values()) else "UNPROVEN", "repeated-run reliability": "UNPROVEN"}, + "arm_lifecycle_verdicts": lifecycle, "arm_timing_verdicts": {label: rec.get("data", {}).get("timing_verdict", "NOT RUN") for label, rec in (("baseline", a), ("candidate", b))}} if media else {})}) diff --git a/src/inferencekit/ops/cpu.py b/src/inferencekit/ops/cpu.py new file mode 100644 index 0000000..d0bd74b --- /dev/null +++ b/src/inferencekit/ops/cpu.py @@ -0,0 +1,124 @@ +"""Prepare a CPU baseline thread/placement screen; never launch it.""" +from __future__ import annotations + +import copy +import json +from pathlib import Path + +from inferencekit.cpuinfo import cpu_facts, cpu_identity, cpu_list +from inferencekit.ops.manifest import check_manifest, load_manifest +from inferencekit.registry import Arg, OpError, operation + +THREAD_ENV = ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", "BLIS_NUM_THREADS", "NUMEXPR_NUM_THREADS") + + +def placement_order(facts, selected, spread=False, include_smt=False): + rows = [r for r in facts.get("topology", []) if r["cpu"] in selected] + if len(rows) != len(selected) or any(r.get("package") is None or r.get("core") is None for r in rows): + raise OpError("CPU topology is incomplete for the selected CPUs; pinning needs verified physical cores") + by_core = {} + for r in sorted(rows, key=lambda r: r["cpu"]): + by_core.setdefault((r["package"], r["core"]), []).append(r) + # Each physical core contributes once before SMT siblings. Spread round-robins + # observed LLC/NUMA/core-type groups; it does not invent P/E labels. + groups = {} + for key, siblings in by_core.items(): + r = siblings[0] + l3 = next((i for i, c in enumerate(facts.get("l3_groups", [])) if r["cpu"] in c["cpus"]), -1) + node = next((n["node"] for n in facts.get("numa_nodes", []) if r["cpu"] in n["cpus"]), -1) + group = (r["package"], node, l3, r.get("core_type") or 0) + groups.setdefault(group, []).append(siblings) + ordered = [groups[k] for k in sorted(groups)] + cores = ([g[i] for i in range(max(map(len, ordered))) for g in ordered if i < len(g)] + if spread else [core for g in ordered for core in g]) + result = [core[0]["cpu"] for core in cores] + if include_smt: + result += [r["cpu"] for core in cores for r in core[1:]] + return result + + +def screen_manifest(manifest, *, facts, threads, pinning, thread_env=(), cpus=None, include_smt=False): + """Pure manifest expansion. Engine knobs change only through explicit placeholders/env names.""" + if len(manifest.get("engines", [])) != 1: + raise OpError("CPU screen needs exactly one baseline engine") + if not threads or any(type(n) is not int or n < 1 for n in threads): + raise OpError("threads must be positive integers") + if not pinning or set(pinning) - {"none", "compact", "spread"}: + raise OpError("pinning must contain none, compact, or spread") + if set(thread_env) - set(THREAD_ENV): + raise OpError(f"thread_env must use only {list(THREAD_ENV)}") + baseline = manifest["engines"][0] + argv = baseline.get("launch", []) + if not any("{threads}" in arg for arg in argv) and not thread_env: + raise OpError("baseline launch needs {threads}, or specify --thread-env for a supported runtime") + if not any("{port}" in arg for arg in argv): + raise OpError("baseline launch needs {port} so each variant can use its own port") + if not argv or Path(argv[0]).name in ("sh", "bash", "dash", "zsh", "taskset", "numactl"): + raise OpError("baseline launch must be direct engine argv without shell or existing placement wrapper") + allowed = facts.get("allowed_cpus") + if not allowed: + raise OpError("process CPU affinity is unknown; cannot prepare a placement screen") + selected = sorted(set(cpus if cpus is not None else allowed)) + if not selected or set(selected) - set(allowed): + raise OpError("selected CPUs must be a nonempty subset of the process's allowed CPUs") + counts, policies = sorted(set(threads)), list(dict.fromkeys(pinning)) + if max(counts) > len(selected): + raise OpError("thread count exceeds selected logical CPUs") + result = copy.deepcopy(manifest) + result["hardware"] = {**{k: v for k, v in manifest.get("hardware", {}).items() if k != "gpu"}, + "backend": "cpu", "cpu": cpu_identity(facts), "count": 1} + result["engines"] = [] + plan = [] + for policy in policies: + order = placement_order(facts, selected, policy == "spread", include_smt) if policy != "none" else selected + for n in counts: + if n > len(order): + raise OpError("thread count exceeds selected physical cores; use --include-smt to screen siblings") + port = baseline["port"] + len(plan) + if port > 65535: + raise OpError("variant ports exceed 65535; choose a lower baseline port") + label = f"{baseline['label'][:40]}-t{n}-{policy}" + engine = copy.deepcopy(baseline) + engine.update(label=label, port=port) + engine.pop("resolved_config", None) # baseline's observed settings are not evidence for a new variant + launch = [arg.replace("{threads}", str(n)).replace("{port}", str(port)) for arg in argv] + mask = order[:n] if policy != "none" else selected + # Even the no-binding case is contained in the selected engine pool; + # runtime workers may migrate within it, leaving reserved client CPUs free. + engine["launch"] = ["taskset", "-c", ",".join(map(str, mask)), *launch] + engine["env"] = {**engine.get("env", {}), **{k: str(n) for k in thread_env}, + "OMP_PLACES": "cores", "OMP_PROC_BIND": {"none": "false", "compact": "close", "spread": "spread"}[policy]} + result["engines"].append(engine) + plan.append({"label": label, "threads": n, "pinning": policy, "cpus": mask, "port": port}) + checked = check_manifest(result) + if not checked["valid"]: + raise OpError("invalid CPU screen manifest: " + "; ".join(checked["errors"])) + return result, plan + + +@operation("cpu_screen_manifest", ("cpu", "screen-manifest"), + "Prepare thread x pinning variants of one CPU baseline; writes a manifest, launches nothing.", + [Arg("manifest", "string", "Single-engine manifest; launch uses {threads} and {port} placeholders.", required=True, positional=True), + Arg("out", "string", "New JSON manifest beside the source; refuses overwrite.", required=True), + Arg("threads", "array", "Thread counts to screen; repeat --threads for each count.", required=True, items="integer"), + Arg("pinning", "array", "Placement policies.", default=["none", "compact", "spread"], choices=("none", "compact", "spread")), + Arg("thread_env", "array", "Explicit runtime thread variables to set; launch {threads} is an alternative.", default=[], choices=THREAD_ENV), + Arg("cpus", "string", "Engine CPU pool, Linux CPU list; leave client cores outside it."), + Arg("include_smt", "boolean", "Allow sibling threads after one thread per physical core.", default=False)]) +def cpu_screen_manifest(call, manifest, out, threads, pinning, thread_env, cpus, include_smt): + target, source = Path(out), Path(manifest) + if target.exists(): + raise OpError("output already exists; choose a new path") + if target.resolve().parent != source.resolve().parent: + raise OpError("write the screen manifest beside the source to preserve relative paths") + selected = cpu_list(cpus) if cpus is not None else None + if cpus is not None and not selected: + raise OpError("cpus must be a valid nonempty Linux CPU list") + result, plan = screen_manifest(load_manifest(source), facts=cpu_facts(), threads=threads, pinning=pinning, + thread_env=thread_env, cpus=selected, include_smt=include_smt) + with target.open("x") as handle: + handle.write(json.dumps(result, indent=2) + "\n") + return {"path": str(target), "variants": plan, "ran": False, + "limitations": ["Screen candidates only: verify actual threads, affinity and dispatch in engine logs; " + "select the baseline for the declared workload metric, then use independent interleaved evidence.", + "taskset controls CPU affinity, not NUMA memory policy; record memory placement separately."]} diff --git a/src/inferencekit/ops/doctor.py b/src/inferencekit/ops/doctor.py index b9d53d1..0977e98 100644 --- a/src/inferencekit/ops/doctor.py +++ b/src/inferencekit/ops/doctor.py @@ -23,6 +23,7 @@ from pathlib import Path from inferencekit import evidence as ev +from inferencekit.cpuinfo import cpu_facts, cpu_identity from inferencekit.gpuinfo import clocks_state, driver_cuda_version, query_gpus from inferencekit.hardware import discover_non_nvidia, profiler_route from inferencekit.proc import run, tool_version, which @@ -368,7 +369,7 @@ def gpu_processes() -> list[dict]: return out -@operation("doctor", ("doctor",), "Hardware identity and profiling route (NVIDIA, AMD ROCm, Apple Metal); " +@operation("doctor", ("doctor",), "Hardware identity and profiling route (NVIDIA, AMD ROCm, Apple Metal, CPU); " "selected AMD/Apple OS facts; NVIDIA driver, CUDA, clocks, nsys/ncu and profiling-counter permission " "(driver setting and CAP_SYS_ADMIN, never assumed from root), engine versions; optionally a tiny ncu and " "nsys test capture.", @@ -384,6 +385,8 @@ def doctor(call, python, probe_profilers=False): hardware = None if gpus else discover_non_nvidia() if hardware and (hardware["devices"] or hardware["platform"] == "unknown"): return _vendor_doctor(call, python, probe_profilers, hardware) + if hardware and hardware["platform"] == "cpu": + return _cpu_doctor(call, python, probe_profilers, hardware) limitations, missing = [], [] if not gpus: missing.append("gpu") @@ -451,6 +454,42 @@ def doctor(call, python, probe_profilers=False): return ev.Store().save(record) +def _cpu_doctor(call, python, probe_profilers, hardware): + facts = hardware.get("cpu") or cpu_facts() + hardware = {**hardware, "cpu": facts} + limitations = list(hardware.get("limitations", [])) + list(facts.get("limitations", [])) + limitations += ["NVIDIA-only ceiling bw and nsys/ncu capture are NOT RUN on CPU.", + "CPU profiler permission is inferred from policy only; containers and security policy may still block perf.", + facts["library_dispatch"]["reason"]] + engines = engine_versions(python) + # Package metadata is a presence/version observation, never a framework import or dispatch probe. + libraries = {} + for name in ("numpy", "scipy", "mkl", "intel-openmp", "onednn"): + if not python or os.path.realpath(python) == os.path.realpath(sys.executable): + try: + libraries[name] = metadata.version(name) + except metadata.PackageNotFoundError: + pass + facts["library_dispatch"]["package_versions"] = libraries + data = {"hardware": hardware, "gpus": [], "tools": {"perf": tool_version("perf")}, + "profiling_route": profiler_route("cpu", facts), + "profiling_counters": {"status": "not_applicable", "reason": "NVIDIA counters do not apply; CPU perf policy is in hardware.cpu.perf"}, + "nvcc_toolkit": None, "engines": engines, "llamacpp": llamacpp_version(), "gpu_processes": [], + "env_set": sorted(facts.get("tuning_env", {}))} + if probe_profilers: + data["profiler_probe"] = {"program": None, "ncu": {"ran": False, "capture_ok": None, "error": "NVIDIA-only probe"}, + "nsys": {"ran": False, "capture_ok": None, "error": "NVIDIA-only probe"}} + limitations.append("--probe-profilers is NVIDIA-only; no CPU workload or capture was launched.") + record = ev.build(call, kind="environment", claim_class="observed", + summary=f"CPU {facts.get('model') or facts['architecture']}; {facts.get('physical_cores')} physical / " + f"{facts.get('logical_cores')} logical cores; perf policy {facts['perf']['access']}; dispatch unverified", + identity_=ev.identity(cpu=cpu_identity(facts)), + metrics=[ev.metric("cpu_logical_cores", facts.get("logical_cores"), "count", "observed")], + missing=["CPU runtime library dispatch", "verified CPU profiler permission"], + limitations=limitations, data=data) + return ev.Store().save(record) + + def _vendor_doctor(call, python, probe_profilers, hardware): """Report observed vendor facts without running CUDA-only permission/probe code.""" backend, devices = hardware["platform"], hardware["devices"] diff --git a/src/inferencekit/ops/intake.py b/src/inferencekit/ops/intake.py index 48cafed..87a4ed7 100644 --- a/src/inferencekit/ops/intake.py +++ b/src/inferencekit/ops/intake.py @@ -349,8 +349,20 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: if "input_rate" not in io: warnings.append("io_modes.input_rate missing: the replay pace of streaming input is unknown") - d.setdefault("buckets", list(BUCKETS)) + wl = d.setdefault("workload", {}) + tts = wl.get("tts") + if tts is not None and modality != "tts": + errors.append("workload.tts applies only to modality tts") + text_buckets = (tts or {}).get("text_length_buckets", []) + names = [b["name"] for b in text_buckets] + if len(names) != len(set(names)): + errors.append("workload.tts.text_length_buckets names must be unique") + if any(b["max_chars"] < b["min_chars"] for b in text_buckets): + errors.append("workload.tts.text_length_buckets max_chars must be >= min_chars") + d.setdefault("buckets", names or list(BUCKETS)) d["buckets"] = [b for b in BUCKETS if b in d["buckets"]] + if text_buckets and set(d["buckets"]) - set(names): + errors.append("buckets must be defined in workload.tts.text_length_buckets when character bounds are supplied") if modality in ("image", "video"): # modality workload set, not text length buckets if targets.pop("buckets", None) is not None: warnings.append(f"goals.targets.buckets do not apply to modality {modality}: a target holds over the " @@ -362,10 +374,19 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: elif any(k != "buckets" for k in targets): targets["buckets"] = list(d["buckets"]) - wl = d.setdefault("workload", {}) - wl.setdefault("max_tokens", 256) + if modality == "llm": + wl.setdefault("max_tokens", 256) + wl.setdefault("prompt_set", "dev") + if wl["max_tokens"] < 2: + errors.append("workload.max_tokens must be >= 2 for modality llm") + elif "max_tokens" in wl: + warnings.append(f"workload.max_tokens does not apply to modality {modality}; retained for compatibility, " + "but unused by its timing client") wl.setdefault("rounds", 5) - wl.setdefault("prompt_set", "dev") + client = wl.get("client") + expected_client = {"llm": "bench_serve", "tts": "bench_audio", "image": "bench_image", "video": "bench_clip"} + if client is not None and client not in ("external", expected_client.get(modality)): + errors.append(f"workload.client {client!r} does not support modality {modality}") if needs_load: wl.setdefault("concurrency", list(DEFAULT_CONCURRENCY)) if "concurrency" in wl: @@ -532,9 +553,13 @@ def normalize(data: dict) -> tuple[dict, list[str], list[str]]: cands.setdefault("include", []) cands.setdefault("exclude", []) cands.setdefault("max_engines", 5) - if modality in ("tts", "stt"): - warnings.append(f"modality {modality}: bench serve times OpenAI-compatible text endpoints; audio " - "latencies need a modality client, and claims without one are UNPROVEN") + if modality in ("tts", "stt") and client is None: + hint = "configure workload.client: bench_audio for speech synthesis, or external" if modality == "tts" else "configure workload.client: external for speech recognition" + warnings.append(f"modality {modality}: no audio timing client is configured; {hint}. " + "Audio latency claims without a matching client are UNPROVEN") + if modality == "tts" and client == "bench_audio": + warnings.append("bench audio supplies client-timed speech screening; quality and fresh-process " + "interleaved comparison remain UNPROVEN without their separate evidence") if modality == "image": warnings.append("modality image: bench serve token metrics and bench clip video metrics do not prove image " "latency or quality; a client-timed image adapter and modality quality protocol are required; " @@ -761,7 +786,9 @@ def derive(intake: dict) -> dict: if goals["lead"] == "capacity" or "n_star" in targets: modes.append("load") if intake["modality"] == "image": - modes = ["image"] # requires an external image adapter; no built-in command implied + modes = ["image"] + if intake["modality"] == "tts" and intake["workload"].get("client") == "bench_audio": + modes = ["audio"] if intake["modality"] == "video": modes = ["clip"] # bench rounds --mode clip: one clip at a time questions = [] @@ -970,12 +997,15 @@ def display_path(path: str | os.PathLike) -> str: return s -def redact(intake: dict) -> dict: +def redact(intake: dict, *, home: str | None = None) -> dict: """A copy of an intake that is safe to keep in the run directory (which is synced, reviewed and published). * every ``hardware.access`` field except ``kind`` becomes ```` (host, user, port, key_path); * the host anywhere else (e.g. inside ``quality.reference.base_url``) and ``@`` become ````; * the home directory at the start of a path becomes ``~`` (it names the local user). + + ``home`` defaults to this account's home; an empty string disables home substitution. + Resume validation can supply the original prefix when checking a copied run. """ d = json.loads(json.dumps(intake, default=str)) access = (d.get("hardware") or {}).get("access") if isinstance(d.get("hardware"), dict) else None @@ -985,7 +1015,7 @@ def redact(intake: dict) -> dict: user = access.get("user") if isinstance(access.get("user"), str) and access.get("user") != REDACTED else None host_re = re.compile(r"(? dict: kv_cache_dtype_allowed: [checkpoint_default] # adding fp8 (when the checkpoint does not ship it) is lossy # tokenizer: # default: model.id -modality: llm # llm | tts | stt | video +modality: llm # llm | image | tts | stt | video hardware: gpu: RTX 5090 @@ -1145,10 +1175,14 @@ def start_run(run_dir: Path, src: Path, raw: bytes, result: dict) -> dict: buckets: [very_short, short, medium, long] # 1-5 words / ~20-50 tok / ~300 tok / 2-4k tok workload: - max_tokens: 256 # generated per request, identical for every engine (ignore_eos) + max_tokens: 256 # LLM only: generated per request, identical for every engine (ignore_eos) rounds: 5 # interleaved rounds across engines (>= 3) # concurrency: [1, 2, 4, 8, 16, 32, 64] # load-mode grid (capacity) # chat_template_kwargs: {enable_thinking: false} # binding pin: the manifest and every command must use it + # client: bench_audio # TTS: omit max_tokens; bench_audio | external + # tts: {voice: default, sample_rate: 24000, lead_metric: first_audio_ms} + # TTS text_length_buckets: [{name: short, min_chars: 1, max_chars: 80}] inside tts + # io_modes.output selects streaming or complete utterance budget: gpu_hours: 6 # GPU-hours = host hours x count @@ -1185,7 +1219,7 @@ def start_run(run_dir: Path, src: Path, raw: bytes, result: dict) -> dict: # wheel: -py3-none-any.whl> # wheel_sha256: # candidates: -# include: [] # engines to add (e.g. a custom engine with an OpenAI API) +# include: [] # engines to add (e.g. a custom engine with an OpenAI-compatible endpoint) # exclude: [] # max_engines: 5 """ diff --git a/src/inferencekit/ops/quality_audio.py b/src/inferencekit/ops/quality_audio.py new file mode 100644 index 0000000..da85b49 --- /dev/null +++ b/src/inferencekit/ops/quality_audio.py @@ -0,0 +1,301 @@ +"""Registered speech quality against the exact PCM saved by bench audio.""" + +from __future__ import annotations + +import datetime as dt +import json +from pathlib import Path + +from inferencekit import evidence as ev, stats +from inferencekit.canonical import canonical_json +from inferencekit.ops.bench_audio import load_fixtures +from inferencekit.quality import audio as qa, metrics as qm +from inferencekit.registry import Arg, OpError, operation +from inferencekit.serve.round_evidence import round_coverage + + +def _write(root, name, value): + path = root / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(value, indent=2, allow_nan=False) + "\n") + return path + + +@operation("quality_audio_policy", ("quality", "audio-policy"), + "Register a speech quality policy and fixture identity before benchmark capture; no model execution.", + [Arg("policy", "string", "JSON acceptance, margins, scorer pins and bootstrap plan.", required=True), + Arg("fixtures", "string", "The speech fixture JSONL to be benchmarked.", required=True), + Arg("run_dir", "string", "Also copy the registered policy evidence to this run.")]) +def quality_audio_policy(call, policy, fixtures, run_dir): + doc, _ = qa.read_json(policy) + protocol = qa.validate_policy(doc) + fixture_rows, fixture_sha = load_fixtures(fixtures) + for fixture in fixture_rows: + if not qa.words(fixture.text, protocol["normalization"]): + raise OpError("audio policy fixture text has no words after normalization") + data = {"format": qa.POLICY_FORMAT, "source": "quality audio-policy", "policy": protocol, + "policy_sha256": qa.sha(canonical_json(protocol).encode()), "fixture_sha256": fixture_sha, + "fixtures": [f.identity() for f in fixture_rows]} + store = ev.Store() + with store.staging_dir() as staging: + relative = "quality/audio/policies/" + qa.sha(canonical_json(data).encode()) + ".json" + path = _write(staging, relative, data) + if run_dir: + _write(Path(run_dir), relative, data) + record = ev.build(call, kind="quality_audio", claim_class="derived", summary="Speech quality policy registered; measurements NOT RUN", + artifacts=[ev.artifact(path, staging, "audio_quality_policy")], + data={**data, "verdict": "NOT RUN"}) + return store.save(record, staging) + + +def _timestamp(value): + try: + return dt.datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp() + except (ValueError, AttributeError, TypeError): + return None + + +def _source_compatibility(a, b, policy): + """Engine/precision changes cannot change the spoken task, conditioning or seeds.""" + left, right = a["data"], b["data"] + if left["fixture_set"] != right["fixture_set"]: + raise OpError("audio quality requires the same complete fixture set") + if {r["index"]: r["seed"] for r in left["rounds"]} != {r["index"]: r["seed"] for r in right["rounds"]}: + raise OpError("audio quality requires identical round indices and seeds") + for key in ("id", "revision"): + if a["identity"]["model"].get(key) != b["identity"]["model"].get(key): + raise OpError("audio quality requires the same model and checkpoint revision") + if not policy["allow_numerics_change"] and any(a["identity"]["model"].get(k) != b["identity"]["model"].get(k) + for k in ("dtype", "quant")): + raise OpError("audio numerical changes must be explicitly allowed in the registered policy") + def settings(data): + config = data["audio_config"] + ignored = {"engine_version"} | ({"dtype", "quant"} if policy["allow_numerics_change"] else set()) + return {k: config[k] for k in ("sample_rate", "channels", "seed_base")} | { + "settings": {k: v for k, v in config["settings"].items() if k not in ignored}} + if settings(left) != settings(right): + raise OpError("audio quality requires matching generation and audio settings") + + +def _metric_check(name, policy, observations, keys): + """Pair on fixtures, retaining all rounds within each resampled fixture.""" + spec = policy["metrics"][name] + asr_metric = name in qa.ASR_METRICS + denominator = "words" if name == "wer" else "characters" + retry_name = f"{name}_retry" + totals = {"baseline": {}, "candidate": {}} + missing, outliers, retry_missing = [], [], [] + retry_totals = {"baseline": {}, "candidate": {}} + for fid, round_index in keys: + pair = {arm: observations.get((arm, fid, round_index), {}) for arm in totals} + if any(name not in v for v in pair.values()): + missing.append(f"{fid}/r{round_index}") + continue + if name == "listening": + left, right = pair["baseline"][name], pair["candidate"][name] + if set(left) != set(right) or len(left) < spec["min_raters"]: + missing.append(f"{fid}/r{round_index}: paired listener panel") + continue + outlier = asr_metric and any(v[name][name] > spec[f"outlier_{name}"] for v in pair.values()) + if outlier: + outliers.append(f"{fid}/r{round_index}") + if any(retry_name not in v for v in pair.values()): + retry_missing.append(f"{fid}/r{round_index}") + for arm, values in pair.items(): + value = values[name] + amount, count = (value["errors"], value[denominator]) if asr_metric else ( + sum(value.values()) / len(value), 1) if name == "listening" else (value, 1) + old = totals[arm].get(fid, (0, 0)) + totals[arm][fid] = (old[0] + amount, old[1] + count) + if asr_metric: + retry = values.get(retry_name) if outlier else value + if retry is not None: + old = retry_totals[arm].get(fid, (0, 0)) + retry_totals[arm][fid] = (old[0] + retry["errors"], old[1] + retry[denominator]) + ids = sorted(set(totals["baseline"]) & set(totals["candidate"])) + def score(values): + baseline = [values["baseline"][i] for i in ids] + candidate = [values["candidate"][i] for i in ids] + result = qm.paired_delta_ci(candidate, baseline, policy["resamples"], policy["seed"]) + lo, hi = result["ci95"] + if not asr_metric: + lo, hi = (-hi if hi is not None else None), (-lo if lo is not None else None) + verdict, reason = qm.margin_verdict(lo, hi, spec["margin"]) + return {**result, "baseline": qm.pooled(baseline), "candidate": qm.pooled(candidate), + "worse_ci95": [lo, hi], "verdict": verdict, "reason": reason} + result = score(totals) + result.update(margin=spec["margin"], missing=missing, outliers=outliers, + method="paired fixture-cluster bootstrap; all rounds retained per fixture") + if name == "cer": + result["normalization"] = {"name": policy["normalization"], "unit": "unicode_code_point", "whitespace": "removed"} + if missing or len(ids) < policy["min_fixtures"]: + # A sufficiently complete observed regression remains a regression. Otherwise no omitted cases may PASS. + if result["verdict"] != "FAIL": + result.update(verdict="UNPROVEN", reason="missing observations or fewer than the registered fixture count") + if outliers: + result["primary_verdict"] = result["verdict"] + if retry_missing: + result.update(verdict="UNPROVEN", reason="ASR outliers require both arms' registered corroborating decode", + retry_missing=retry_missing) + else: + retry = score(retry_totals) + result["retry"] = retry + if retry["verdict"] != result["verdict"]: + result.update(verdict="UNPROVEN", reason="ASR primary and corroborating decode disagree; review both transcripts") + return result + + +@operation("quality_audio", ("quality", "audio"), + "Score saved speech PCM with exact parity and registered, hash-bound ASR/MOS/listening observations.", + [Arg("baseline", "string", "Baseline bench audio evidence id.", required=True), + Arg("candidate", "string", "Candidate bench audio evidence id.", required=True), + Arg("fixtures", "string", "Original speech fixture JSONL.", required=True), + Arg("protocol", "string", "Evidence id from quality audio-policy.", required=True), + Arg("scores", "string", "Optional external observation JSON; never executes or downloads models."), + Arg("run_dir", "string", "Also copy evidence and quality/audio_score.json to this run.")]) +def quality_audio(call, baseline, candidate, fixtures, protocol, scores, run_dir): + store = ev.Store() + registered, policy_root = qa.load_record(store, protocol, "quality_audio") + if registered["data"].get("format") != qa.POLICY_FORMAT: + raise OpError("protocol must be a quality audio-policy record") + qa.artifact_bytes(policy_root, qa.one_artifact(registered, "audio_quality_policy")) + policy = qa.validate_policy(registered["data"]["policy"]) + fixture_rows, fixture_sha = load_fixtures(fixtures) + if fixture_sha != registered["data"]["fixture_sha256"]: + raise OpError("audio fixture set differs from the registered policy") + a, arows = qa.load_benchmark(store, baseline) + b, brows = qa.load_benchmark(store, candidate) + _source_compatibility(a, b, policy) + if a["data"]["fixture_set"]["sha256"] != fixture_sha: + raise OpError("audio source fixtures differ from the policy and supplied texts") + if set(arows) != set(brows): + raise OpError("audio source request pairs are incomplete") + keys = sorted(arows) + registered_at = _timestamp(registered["provenance"]["ended"]) + starts = [r.get("t_start") for rec in (a, b) for r in rec["data"]["rounds"]] + preregistered = bool(registered_at is not None and starts and all( + qa.finite(t) and registered_at <= t for t in starts)) + observations, imported = qa.load_scores(scores, registered["evidence_id"], a["evidence_id"], b["evidence_id"], + policy, {"baseline": arows, "candidate": brows}, + {f.id: f.text for f in fixture_rows}) + missing, pairs = [], [] + source_coverage = {arm: round_coverage(rec) for arm, rec in (("baseline", a), ("candidate", b))} + for arm, rec in (("baseline", a), ("candidate", b)): + coverage = source_coverage[arm] + if (coverage["complete"] is False or coverage["gaps"] or + rec["data"].get("scheduler_final") and coverage["complete"] is not True): + missing.append(f"complete {arm} source round coverage") + round_verdict = "UNPROVEN" if missing else "PASS" + structural = "PASS" + for key in keys: + left, right = arows[key], brows[key] + pair = {"id": key[0], "round": key[1], "seed": left["seed"], + "baseline_pcm_sha256": left["pcm_sha256"], "candidate_pcm_sha256": right["pcm_sha256"]} + if not left["ok"] or not right["ok"]: + structural = "FAIL" + pair["error"] = "source request failed structural checks" + elif left["pcm"] is None or right["pcm"] is None: + missing.append(f"saved PCM {key[0]}/r{key[1]}") + else: + pair.update(qa.parity(left["pcm"], right["pcm"], left["channels"])) + pairs.append(pair) + if any(rec["data"].get("structural_verdict") == "FAIL" for rec in (a, b)): + structural = "FAIL" + exact = "FAIL" if structural == "FAIL" or any(p.get("bit_exact") is False for p in pairs) else ( + "UNPROVEN" if missing else "PASS") + checks = {"structural": {"verdict": structural}, + "parity": {"verdict": exact, "required": policy["acceptance"] == "exact"}, + "round_coverage": {"verdict": round_verdict, "arms": source_coverage}, + "registration": {"verdict": "PASS" if preregistered else "UNPROVEN"}} + if not preregistered: + missing.append("policy registered before every measured round") + rounds = len(a["data"]["rounds"]) + enough = len(fixture_rows) >= policy["min_fixtures"] and rounds >= policy["min_rounds"] + if not enough: + missing.append("registered minimum fixture/round count") + compared = [p for p in pairs if "bit_exact" in p] + snrs = [p["snr_db"] for p in compared if p["snr_db"] is not None] + maxima = [p["maxabs"] for p in compared if p["maxabs"] is not None] + metric_rows = [ev.metric("bit_exact_pct", 100 * sum(p["bit_exact"] for p in compared) / len(compared) + if compared else None, "%", "fraction", n=len(compared), subject=b["data"]["label"]), + ev.metric("same_sample_count_pct", 100 * sum(p["same_sample_count"] for p in compared) / len(compared) + if compared else None, "%", "fraction", n=len(compared)), + ev.metric("waveform_snr_db", min(snrs) if snrs else None, "dB", "min_finite", n=len(snrs)), + ev.metric("waveform_maxabs", max(maxima) if maxima else None, "full_scale", "max", n=len(maxima))] + for name in policy["metrics"]: + result = _metric_check(name, policy, observations, keys) + checks[name] = result + if result["missing"]: + missing.append(f"complete {name} observations") + if result["verdict"] == "UNPROVEN": + missing.append(f"conclusive {name} evidence") + metric_rows += [ev.metric(f"audio_{name}", result[arm], "fraction" if name in qa.ASR_METRICS else "score", "pooled", + n=result["n_prompts"], subject=arm) for arm in ("baseline", "candidate")] + metric_rows.append(ev.metric(f"audio_{name}_delta", result["delta"], "fraction" if name in qa.ASR_METRICS else "score", + "candidate_minus_baseline", n=result["n_prompts"], ci95=result["ci95"], + subject=b["data"]["label"])) + required = [structural, checks["registration"]["verdict"], "PASS" if enough and not missing else "UNPROVEN"] + if policy["acceptance"] == "exact": + required.append(exact) + required.extend(checks[name]["verdict"] for name in policy["metrics"]) + verdict = stats.combine_claims(required) + scope = {"exact": "bit-exact preservation of the named baseline outputs", + "task_noninferiority": "registered speech task/proxy noninferiority (predicted MOS, not human listening)", + "perceptual_and_task_noninferiority": "registered speech task and blinded-listening noninferiority"}[policy["acceptance"]] + limitations = ["Quality is relative to the named baseline; baseline intelligibility and suitability are not established.", + "Imported scorer outputs and listening blinding are producer attestations; hashes bind inputs and configuration, not methodological honesty.", + "SNR and maxabs use aligned raw PCM only; unequal lengths are not truncated or padded. Numerical parity is diagnostic outside exact acceptance.", + "Paired CIs resample fixtures, retaining all rounds; listening inference is conditional on the supplied listener panel.", + "Quality does not prove fresh-process lifecycle, timing, memory fit or sustained reliability."] + if policy["acceptance"] == "task_noninferiority": + limitations.append("ASR error rates, predicted MOS and speaker scores are proxies; human perceptual quality remains UNPROVEN.") + data = {"format": qa.RESULT_FORMAT, "source": "quality audio", "label": b["data"]["label"], + "baseline": a["evidence_id"], "candidate": b["evidence_id"], "protocol": registered["evidence_id"], + "policy": policy, "fixture_sha256": fixture_sha, "preregistered": preregistered, + "acceptance": policy["acceptance"], "scope": scope, "checks": checks, "pairs": pairs, + "verdict": verdict, "reason": scope, "scorer_provenance": policy["scorers"]} + with store.staging_dir() as staging: + artifact_prefix = f"quality/audio/{b['data']['label']}/" + qa.sha(canonical_json(data).encode()) + relative = f"{artifact_prefix}/audio_score.json" + path = _write(staging, relative, data) + artifacts = [ev.artifact(path, staging, "audio_quality_result")] + if imported is not None: + imported_relative = f"{artifact_prefix}/imported_scores.json" + imported_path = staging / imported_relative + imported_path.write_bytes(imported) + artifacts.append(ev.artifact(imported_path, staging, "audio_quality_observations")) + if run_dir: + _write(Path(run_dir), relative, data) + if imported is not None: + (Path(run_dir) / imported_relative).write_bytes(imported) + _write(Path(run_dir), "quality/audio_score.json", data) + record = ev.build(call, kind="quality_audio", claim_class="derived", summary=f"Speech quality {verdict}: {scope}", + identity_=b["identity"], metrics=metric_rows, artifacts=artifacts, missing=missing, + limitations=limitations, data=data) + return store.save(record, staging) + + +def report_verdict(record): + """Require the retained observations, not just a claimed verdict in data.""" + if record.get("data", {}).get("format") != qa.RESULT_FORMAT: + return "NOT RUN", "speech quality protocol; measurements not run" + try: + check = ev.verify(record) + if not check["id_ok"] or not check["schema_ok"]: + raise OpError("invalid quality evidence") + root = ev.Store().run_dir(record["evidence_id"]) + if not root.is_dir(): + copied_run = record["data"].get("request", {}).get("args", {}).get("run_dir") + if copied_run: + root = Path(copied_run).expanduser() + for item in record["artifacts"]: + qa.artifact_bytes(root, item) + qa.one_artifact(record, "audio_quality_result") + if record["data"]["policy"]["metrics"]: + qa.one_artifact(record, "audio_quality_observations") + except (OpError, KeyError, TypeError): + return "UNPROVEN", "speech quality evidence or retained observations are missing/changed" + verdict = record["data"].get("verdict") + if verdict == "PASS" and (not record["coverage"]["complete"] or not record["data"].get("preregistered")): + return "UNPROVEN", "speech quality coverage or preregistration is incomplete" + return (verdict if verdict in ("PASS", "FAIL", "UNPROVEN", "NOT RUN") else "UNPROVEN"), record["data"]["scope"] diff --git a/src/inferencekit/ops/report.py b/src/inferencekit/ops/report.py index e26c163..a551c5e 100644 --- a/src/inferencekit/ops/report.py +++ b/src/inferencekit/ops/report.py @@ -63,7 +63,9 @@ def claims_of(rec: dict, expect: str, integrity: dict) -> list[dict]: if data.get("source_kind") in ("image_benchmark", "audio_benchmark"): for claim, proof in data.get("required_proofs", {}).items(): out.append({"source": eid, "claim": claim, "verdict": proof, - "reason": "A timing comparison does not supply this proof; cite external modality evidence."}) + "reason": "Bound scheduler records confirm fresh launches, readiness and process-group cleanup." + if claim == "fresh-process lifecycle" and proof == "PASS" else + "A timing comparison does not supply this proof; cite external modality evidence."}) if data.get("source_kind") == "audio_benchmark": for arm, timing in data.get("arm_timing_verdicts", {}).items(): out.append({"source": eid, "claim": f"{arm} speech latency and streaming SLOs", @@ -82,6 +84,13 @@ def claims_of(rec: dict, expect: str, integrity: dict) -> list[dict]: out.append({"source": eid, "claim": "media quality and usability", "verdict": "UNPROVEN", "reason": "Timing and structural checks do not establish image quality, " "speech intelligibility, speaker fidelity or repeated-run reliability; cite task-specific quality evidence."}) + elif kind == "quality_audio": + from inferencekit.ops.quality_audio import report_verdict + + verdict, reason = report_verdict(rec) + if data.get("source") == "quality audio": + out.append({"source": eid, "claim": "speech quality preservation against baseline", + "verdict": verdict, "reason": reason}) elif kind == "correctness": out.append({"source": eid, "claim": "greedy outputs agree", "verdict": data.get("verdict", "UNPROVEN"), "reason": data.get("reason", "")}) @@ -126,16 +135,45 @@ def workload_text(w: dict) -> str: def environment_rows(recs: list[dict]) -> list[dict]: + from inferencekit.serve.round_evidence import memory_of + seen, rows = set(), [] - for r in recs: - i = r.get("identity", {}) + identities = [] + for record in recs: + if record.get("kind") == "comparison" and record.get("data", {}).get("baseline_identity"): + memories = record["data"].get("memory") or {} + identities.append((record["data"]["baseline_identity"], memories.get("baseline", {}))) + identities.append((record.get("identity", {}), memories.get("candidate", {}))) + else: + identities.append((record.get("identity", {}), memory_of(record))) + for i, memory in identities: g, e, m, w = i.get("gpu", {}), i.get("engine", {}), i.get("model", {}), i.get("workload", {}) + cpu = i.get("cpu") or {} gpu = g.get("name") or "?" - row = {"gpu": f"{g['count']}x {gpu}" if g.get("count") else gpu, - "driver / CUDA": f"{g.get('driver') or '?'} / {g.get('cuda') or '?'}", "clocks": g.get("clocks") or "?", + gpu_label = f"{g['count']}x {gpu}" if g.get("count") else gpu + tuning = (e.get("flags") or {}).get("server_env") + controls = (tuning or {}).get("controls") or {} + parts = [f"{k}={v if v is not None else 'unknown'}" for k, v in sorted(controls.items())] + for lib in (tuning or {}).get("ld_preload") or []: + parts.append("LD_PRELOAD=" + (lib.get("name") or "redacted") + " (" + + (lib.get("sha256") or lib.get("status") or "unverified") + ")") + if (tuning or {}).get("varied_between_rounds"): + parts.append("changed between rounds") + memory_parts = [] + for name, measurement in memory.items(): + qualifiers = [measurement["method"]] if measurement.get("method") else [] + if measurement.get("complete") is not True: + qualifiers.append("partial" if measurement.get("complete") is False else "coverage unknown") + memory_parts.append(f"{name}: {_fmt(measurement.get('value'))} MiB" + + (" (" + "; ".join(qualifiers) + ")" if qualifiers else "")) + row = {"gpu": gpu_label, "hardware": "CPU: " + (cpu.get("model") or "unknown") if cpu else gpu_label, + "driver / CUDA": "n/a" if cpu else f"{g.get('driver') or '?'} / {g.get('cuda') or '?'}", + "clocks": g.get("clocks") or "?", "engine": " ".join(str(x) for x in (e.get("name"), e.get("version")) if x) or "-", "model": " ".join(str(x) for x in (m.get("id"), m.get("dtype"), m.get("quant")) if x) or "-", - "workload": workload_text(w)} + "workload": workload_text(w), "peak memory": "; ".join(memory_parts) or "unknown", + "server tuning (requested)": "; ".join(parts) if parts else ( + "none recorded" if tuning is not None else "unknown")} key = json.dumps(row, sort_keys=True) if key not in seen: seen.add(key) @@ -239,6 +277,27 @@ def _subject(rec: dict, m: dict) -> str: def build_report(recs: list[dict], expect: str, title: str) -> dict: integrity = {r["evidence_id"]: ev.verify(r) for r in recs} claims = [c for r in recs for c in claims_of(r, expect, integrity[r["evidence_id"]])] + # Quality attaches only to the exact compared pair. A newer PASS cannot erase an + # earlier FAIL, and proxy acceptance is labelled rather than called listening. + audio_quality = [r for r in recs if r.get("kind") == "quality_audio" and + r.get("data", {}).get("source") == "quality audio"] + for rec in recs: + data = rec.get("data", {}) + if rec.get("kind") != "comparison" or data.get("source_kind") != "audio_benchmark": + continue + matched = [q for q in audio_quality if all(q["data"].get(k) == data.get(k) + for k in ("baseline", "candidate"))] + if not matched or not integrity[rec["evidence_id"]]["id_ok"]: + continue + from inferencekit.ops.quality_audio import report_verdict + + verdicts = [report_verdict(q) for q in matched] + for claim in claims: + if claim["source"] == rec["evidence_id"] and claim["claim"] == "media quality": + prior = claim["verdict"] + claim["verdict"] = stats.combine_claims([v[0] for v in verdicts] + ([prior] if prior == "FAIL" else [])) + claim["claim"] = "speech quality preservation against baseline" + claim["reason"] = "; ".join(f"{q['evidence_id']}: {v[1]}" for q, v in zip(matched, verdicts)) verdict = stats.combine_claims(c["verdict"] for c in claims) results = [] for r in recs: @@ -269,7 +328,11 @@ def to_markdown(rep: dict) -> str: else: lines.append("No comparison, correctness or benchmark evidence: nothing was claimed (NOT RUN).") lines += ["", "## Environment card", ""] - cols = ["gpu", "driver / CUDA", "clocks", "engine", "model", "workload"] + cols = ["hardware", "driver / CUDA", "clocks", "engine", "model", "workload"] + if any(row.get("peak memory") != "unknown" for row in rep["environment"]): + cols.append("peak memory") + if any(row.get("server tuning (requested)") not in ("unknown", "none recorded") for row in rep["environment"]): + cols.append("server tuning (requested)") lines += ["| " + " | ".join(cols) + " |", "|" + "---|" * len(cols)] lines += ["| " + " | ".join(_cell(row[c]) for c in cols) + " |" for row in rep["environment"]] lines += ["", "## Results", "", "| evidence | metric | subject | value | unit | stat | n | 95% CI |", diff --git a/src/inferencekit/ops/resume.py b/src/inferencekit/ops/resume.py new file mode 100644 index 0000000..ffd5a44 --- /dev/null +++ b/src/inferencekit/ops/resume.py @@ -0,0 +1,347 @@ +"""Read-only run resumption: pinned intake, state, budget and cited evidence in one call.""" + +from __future__ import annotations + +import datetime as dt +import json +import math +from pathlib import Path +import re + +from inferencekit import evidence as ev +from inferencekit.canonical import canonical_json +from inferencekit.ops import intake +from inferencekit.ops.store import page +from inferencekit.registry import Arg, OpError, operation + + +_BUDGET_BASIS = ("Calculated from mutable checkpoint run_started, meter and stage timing; history is not " + "authenticated. Remaining allowance and the calculated action are not verified authorization.") + + +def _object(path: Path) -> dict: + try: + value = json.loads(path.read_text()) + except (OSError, ValueError) as exc: + raise OpError(f"cannot read {path}: {exc}") from None + if not isinstance(value, dict): + raise OpError(f"{path}: expected a JSON object") + return value + + +def _citations(value, location=""): + """Find every evidence list, including candidates and best_history; do not infer proof from it.""" + if isinstance(value, dict): + for key, child in value.items(): + at = f"{location}.{key}" if location else key + if key == "evidence": + if not isinstance(child, list): + raise OpError(f"state.json: {at} must be a list of full evidence ids") + for i, ref in enumerate(child): + if not isinstance(ref, str) or not re.fullmatch(r"ev_[0-9a-f]{64}", ref): + raise OpError(f"state.json: {at}.{i} must be a full evidence id") + yield ref, at + else: + yield from _citations(child, at) + elif isinstance(value, list): + for i, child in enumerate(value): + yield from _citations(child, f"{location}.{i}") + + +def _changed(actual, expected, location): + """Report edited policy paths, never their potentially sensitive values.""" + if isinstance(actual, dict) and isinstance(expected, dict): + for key in sorted(actual.keys() | expected.keys()): + if key not in actual or key not in expected: + yield f"{location}.{key}" + else: + yield from _changed(actual[key], expected[key], f"{location}.{key}") + elif canonical_json(actual) != canonical_json(expected): + yield location + + +def _saved_intake_policy(validated, saved): + """Reproduce saved home redaction without weakening the exact policy comparison.""" + raw = intake.redact(validated, home="") + prefixes = set() + + def find_prefixes(original, stored): + if isinstance(original, dict) and isinstance(stored, dict): + for key in original.keys() & stored.keys(): + find_prefixes(original[key], stored[key]) + elif isinstance(original, list) and isinstance(stored, list): + for left, right in zip(original, stored): + find_prefixes(left, right) + elif (isinstance(original, str) and isinstance(stored, str) and stored.startswith("~/") + and original.startswith("/") and original.endswith(stored[1:])): + prefix = original[:-len(stored[1:])] + if prefix: + prefixes.add(prefix) + + find_prefixes(raw, saved) + if len(prefixes) == 1: + return intake.redact(validated, home=prefixes.pop()) + # No prior home substitution, or conflicting prefixes: the full comparison below + # accepts unchanged absolute paths and rejects incompatible saved redactions. + return raw + + +def _check_policy(state, normalized, validated, original): + # run_id is assigned once when absent from the original intake; validating again must not invent a new + # timestamp or rename the run when its directory moves. All explicitly pinned intake fields stay fixed. + if not original.get("run_id"): + if not isinstance(state.get("run_id"), str) or not state["run_id"]: + raise OpError("state.json: run_id must be nonempty text") + validated["intake"]["run_id"] = state["run_id"] + validated["derived"] = intake.derive(validated["intake"]) + expected = {"intake": _saved_intake_policy(validated["intake"], normalized.get("intake")), + "derived": validated["derived"], + "ok": True, "placeholders": []} + # Older normalizers added LLM defaults to media intakes. Accept only those exact + # implied values; explicit source fields and every other saved policy pin stay binding. + saved_intake = normalized.get("intake") + saved_workload = saved_intake.get("workload") if isinstance(saved_intake, dict) else None + if validated["intake"]["modality"] in ("tts", "stt", "image", "video") and isinstance(saved_workload, dict): + source_workload = original.get("workload") or {} + for key, default in (("max_tokens", 256), ("prompt_set", "dev")): + if (key not in source_workload and key not in expected["intake"]["workload"] + and saved_workload.get(key) == default): + expected["intake"]["workload"][key] = default + changed = [] + for key, value in expected.items(): + changed.extend(_changed(normalized.get(key), value, f"intake.normalized.json.{key}")) + initial = intake.initial_state(validated, validated["intake_sha256"]) + budget = state.get("budget") + if not isinstance(budget, dict): + raise OpError("state.json: budget must be an object") + # Meter entries, stage progress and reassigned stage time boxes are mutable run history. Spending caps + # and policy overrides are not: changing those requires a new pinned intake, not a state-only edit. + for key, value in initial["budget"].items(): + if key not in ("meter", "run_started"): + changed.extend(_changed(budget.get(key), value, f"state.json.budget.{key}")) + for key in ("run_id", "workload_pin"): + changed.extend(_changed(state.get(key), initial.get(key), f"state.json.{key}")) + model = state.get("model") + if not isinstance(model, dict): + raise OpError("state.json: model must be an object") + for key in ("id", "revision"): + changed.extend(_changed(model.get(key), initial["model"].get(key), f"state.json.model.{key}")) + revision = initial["model"].get("revision") + resolved = model.get("revision_resolved") + # A branch/tag may legitimately acquire a resolved commit during stage 1. An immutable commit pin + # cannot resolve to another commit, even though revision_resolved is otherwise dynamic checkpoint data. + if (isinstance(revision, str) and re.fullmatch(r"[0-9a-fA-F]{40}", revision) and resolved is not None + and (not isinstance(resolved, str) or resolved.lower() != revision.lower())): + changed.append("state.json.model.revision_resolved") + if changed: + raise OpError("pinned run policy changed: " + ", ".join(changed) + + ". Restore the saved policy or start a new run from a changed intake; nothing was written.") + + +def _evidence(root: Path, ref: str, locations: list[str]) -> dict: + # Both normal Store output and the run-directory copy made by benchmark/quality operations are valid + # sources. Do not consult the global store: a resume must work from this run's synced artifacts alone. + path = root / ref / "evidence.json" + if not path.exists(): + path = root / f"{ref}.json" + row = {"evidence_id": ref, "cited_by": locations} + if not path.is_file(): + return {**row, "status": "missing", "reason": "no record in this run's evidence directory"} + row["path"] = str(path) + try: + rec = _object(path) + integrity = ev.verify(rec) + except (OpError, TypeError, ValueError, OverflowError) as exc: + return {**row, "status": "invalid", "reason": str(exc)} + integrity["cited_id_ok"] = rec.get("evidence_id") == ref + row["integrity"] = integrity + if not all(integrity[key] for key in ("id_ok", "schema_ok", "cited_id_ok")): + return {**row, "status": "invalid", "reason": "evidence id, content hash or schema check failed"} + return {**row, "status": "available", "kind": rec["kind"], "summary": rec["summary"], + "coverage": rec["coverage"], "limitations": rec["limitations"]} + + +def _audit_evidence(state, root): + refs: dict[str, list[str]] = {} + for ref, location in _citations(state): + locations = refs.setdefault(ref, []) + if location not in locations: + locations.append(location) + rows = [_evidence(root / "evidence", ref, locations) for ref, locations in refs.items()] + rows.sort(key=lambda row: row["status"] == "available") + counts = {status: sum(row["status"] == status for row in rows) + for status in ("available", "missing", "invalid")} + return rows, {"checked": len(rows), **counts, "order": "missing/invalid first"} + + +def _blocked(reason, root, source, state, normalized, validated, query, offset, limit): + """Keep exit 1 / MCP is_error while avoiding a second manual diagnostic workflow. + + The original pin failed: no saved policy, budget or next action is authorized by this response. Evidence + content checks are independent of that failed policy check. Return counts and bounded issue pages only. + """ + def brief(value, length=240): + if isinstance(value, str): + return value if len(value) <= length else {"text": value[:length], "truncated": True} + if isinstance(value, float) and not math.isfinite(value): + return "see source (non-finite value)" + return value if value is None or isinstance(value, (bool, int, float)) else "see source (non-scalar)" + + def snapshot(data): + if not isinstance(data, dict): + return {"error": "malformed saved intake; see source"} + budget = data.get("budget") if isinstance(data.get("budget"), dict) else {} + return {"budget": {key: brief(budget[key]) for key in + ("approved", "wall_clock_h", "gpu_hours", "usd", "spend_cap", "deadline") if key in budget}, + "fidelity": brief(data.get("fidelity"))} + + result = { + "status": "blocked", "reason": brief(reason, 2000), + "action": "Do not resume. Restore the pinned files or start a new run for policy changes.", + "trust": {"saved_policy_and_budget": "UNPROVEN: not validated against the original intake", + "authorization": "not evaluated; saved approval fields are diagnostic only", + "evidence": "content hash and schema checks only; not performance or quality proof"}, + "sources": {"original_intake": str(source), "state": str(root / "state.json"), + "normalized_intake": str(root / "intake.normalized.json")}, + "intake_hashes": {"supplied": (validated or {}).get("intake_sha256"), + "state": brief(state.get("intake_sha256")), + "normalized": brief(normalized.get("intake_sha256"))}, + "saved_state": {"current_stage": state["current_stage"], + "status": state["stages"][state["current_stage"]]["status"], + "next_action": brief(state["next_action"], 1600)}, + "saved_policy_snapshot_unvalidated": snapshot(normalized.get("intake")), + } + if validated is not None: + result["supplied_intake_snapshot_not_pinned"] = snapshot(validated["intake"]) + try: + timestamp = intake._parse_time(query["now"]) if query.get("now") else dt.datetime.now(dt.timezone.utc) + saved_budget = intake.budget_status(state, timestamp) + saved_budget["basis"] = _BUDGET_BASIS + # A numeric-looking NaN is not a meaningful saved budget, even for diagnostics. + json.dumps(saved_budget, allow_nan=False) + result["saved_budget_calculation_unvalidated"] = saved_budget + except (AttributeError, KeyError, TypeError, ValueError, OverflowError) as exc: + result["saved_budget_calculation_unvalidated"] = {"error": brief(str(exc))} + try: + rows, counts = _audit_evidence(state, root) + problems = [{key: row[key] for key in ("evidence_id", "status", "reason") if key in row} + for row in rows if row["status"] != "available"] + for row in problems: + row["reason"] = brief(row.get("reason"), 500) + def notes(field): + values = sorted({str(note) for row in rows for note in + ((row.get("coverage") or {}).get("missing", []) if field == "missing" + else row.get("limitations", []))}) + return {"total": len(values), "items": [brief(value) for value in values[:10]], + "truncated": len(values) > 10} + result["evidence_summary"] = {**counts, + "incomplete_records": sum((row.get("coverage") or {}).get("complete") is False for row in rows), + "coverage_missing": notes("missing"), "limitations": notes("limitations")} + result["evidence_issue_page"] = page(problems, offset, min(limit, 10)) + if result["evidence_issue_page"]["next_offset"] is not None: + result["next_call"] = {"tool": "intake_resume", "arguments": { + **query, "offset": result["evidence_issue_page"]["next_offset"], "limit": min(limit, 10)}} + except (OpError, TypeError, ValueError) as exc: + result["evidence_summary"] = {"error": brief(str(exc), 800), "complete_check": False} + result = ev.redact(result) + text = json.dumps(result, ensure_ascii=False, separators=(",", ":"), allow_nan=False) + if len(text) > 24000: # Unusually long paths or malformed metadata: retain the hard rejection and pointers. + result = {key: result[key] for key in ("status", "reason", "action", "trust", "sources")} + result["diagnostics_omitted"] = "Inspect the named files; diagnostic output exceeded 24000 characters." + text = json.dumps(result, ensure_ascii=False, separators=(",", ":"), allow_nan=False) + raise OpError(text) + + +@operation("intake_resume", ("intake", "resume"), + "Resume context in one read-only call: validate the pinned original intake, show state and budget, " + "and check every state-cited evidence record's hash and schema. Checkpoint budget history is mutable; " + "does not establish authorization or proof verdicts.", + [Arg("run_dir", "string", "Run directory containing state.json and intake.normalized.json.", + required=True, positional=True), + Arg("file", "string", "Original intake path; overrides state.intake_source if its relative path " + "no longer resolves. The pinned hash must still match."), + Arg("now", "string", "Evaluate the budget at this ISO-8601 UTC time instead of the current time."), + Arg("offset", "integer", "First evidence item to return; missing/invalid records sort first.", default=0), + Arg("limit", "integer", "Evidence page size. Every cited record is checked on every call.", default=20)]) +def intake_resume(call, run_dir, file, now, offset, limit): + if offset < 0 or limit < 1: + raise OpError("intake resume: offset must be nonnegative and limit must be positive") + query = {"run_dir": run_dir, **({"file": file} if file is not None else {}), + **({"now": now} if now is not None else {})} + root = Path(run_dir).expanduser().resolve() + state = _object(root / "state.json") + normalized = _object(root / "intake.normalized.json") + if state.get("state_version") != 1: + raise OpError(f"{root / 'state.json'}: unsupported state_version {state.get('state_version')!r}") + stages = state.get("stages") + statuses = {"pending", "in_progress", "done", "skipped", "over_box"} + if (not isinstance(stages, dict) or not stages or + any(not isinstance(s, dict) or s.get("status") not in statuses for s in stages.values())): + raise OpError(f"{root / 'state.json'}: malformed stages or stage status") + if (not isinstance(state.get("current_stage"), str) or state["current_stage"] not in stages + or not isinstance(state.get("next_action"), str)): + raise OpError(f"{root / 'state.json'}: current_stage must name a stage and next_action must be text") + source = file or state.get("intake_source") + if not isinstance(source, str) or not source: + raise OpError("state.json has no intake_source; pass --file ") + source = Path(source).expanduser() + if not source.is_file(): + raise OpError(f"original intake not found: {source}; pass --file (the pinned hash " + "must still match)") + # The existing validator rejects redacted copies and invalid policy. With no run_dir it writes nothing. + try: + validated = intake.intake_validate(call, str(source), run_dir=None, allow_placeholders=False) + except OpError as exc: + _blocked(str(exc), root, source, state, normalized, None, query, offset, limit) + if validated["placeholders"]: + raise OpError("cannot resume with placeholder values: " + ", ".join(validated["placeholders"])) + sha = validated["intake_sha256"] + changed = [name for name, data in (("state.json", state), ("intake.normalized.json", normalized)) + if data.get("intake_sha256") != sha] + if changed: + _blocked(f"the intake changed since this run started: {', '.join(changed)} does not match the " + "original intake. Targets are pre-registered; start a new run id. Nothing was written.", + root, source, state, normalized, validated, query, offset, limit) + original, _ = intake.load_file(source) + try: + _check_policy(state, normalized, validated, original) + except OpError as exc: + _blocked(str(exc), root, source, state, normalized, validated, query, offset, limit) + except (TypeError, ValueError) as exc: + _blocked(f"malformed pinned run policy ({exc})", root, source, state, normalized, + validated, query, offset, limit) + try: + timestamp = intake._parse_time(now) if now else dt.datetime.now(dt.timezone.utc) + budget = intake.budget_status(state, timestamp) + budget["basis"] = _BUDGET_BASIS + json.dumps(budget, allow_nan=False) + except (AttributeError, KeyError, TypeError, ValueError, OverflowError) as exc: + raise OpError(f"{root / 'state.json'}: malformed budget/stage fields ({exc})") from None + evidence, counts = _audit_evidence(state, root) + issues = counts["missing"] + counts["invalid"] + pinned = intake.redact(validated["intake"]) + result = { + "run_dir": str(root), "run_id": state.get("run_id"), + "status": "needs_attention" if issues or budget["verdict"] != "ok" else "context_ready", + "scope": "resume context and evidence integrity only; not a benchmark or quality proof", + "trust": {"checkpoint_history_authenticated": False, "remaining_authorization_verified": False, + "authorization": "not evaluated; context_ready and budget.action do not authorize work"}, + "current_stage": state["current_stage"], "next_action": ev.redact(state["next_action"]), + "stages": ev.redact({key: {field: value for field, value in stage.items() if field in ("status", "notes")} + for key, stage in stages.items()}), + "best": ev.redact(state.get("best")), "budget": budget, + "policy": ev.redact({"source": "validated intake only; session authorization is not evaluated", + "budget_approved": pinned["budget"]["approved"], "fidelity": pinned["fidelity"], + "quality": pinned["quality"], + "architecture_changes": validated["derived"]["architecture_changes"]}), + "intake": {"source": str(source.resolve()), "sha256": sha, "warnings": ev.redact(validated["warnings"])}, + "evidence_root": str(root / "evidence"), + "evidence_summary": counts, + **page(ev.redact(evidence), offset, limit), + "pagination": {"tool": "intake_resume", "args": query}, + "state_path": str(root / "state.json"), + } + if result["next_offset"] is not None: + result["next_call"] = {"tool": "intake_resume", + "arguments": {**query, "offset": result["next_offset"], "limit": limit}} + return result diff --git a/src/inferencekit/ops/store.py b/src/inferencekit/ops/store.py index 5e39e64..174e4be 100644 --- a/src/inferencekit/ops/store.py +++ b/src/inferencekit/ops/store.py @@ -2,14 +2,36 @@ from __future__ import annotations +from collections import Counter + from inferencekit import evidence as ev from inferencekit.registry import Arg, OpError, operation +_FAILURES = {"FAIL", "failed", "error", "regressed", "port_busy", "launch_failed", "startup_timeout", + "exited_early", "warmup_failed", "serve_failed", "serve_no_metrics", "stop_failed", "timeout"} +_UNPROVEN = {"UNPROVEN", "NOT RUN", "inconclusive"} +_SUCCESSES = {"PASS", "ok", "success", "improved", "within_noise"} +_COUNTED_VALUES = _FAILURES | _UNPROVEN | _SUCCESSES | {"RESOLVED", "SELF_REPORTED"} + + +def join_path(path: str, key: str | int) -> str: + """Keep familiar dotted paths where unambiguous; use JSON Pointer for dotted or empty keys.""" + key = str(key) + if not path: + return "/" + key.replace("~", "~0").replace("/", "~1") if not key or "." in key else key + if path.startswith("/"): + return path + "/" + key.replace("~", "~0").replace("/", "~1") + if not key or "." in key: + return "/" + "/".join(p.replace("~", "~0").replace("/", "~1") for p in [*path.split("."), key]) + return f"{path}.{key}" + def get_path(record: dict, path: str): - """Follow a dotted path such as 'data.top_kernels' or 'metrics.0'.""" + """Follow a dotted path, or a JSON Pointer such as '/data/verdicts/engine.v1'.""" node = record - for part in [p for p in path.split(".") if p]: + parts = [p.replace("~1", "/").replace("~0", "~") for p in path[1:].split("/")] if path.startswith("/") \ + else [p for p in path.split(".") if p] + for part in parts: if isinstance(node, dict) and part in node: node = node[part] elif isinstance(node, list) and part.lstrip("-").isdigit() and -len(node) <= int(part) < len(node): @@ -26,6 +48,83 @@ def page(items: list, offset: int, limit: int) -> dict: return {"total": len(items), "offset": offset, "items": chunk, "next_offset": nxt if nxt < len(items) else None} +def _findings(node, path="data", verdict_map=False): + """Stored decisions only: never infer a verdict from coverage or the absence of failures.""" + if isinstance(node, dict): + for key, value in node.items(): + here = join_path(path, key) + is_verdict = key == "verdict" or key.endswith("_verdict") or verdict_map + if (is_verdict and isinstance(value, str)) or \ + (key == "status" and isinstance(value, str)) or \ + (key == "valid" and isinstance(value, bool)): + finding = {"path": here, "field": "verdict" if is_verdict else key, "value": value} + # Keep decision qualifiers exact; detailed metrics and raw samples stay at the source path. + finding.update({k: node[k] for k in ("reason", "applicable", "error", "detail", "violations", "warnings", + "limitations") if k in node}) + yield finding + yield from _findings(value, here, verdict_map or key == "verdicts" or key.endswith("_verdicts") + or key == "required_proofs") + elif isinstance(node, list): + for i, value in enumerate(node): + yield from _findings(value, join_path(path, i)) + + +def _finding_order(finding): + value = finding["value"] + # Surface every explicit failure before successful checks, regardless of its position in a large list. + if value is False or value in _FAILURES: + return 0 + if value in _UNPROVEN: + return 1 + if value is True or value in _SUCCESSES: + return 3 + return 2 + + +def summary_view(rec: dict, integrity: dict, ref: str, offset: int, limit: int) -> dict: + """An opt-in inspection view, not a new evidence record or a new overall verdict. + + All decisions are scanned before pagination. Counts describe stored fields (including duplicate + aggregate and individual verdicts), not independent checks or an inferred pass/fail decision. + """ + data = rec.get("data") if isinstance(rec.get("data"), dict) else {} + findings = sorted(_findings(data), key=_finding_order) + # Free-form status/verdict strings must not become unbounded JSON keys. Unknown values remain exact + # in paged findings, and their combined count is explicit instead of being silently dropped. + counts = Counter() + for finding in findings: + value = finding["value"] + counted = value if isinstance(value, bool) or value in _COUNTED_VALUES else "" + counts[f"{finding['field']}:{counted}"] += 1 + index = {} + for key, value in data.items(): + if isinstance(value, (dict, list)): + index[key] = {"path": join_path("data", key), "type": "object" if isinstance(value, dict) else "array", + "count": len(value)} + else: + index[key] = value + result = {"view": "summary", "source": ref, "integrity": integrity, + **{k: rec.get(k) for k in ("evidence_id", "kind", "claim_class", "summary", "identity", "provenance", + "metrics", "coverage", "limitations")}, + "artifact_count": len(rec["artifacts"]) if isinstance(rec.get("artifacts"), list) else None, + "data_index": index, + "finding_counts": dict(sorted(counts.items())), **page(findings, offset, limit), + "note": "Stored decision fields, failures first; counts include aggregate and individual fields. " + "Unrecognized values are counted under and retained in paged items. " + "No overall verdict is inferred. Inspect source paths for details or view='full' for the record."} + accounting = data.get("request_accounting") + if rec.get("kind") == "serve_benchmark" and isinstance(accounting, dict): + keys = ("timed_requests", "successful_requests", "failed_requests", "selected_token_counts", + "unknown_token_counts", "potentially_estimated_counts", "selected_output_tokens_total", "selected_output_tokens_min", "selected_output_tokens_max", + "successful_count_mismatches") + result["request_accounting"] = {k: accounting[k] for k in keys + if k in accounting and (accounting[k] is None or type(accounting[k]) is int)} + if result["next_offset"] is not None: + result["more"] = (f"inspect_evidence(id={ref!r}, view='summary', offset={result['next_offset']}, " + f"limit={max(1, limit)})") + return result + + @operation("list_evidence", ("evidence", "list"), "List stored evidence records, newest first.", [Arg("kind", "string", "Only this kind.", choices=ev.KINDS), Arg("offset", "integer", "Skip this many records.", default=0), @@ -42,16 +141,23 @@ def list_evidence(call, kind, offset, limit): return {"root": str(ev.Store().root), **page(rows, offset, limit)} -@operation("inspect_evidence", ("evidence", "show"), "Show one evidence record (or a page of a list inside it, " - "e.g. path=data.top_kernels), with an integrity check.", +@operation("inspect_evidence", ("evidence", "show"), "Inspect evidence with an integrity check. Use view=summary " + "for metrics, limitations and stored decisions; path=data.top_kernels for detail; default is full.", [Arg("id", "string", "Evidence id, a unique prefix (6+ hex chars), or a path to evidence.json.", required=True, positional=True), - Arg("path", "string", "Dotted path inside the record, e.g. data.kernels or metrics."), - Arg("offset", "integer", "For list values: first item.", default=0), - Arg("limit", "integer", "For list values: page size.", default=50)]) -def inspect_evidence(call, id, path, offset, limit): # noqa: A002 + Arg("path", "string", "Dotted path (data.kernels), or JSON Pointer (/data/verdicts/engine.v1) for " + "keys containing dots. Use paths returned by summary."), + Arg("offset", "integer", "For list values or summary findings: first item.", default=0), + Arg("limit", "integer", "For list values or summary findings: page size.", default=50), + Arg("view", "string", "Full record (default), or summary with all decision counts and paged findings. " + "Summary cannot be combined with path.", default="full", choices=("full", "summary"))]) +def inspect_evidence(call, id, path, offset, limit, view="full"): # noqa: A002 + if view == "summary" and path: + raise OpError("view=summary cannot be combined with path; omit view to inspect the exact source path") rec = ev.Store().load(id) - integrity = ev.verify(rec) + integrity = ev.verify(rec, include_error_count=True) if view == "summary" else ev.verify(rec) + if view == "summary": + return summary_view(rec, integrity, id, offset, limit) if not path: return {"integrity": integrity, "record": rec} value = get_path(rec, path) diff --git a/src/inferencekit/quality/audio.py b/src/inferencekit/quality/audio.py new file mode 100644 index 0000000..0559df4 --- /dev/null +++ b/src/inferencekit/quality/audio.py @@ -0,0 +1,345 @@ +"""Speech artifact validation and CPU-only metrics. No model downloads or execution. + +Imported observations are attestations by their producer, bound to the exact PCM, +fixture text, scorer configuration and registered protocol. Hashes do not certify +the producer's methodology. Numerical parity and MOS predictors are not listening. +""" + +from __future__ import annotations + +import hashlib +import json +import math +import re +import struct +import unicodedata +from pathlib import Path + +from inferencekit import evidence as ev +from inferencekit.registry import OpError + +POLICY_FORMAT = "inferencekit.quality.audio.policy.v1" +SCORES_FORMAT = "inferencekit.quality.audio.scores.v1" +RESULT_FORMAT = "inferencekit.quality.audio.v1" +NORMALIZATIONS = ("whitespace_v1", "nfkc_casefold_punctuation_v1") +ASR_METRICS = ("wer", "cer") + + +def sha(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def read_json(path): + try: + raw = Path(path).read_bytes() + value = json.loads(raw) + json.dumps(value, allow_nan=False) + return value, raw + except (OSError, ValueError, TypeError): + raise OpError("cannot read finite audio quality JSON") from None + + +def finite(value): + return type(value) in (int, float) and math.isfinite(value) + + +def validate_policy(doc): + if not isinstance(doc, dict) or set(doc) - { + "acceptance", "min_fixtures", "min_rounds", "resamples", "seed", "allow_numerics_change", + "metrics", "scorers", "normalization"}: + raise OpError("invalid audio policy fields") + out = {"min_fixtures": 2, "min_rounds": 1, "resamples": 2000, "seed": 0, + "allow_numerics_change": False, "metrics": {}, "scorers": {}, + "normalization": "nfkc_casefold_punctuation_v1", **doc} + if out.get("acceptance") not in ("exact", "task_noninferiority", "perceptual_and_task_noninferiority"): + raise OpError("audio acceptance must be exact, task_noninferiority or perceptual_and_task_noninferiority") + for key in ("min_fixtures", "min_rounds", "resamples", "seed"): + if type(out[key]) is not int or out[key] < (0 if key == "seed" else 1): + raise OpError(f"audio policy {key} must be a positive integer (seed may be zero)") + if out["min_fixtures"] < 2 or out["resamples"] < 100: + raise OpError("audio policy needs at least two fixtures and 100 bootstrap resamples") + if type(out["allow_numerics_change"]) is not bool or out["normalization"] not in NORMALIZATIONS: + raise OpError("invalid audio numerical-change or normalization policy") + metrics, scorers = out["metrics"], out["scorers"] + if not isinstance(metrics, dict) or set(metrics) - {"wer", "cer", "naturalness", "speaker", "listening"}: + raise OpError("unknown audio quality metric") + if not isinstance(scorers, dict) or set(scorers) - {"asr", "asr_retry", "naturalness", "speaker", "listening"}: + raise OpError("unknown audio scorer") + if out["acceptance"] != "exact" and (not set(ASR_METRICS).intersection(metrics) or "naturalness" not in metrics): + raise OpError("noninferiority requires registered WER or CER (or both) and naturalness margins") + if out["acceptance"] == "perceptual_and_task_noninferiority" and "listening" not in metrics: + raise OpError("perceptual acceptance requires a registered listening metric") + for name, spec in metrics.items(): + allowed = {"margin", f"outlier_{name}"} if name in ASR_METRICS else {"margin", "min_raters"} if name == "listening" else {"margin"} + if not isinstance(spec, dict) or set(spec) - allowed or not finite(spec.get("margin")) or spec["margin"] < 0: + raise OpError(f"{name} requires a finite nonnegative margin") + if name in ASR_METRICS and (not finite(spec.get(f"outlier_{name}")) or spec[f"outlier_{name}"] <= 0): + raise OpError(f"{name.upper()} requires a positive outlier_{name} threshold, registered before decoding") + if name == "listening" and (type(spec.get("min_raters")) is not int or spec["min_raters"] < 2): + raise OpError("listening requires min_raters >= 2") + scorer = "asr" if name in ASR_METRICS else name + if scorer not in scorers: + raise OpError(f"{name} requires a pinned {scorer} scorer") + for name, spec in scorers.items(): + required = {"name", "version", "config", "protocol_sha256"} if name == "listening" else { + "name", "version", "model", "revision", "config"} + if not isinstance(spec, dict) or not required.issubset(spec) or set(spec) - (required | {"scale"}): + raise OpError(f"{name} scorer requires name/version, model/revision (or listening protocol hash), and config") + if any(not isinstance(spec[key], str) or not spec[key].strip() for key in required - {"config"}): + raise OpError(f"{name} scorer pins must be nonempty strings") + if not isinstance(spec["config"], dict) or ev.redact(spec) != spec: + raise OpError("audio scorer config must be an object without secrets") + if name == "asr_retry" and (spec["config"].get("vad_filter") is not True or + spec["config"].get("condition_on_previous_text") is not False): + raise OpError("ASR corroboration requires VAD on and condition_on_previous_text false") + if name in ("naturalness", "speaker", "listening"): + scale = spec.get("scale") + if not isinstance(scale, list) or len(scale) != 2 or not all(finite(x) for x in scale) or scale[0] >= scale[1]: + raise OpError(f"{name} scorer needs a finite increasing scale") + if name == "listening" and (spec["config"].get("blinded") is not True or + not re.fullmatch(r"[a-f0-9]{64}", spec["protocol_sha256"])): + raise OpError("listening requires a blinded protocol and its SHA-256") + return out + + +def artifact_bytes(root, item): + """Read only a hash-checked, contained artifact; never trust a path from JSON.""" + try: + path = (Path(root) / item["path"]).resolve() + if not path.is_relative_to(Path(root).resolve()): + raise ValueError + raw = path.read_bytes() + if len(raw) != item["bytes"] or sha(raw) != item["sha256"]: + raise ValueError + return raw + except (OSError, KeyError, TypeError, ValueError): + raise OpError("audio evidence artifact missing, changed or outside its record") from None + + +def one_artifact(record, role): + items = [a for a in record.get("artifacts", []) if a.get("role") == role] + if len(items) != 1: + raise OpError(f"audio evidence requires exactly one {role} artifact") + return items[0] + + +def load_record(store, ref, kind): + record = store.load(ref) + verification = ev.verify(record) + if record.get("kind") != kind or not verification["id_ok"] or not verification["schema_ok"]: + raise OpError(f"audio quality requires intact {kind} evidence") + p = Path(ref) + root = p.parent if p.is_file() else p if p.is_dir() else store.run_dir(record["evidence_id"]) + if root.name == "evidence" and p.is_file(): + root = root.parent # copied run-dir evidence/.json uses run-relative artifacts + return record, root + + +def load_benchmark(store, ref): + record, root = load_record(store, ref, "audio_benchmark") + data = record["data"] + if data.get("source") != "bench audio": + raise OpError("audio quality requires bench audio source evidence") + try: + rows = [json.loads(line) for line in artifact_bytes(root, one_artifact(record, "requests")).splitlines() + if line.strip()] + index = json.loads(artifact_bytes(root, one_artifact(record, "outputs_index"))) + if not isinstance(index, list): + raise ValueError + fixtures = {f["id"]: f for f in data["fixture_set"]["fixtures"]} + rounds = {r["index"]: r for r in data["rounds"]} + if len(fixtures) != len(data["fixture_set"]["fixtures"]) or len(rounds) != len(data["rounds"]): + raise ValueError + keyed = {} + for row in rows: + key = (row["id"], row["round"]) + if key in keyed or row["id"] not in fixtures or row["round"] not in rounds: + raise ValueError + if row["seed"] != rounds[row["round"]]["seed"] or any(row[k] != v for k, v in fixtures[row["id"]].items()): + raise ValueError + keyed[key] = {**row, "pcm": None} + if set(keyed) != {(f, r) for f in fixtures for r in rounds} or len(rows) != data["total_requests"]: + raise ValueError + artifacts = {a["path"]: a for a in record["artifacts"] if a["role"] == "audio_pcm"} + if len(artifacts) != sum(a["role"] == "audio_pcm" for a in record["artifacts"]): + raise ValueError + used = set() + label = data["label"] + for item in index: + match = re.fullmatch(r"outputs/" + re.escape(label) + r"/r(\d+)/([^/]+)\.pcm", item["path"]) + if not match: + raise ValueError + key = (match[2], int(match[1])) + if key not in keyed or key in used: + raise ValueError + row = keyed[key] + if ("id" in item and item["id"] != key[0]) or ("round" in item and item["round"] != key[1]): + raise ValueError + if not row["ok"] or item["encoding"] != "pcm_s16le": + raise ValueError + if any(item[k] != row[k] for k in ("sample_rate", "channels")): + raise ValueError + if item["sha256"] != row["pcm_sha256"] or item["bytes"] != row["pcm_bytes"]: + raise ValueError + raw = artifact_bytes(root, artifacts[item["path"]]) + if sha(raw) != item["sha256"] or len(raw) != item["bytes"] or not raw or len(raw) != 2 * row["sample_count"] * row["channels"]: + raise ValueError + row["pcm"] = raw + used.add(key) + if set(artifacts) != {item["path"] for item in index}: + raise ValueError + except (ValueError, KeyError, TypeError): + raise OpError("audio benchmark request/output identities are inconsistent") from None + return record, keyed + + +def parity(baseline: bytes, candidate: bytes, channels: int): + result = {"bit_exact": baseline == candidate, + "same_sample_count": len(baseline) == len(candidate), + "baseline_frames": len(baseline) // (2 * channels), + "candidate_frames": len(candidate) // (2 * channels), + "snr_db": None, "maxabs": None, "snr_status": "length_mismatch"} + if len(baseline) != len(candidate): + return result # no alignment, truncation, resampling or padding can hide missing audio + energy = error = maximum = 0 + for (a,), (b,) in zip(struct.iter_unpack(" frames: + raise OpError("naturalness windows must cover complete PCM once, in order") + total += (stop - start) * scaled(window["score"], scorer) + end = stop + if end != frames: + raise OpError("naturalness windows omit PCM frames") + return total / frames + + +def load_scores(path, protocol_id, baseline_id, candidate_id, policy, sources, texts): + if not path: + return {}, None + doc, raw = read_json(path) + if not isinstance(doc, dict) or set(doc) != {"format", "protocol", "baseline", "candidate", "scorers", "rows"}: + raise OpError("invalid audio score import envelope") + if (doc["format"], doc["protocol"], doc["baseline"], doc["candidate"]) != ( + SCORES_FORMAT, protocol_id, baseline_id, candidate_id) or doc["scorers"] != policy["scorers"]: + raise OpError("audio scores do not match the protocol, source evidence or pinned scorers") + if not isinstance(doc["rows"], list): + raise OpError("audio scores rows must be a list") + result = {} + for item in doc["rows"]: + if not isinstance(item, dict) or set(item) - {"arm", "id", "round", "seed", "pcm_sha256", "text_sha256", + "asr", "naturalness", "speaker", "listening", "error"}: + raise OpError("invalid imported audio observation") + try: + key = (item["arm"], item["id"], item["round"]) + row = sources[item["arm"]][(item["id"], item["round"])] + if key in result or not row["ok"] or row["pcm"] is None or any( + item[k] != row[k] for k in ("seed", "pcm_sha256", "text_sha256")): + raise ValueError + except (KeyError, TypeError, ValueError): + raise OpError("imported audio observation has a duplicate, unknown or mismatched PCM/text identity") from None + values = {} + if "asr" in item: + asr = item["asr"] + if not set(ASR_METRICS).intersection(policy["metrics"]) or not isinstance(asr, dict) or set(asr) - {"text", "retry_text"}: + raise OpError("invalid ASR observation") + for source, suffix in (("text", ""), ("retry_text", "_retry")): + if source == "text" or source in asr: + if not isinstance(asr.get(source), str) or (source == "retry_text" and "asr_retry" not in policy["scorers"]): + raise OpError("ASR text and a registered retry scorer are required") + for name, score in (("wer", word_error), ("cer", character_error)): + if name in policy["metrics"]: + values[name + suffix] = score(texts[item["id"]], asr[source], policy["normalization"]) + if "naturalness" in item: + if "naturalness" not in policy["metrics"]: + raise OpError("unregistered naturalness metric") + values["naturalness"] = window_score(item["naturalness"], row["sample_count"], policy["scorers"]["naturalness"]) + if "speaker" in item: + speaker = item["speaker"] + if "speaker" not in policy["metrics"] or not isinstance(speaker, dict) or set(speaker) != {"score", "reference_sha256"}: + raise OpError("invalid speaker observation") + if not row["reference_sha256"] or speaker["reference_sha256"] != row["reference_sha256"]: + raise OpError("speaker score must bind the fixture's voice reference") + values["speaker"] = scaled(speaker["score"], policy["scorers"]["speaker"]) + if "listening" in item: + listen = item["listening"] + if "listening" not in policy["metrics"] or not isinstance(listen, dict) or set(listen) != {"ratings"} or not isinstance(listen["ratings"], list): + raise OpError("invalid listening observations") + ratings = {} + for rating in listen["ratings"]: + if not isinstance(rating, dict) or set(rating) != {"rater", "score"} or not isinstance(rating["rater"], str) or not rating["rater"] or rating["rater"] in ratings: + raise OpError("listening needs distinct anonymous rater IDs") + ratings[rating["rater"]] = scaled(rating["score"], policy["scorers"]["listening"]) + values["listening"] = ratings + result[key] = values + return result, raw diff --git a/src/inferencekit/quality/audio_scorers.py b/src/inferencekit/quality/audio_scorers.py new file mode 100644 index 0000000..13a15a4 --- /dev/null +++ b/src/inferencekit/quality/audio_scorers.py @@ -0,0 +1,514 @@ +"""Local speech scorers, isolated from inferencekit's lightweight environment. + +This file is also the worker entry point and intentionally imports only stdlib +at module scope. Optional model libraries belong in the configured Python env. +Backend APIs: SYSTRAN/faster-whisper; sarulab-speech/UTMOSv2 (create_model, +predict); speechbrain.inference.classifiers.EncoderClassifier (encode_batch). +""" + +from __future__ import annotations + +import contextlib +import hashlib +import importlib.metadata +import importlib.util +import json +import math +import os +from pathlib import Path +import random +import subprocess +import sys +import tempfile + +CONFIG_FORMAT = "inferencekit.quality.audio.scorers.v1" +DESCRIPTORS_FORMAT = "inferencekit.quality.audio.scorer-descriptors.v1" +WORKER_FORMAT = "inferencekit.quality.audio.scorer-worker.v1" +BACKENDS = { + "faster-whisper": ("faster_whisper", ("faster-whisper", "ctranslate2", "av", "tokenizers", "onnxruntime", "numpy", "scipy", "huggingface-hub")), + "utmosv2": ("utmosv2", ("utmosv2", "torch", "torchaudio", "torchvision", "timm", "transformers", "librosa", "numpy", "scipy")), + "speechbrain-ecapa": ("speechbrain", ("speechbrain", "torch", "torchaudio", "hyperpyyaml", "numpy", "scipy", "soundfile")), +} + + +def _json(value): + return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False) + + +def _digest(value): + return hashlib.sha256(_json(value).encode()).hexdigest() + + +def _file_hash(path): + h = hashlib.sha256() + size = 0 + with Path(path).open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + h.update(block) + size += len(block) + return h.hexdigest(), size + + +def tree_identity(path): + """Bind bytes and relative names, including HF snapshot symlink targets. + + Cache metadata, Git metadata and Python bytecode are not model/source files. + Absolute paths never enter the public descriptor. + """ + root = Path(path) + if not root.exists() or not (root.is_file() or root.is_dir()): + raise ValueError("local_checkpoint_missing") + paths = [root] if root.is_file() else sorted(root.rglob("*")) + entries = [] + for item in paths: + relative = item.relative_to(root).as_posix() if root.is_dir() else "checkpoint" + if any(p in {".git", ".cache", "__pycache__"} for p in Path(relative).parts) or item.suffix == ".pyc": + continue + if item.is_symlink() and item.is_dir(): + raise ValueError("checkpoint_directory_symlinks_unsupported") + if item.is_file(): + digest, size = _file_hash(item) + entries.append({"file": relative, "sha256": digest, "bytes": size}) + if not entries: + raise ValueError("local_checkpoint_empty") + return {"sha256": _digest(entries), "files": len(entries), "bytes": sum(x["bytes"] for x in entries)} + + +def _positive(value, name, maximum=None, zero=False): + if type(value) is not int or value < (0 if zero else 1) or maximum is not None and value > maximum: + raise ValueError("invalid_" + name) + + +def normalize_config(doc, base=None): + """Strict config; resolve local paths before crossing the worker boundary.""" + if not isinstance(doc, dict) or set(doc) - {"format", "python", "threads", "seed", "batch_size", "timeout_s", "scorers"}: + raise ValueError("invalid_scorer_config_fields") + if doc.get("format") != CONFIG_FORMAT: + raise ValueError("invalid_scorer_config_format") + out = {"python": sys.executable, "threads": 1, "seed": 0, "batch_size": 4, "timeout_s": 3600, **doc} + for key in ("threads", "seed", "batch_size"): + _positive(out[key], key, maximum=2**32 - 1 if key == "seed" else None, zero=key == "seed") + if type(out["timeout_s"]) not in (int, float) or not math.isfinite(out["timeout_s"]) or out["timeout_s"] <= 0: + raise ValueError("invalid_timeout_s") + if not isinstance(out.get("scorers"), dict) or not out["scorers"] or set(out["scorers"]) - {"asr", "asr_retry", "naturalness", "speaker"}: + raise ValueError("invalid_scorer_roles") + if "asr_retry" in out["scorers"] and "asr" not in out["scorers"]: + raise ValueError("asr_retry_requires_asr") + base = Path(base or Path.cwd()) + + def local(value): + if not isinstance(value, str) or not value or "://" in value: + raise ValueError("local_path_required") + return str((base / Path(value).expanduser()).absolute()) + + out["python"] = local(out["python"]) + if not Path(out["python"]).is_file(): + raise ValueError("scorer_python_missing") + normalized = {} + for role, spec in out["scorers"].items(): + if not isinstance(spec, dict) or set(spec) - {"backend", "model", "settings", "assets"}: + raise ValueError("invalid_scorer_fields") + backend = {"asr": "faster-whisper", "asr_retry": "faster-whisper", "naturalness": "utmosv2", "speaker": "speechbrain-ecapa"}[role] + if spec.get("backend") != backend or not isinstance(spec.get("settings", {}), dict): + raise ValueError("invalid_scorer_backend_or_settings") + if role.startswith("asr"): + defaults = {"language": "en", "beam_size": 5, "compute_type": "int8", "vad_filter": role == "asr_retry", "condition_on_previous_text": False} + elif role == "naturalness": + defaults = {"model_name": "fusion_stage3", "window_seconds": 3, "fold": 0, "checkpoint_seed": 42, + "predict_dataset": "sarulab", "num_repetitions": 1} + else: + defaults = {} + if set(spec.get("settings", {})) - set(defaults): + raise ValueError("unknown_scorer_setting") + settings = {**defaults, **spec.get("settings", {})} + if role.startswith("asr"): + _positive(settings["beam_size"], "beam_size") + if (not isinstance(settings["language"], str) or not settings["language"].isalpha() + or settings["compute_type"] not in {"int8", "float32", "int8_float32"} + or type(settings["vad_filter"]) is not bool or settings["condition_on_previous_text"] is not False + or role == "asr_retry" and settings["vad_filter"] is not True): + raise ValueError("invalid_asr_settings") + if role == "naturalness": + if settings["model_name"] != "fusion_stage3": + raise ValueError("utmos_requires_fusion_stage3") + seconds = settings["window_seconds"] + if type(seconds) not in (int, float) or not math.isfinite(seconds) or not 0 < seconds <= 3: + raise ValueError("utmos_window_seconds_must_be_positive_at_most_three") + _positive(settings["fold"], "fold", 4, zero=True) + _positive(settings["checkpoint_seed"], "checkpoint_seed", zero=True) + _positive(settings["num_repetitions"], "num_repetitions") + if settings["predict_dataset"] not in {"sarulab", "bvcc", "somos", "blizzard2008", "blizzard2009", "blizzard2010-EH1", "blizzard2010-EH2", "blizzard2010-ES1", "blizzard2010-ES3", "blizzard2011"}: + raise ValueError("invalid_utmos_predict_dataset") + assets = spec.get("assets", {}) + expected = {"facebook/wav2vec2-base", "tf_efficientnetv2_s.in21k_ft_in1k"} if role == "naturalness" else set() + if not isinstance(assets, dict) or set(assets) != expected: + raise ValueError("utmos_requires_local_backbone_assets" if expected else "unexpected_scorer_assets") + normalized[role] = {"backend": backend, "model": local(spec.get("model")), "settings": settings, + "assets": {k: local(v) for k, v in assets.items()}} + out["scorers"] = normalized + return out + + +def describe_local(config): + """Executed by the scorer Python, never by the core Python environment.""" + descriptors = {} + package_cache, checkpoint_cache = {}, {} + for role, spec in config["scorers"].items(): + backend = spec["backend"] + if backend not in package_cache: + module, packages = BACKENDS[backend] + versions = {p: importlib.metadata.version(p) for p in packages} + found = importlib.util.find_spec(module) + if found is None or not found.origin: + raise ValueError("scorer_backend_missing") + source = tree_identity(Path(found.origin).parent) + package_cache[backend] = (versions, source) + versions, source = package_cache[backend] + identities = {} + for key, path in {"checkpoint": spec["model"], **spec["assets"]}.items(): + if path not in checkpoint_cache: + checkpoint_cache[path] = tree_identity(path) + identities[key] = checkpoint_cache[path] + settings = {**spec["settings"], "device": "cpu", "threads": config["threads"], "seed": config["seed"], + "batch_size": config["batch_size"], "backend_source": source, "packages": versions, + "checkpoints": identities, "adapter_sha256": _file_hash(__file__)[0], + "python_version": sys.version.split()[0], "mono": "arithmetic_channel_mean", + "resample": "scipy.signal.resample_poly_kaiser5", "sample_rate": 16000} + if role.startswith("asr"): + settings.update(temperature=0, best_of=1, num_workers=1, local_files_only=True, batch_size=1) + elif role == "naturalness": + settings.update(remove_silent_section=False, num_workers=0, short_input="model_native_tiling", + windows="contiguous_original_pcm_frames_including_tail", spectrogram_views="model_native_seeded", + identical_windows="score_once", batch_order="sorted_pcm_hash_equal_length") + else: + settings.update(normalize_embeddings=False, similarity="cosine", batching="equal_length", + identical_pcm="embed_once", batch_order="sorted_pcm_hash", cosine_roundoff_atol=8 * sys.float_info.epsilon) + descriptor = {"name": backend, "version": versions[BACKENDS[backend][1][0]], + "model": "fusion_stage3" if role == "naturalness" else "ecapa-tdnn" if role == "speaker" else "local-whisper", + "revision": "sha256:" + _digest(identities), "config": settings} + if role in {"naturalness", "speaker"}: + descriptor["scale"] = [1, 5] if role == "naturalness" else [-1, 1] + descriptors[role] = descriptor + return {"format": DESCRIPTORS_FORMAT, "scorers": descriptors} + + +def _worker_environment(config, home): + # Never inherit API keys, user import paths, download caches or proxy settings. + return {"PATH": os.defpath, "HOME": str(home), "TMPDIR": str(home), "LANG": "C.UTF-8", + "PYTHONHASHSEED": str(config["seed"]), "PYTHONDONTWRITEBYTECODE": "1", + "HF_HUB_OFFLINE": "1", "TRANSFORMERS_OFFLINE": "1", "HF_HUB_DISABLE_TELEMETRY": "1", + "HF_HUB_DISABLE_IMPLICIT_TOKEN": "1", "HF_HOME": str(home / "hf"), "TORCH_HOME": str(home / "torch"), + "CUDA_VISIBLE_DEVICES": "", "OMP_NUM_THREADS": str(config["threads"]), + "MKL_NUM_THREADS": str(config["threads"]), "OPENBLAS_NUM_THREADS": str(config["threads"]), + "NUMBA_NUM_THREADS": str(config["threads"]), "NUMBA_CACHE_DIR": str(home / "numba")} + + +def run_worker(config, action, payload=None): + """Return only structured output; backend stdout/stderr may contain speech.""" + from inferencekit.registry import OpError + + with tempfile.TemporaryDirectory(prefix="inferencekit-audio-worker-") as directory: + root = Path(directory) + request = {"format": WORKER_FORMAT, "action": action, "config": config, "payload": payload} + input_path, output_path = root / "request.json", root / "response.json" + input_path.write_text(_json(request)) + try: + process = subprocess.run([config["python"], "-s", str(Path(__file__).resolve()), "--worker", str(input_path), str(output_path)], + cwd=root, env=_worker_environment(config, root), stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=config["timeout_s"], check=False) + except subprocess.TimeoutExpired: + raise OpError("audio scorer worker timed out; no scores written") from None + except OSError: + raise OpError("cannot start configured audio scorer Python") from None + try: + result = json.loads(output_path.read_text()) + _json(result) + except (OSError, ValueError, TypeError): + raise OpError("audio scorer worker returned missing or invalid output; no scores written") from None + if not isinstance(result, dict) or result.get("request_sha256") != _digest(request): + raise OpError("audio scorer worker output does not bind its request") + if process.returncode or result.get("error"): + error = result.get("error", {}) + if not isinstance(error, dict): + error = {} + stage = error.get("stage", "worker") + kind = error.get("type", "Error") + # Never relay arbitrary model exception messages, filenames or text. + if stage not in {"describe", "asr", "asr_retry", "naturalness", "speaker", "worker", "verify"}: + stage = "worker" + if not isinstance(kind, str) or not kind.isidentifier() or len(kind) > 80: + kind = "Error" + raise OpError(f"audio scorer {stage} failed ({kind}); check installed dependencies, local checkpoints and backend API; no scores written") + if set(result) != {"request_sha256", "result"}: + raise OpError("invalid audio scorer worker response") + return result["result"] + + +def load_config(path): + from inferencekit.quality import audio as qa + from inferencekit.registry import OpError + + doc, _ = qa.read_json(path) + try: + return normalize_config(doc, Path(path).resolve().parent) + except (ValueError, TypeError, OSError) as error: + code = str(error) if isinstance(error, ValueError) else "invalid_config" + raise OpError("audio scorer configuration: " + code) from None + + +def describe(config): + from inferencekit.quality import audio as qa + from inferencekit.registry import OpError + + result = run_worker(config, "describe") + if (not isinstance(result, dict) or set(result) != {"format", "scorers"} or result["format"] != DESCRIPTORS_FORMAT + or not isinstance(result["scorers"], dict) or set(result["scorers"]) != set(config["scorers"])): + raise OpError("invalid audio scorer descriptors") + qa.validate_policy({"acceptance": "exact", "scorers": result["scorers"]}) + return result + + +def windows(frames, rate, seconds): + step = max(1, int(rate * seconds)) + return [(start, min(start + step, frames)) for start in range(0, frames, step)] + + +def _audio(raw, rate, channels): + import numpy as np + from scipy.signal import resample_poly + + values = np.frombuffer(raw, dtype=" dict: log_path.unlink() env, env_info = servers.server_env(spec["env"]) cell["env"] = env_info + cell["server_env"] = resources.runtime_env(env, cwd=spec.get("cwd")) + cpu = cfg.get("resource_mode") == "cpu" if servers.port_in_use(spec["host"], spec["port"]): cell.update(status="port_busy", detail=f"something already listens on {spec['host']}:{spec['port']}; " "refusing to benchmark it in place of the engine") cell["t_end"] = time.time() return cell - gpu_before = cfg["gpu_reader"]() - server = None + gpu_before = None if cpu else cfg["gpu_reader"]() + server = monitor = None try: try: server = servers.launch(spec["launch"], env, log_path, cwd=spec.get("cwd")) except OSError as e: cell.update(status="launch_failed", detail=str(e)[:300]) return cell + if cpu: + monitor = resources.ProcessMonitor(server.pgid, env, cwd=spec.get("cwd")).start() ready = servers.wait_ready(server, base + spec["ready_check"], cfg["startup_timeout_s"], cfg["poll_s"]) cell["ready"] = ready if ready["status"] != "ready": cell["status"] = "startup_timeout" if ready["status"] == "timeout" else "exited_early" return cell + if monitor is not None: + monitor.settled() ctx = {"base_url": base, "clips": str(cfg["clips_path"]), "label": label, "round": k, "first_round": k - 1, "run_dir": str(run_dir), "endpoint_kind": spec["endpoint_kind"], "workflow": spec.get("workflow") or "", "engine_version": spec.get("version") or "", "extend": cfg["last_ev"].get(label) or "", "warmup": cfg["warmup"], "model": cfg.get("model_id") or ""} argv = clip_argv(cfg["clip_cmd"], ctx, cfg["clip_args"]) child_env = dict(os.environ) + child_env.pop("INFERENCEKIT_GPU_LOCK_HELD" if cpu else "INFERENCEKIT_CPU_LOCK_HELD", None) # Same evidence store as this run, and the GPU lock is already held (by this process). root = str(ev.default_root().resolve()) # The child's record is read from its stdout (parse_serve_output): full JSON, never the terse view. child_env.update({"INFERKIT_RUNS_DIR": root, "INFERENCEKIT_RUNS_DIR": root, - "INFERENCEKIT_GPU_LOCK_HELD": str(cfg.get("gpu_lock_path") or ""), + "INFERENCEKIT_CPU_LOCK_HELD" if cpu else "INFERENCEKIT_GPU_LOCK_HELD": + str(cfg.get("cpu_lock_path" if cpu else "gpu_lock_path") or ""), "INFERENCEKIT_OUTPUT": "json"}) t0 = time.monotonic() res = proc.run(argv, timeout=cfg["clip_timeout_s"], env=child_env) # a timeout kills its process group @@ -191,19 +199,25 @@ def run_clip_cell(spec: dict, k: int, position: int, cfg: dict) -> dict: return cell finally: if server is not None: - cell["stop"] = servers.stop(server, cfg["stop_timeout_s"]) + try: + cell["stop"] = servers.stop(server, cfg["stop_timeout_s"]) + finally: + if monitor is not None: + rr.attach_resources(cell, monitor.stop()) if not cell["stop"]["group_gone"] and cell["status"] == "ok": cell["status"] = "stop_failed" - cell["gpu_memory"] = servers.wait_gpu_memory(gpu_before, cfg["gpu_mem_tolerance_mib"], - cfg["gpu_mem_wait_s"], reader=cfg["gpu_reader"]) + if not cpu: + cell["gpu_memory"] = servers.wait_gpu_memory(gpu_before, cfg["gpu_mem_tolerance_mib"], + cfg["gpu_mem_wait_s"], reader=cfg["gpu_reader"]) cell["t_end"] = time.time() # --------------------------------------------------------------------------- summary -def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAULT_RESAMPLES, seed: int = 0) -> dict: +def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAULT_RESAMPLES, seed: int = 0, + expected_rounds: int | None = None) -> dict: """rr.summarize, with clips_per_hour ranked as a rate (rr.higher_is_better does not know '_per_hour').""" - out = rr.summarize(cells, labels, resamples=resamples, seed=seed, mode="single") + out = rr.summarize(cells, labels, resamples=resamples, seed=seed, mode="single", expected_rounds=expected_rounds) by_round: dict[tuple[str, str, str], dict[int, float]] = {} for c in cells: if c.get("status") in rr.STATUSES_OK: @@ -215,7 +229,7 @@ def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAU continue # the text runner ranks it already for bucket in sorted({r["bucket"] for r in out["rows"] if r["metric"] == name}): rows = [r for r in out["rows"] if r["metric"] == name and r["bucket"] == bucket] - eligible = [r for r in rows if not r["failed_requests"]] + eligible = [r for r in rows if not r["failed_requests"] and not r.get("excluded_from_best")] best = (max if higher else min)(eligible, key=lambda r: r["median"])["engine"] if eligible else None for r in rows: r.update(higher_is_better=higher, best=best) @@ -223,6 +237,8 @@ def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAU r["excluded_from_best"] = (f"{r['failed_requests']:g} failed clip(s) over {r['n']} round(s): " "its rate covers only the clips it returned") continue + if r.get("excluded_from_best"): + continue if best is None: continue b, c = by_round[(best, bucket, name)], by_round[(r["engine"], bucket, name)] @@ -234,13 +250,15 @@ def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAU seed=seed) r.update(ratio_to_best=res["ratio"], ratio_ci95=list(res["ci95"]), n_pairs=res["n_pairs"]) out["definitions"] = { + "peak_rss_mib": "median over rounds of sampled peak server-process-group RSS (host RAM, observed); " + "sampling can miss peaks and shared pages can be counted per process", "median": "median over rounds of the per-round value (clip_ms: mean over that round's clips of the task, " "or of every clip for 'all', from bench clip's requests artifact; client timer) (observed)", "clips_per_hour": "per round: 3,600,000 x clips / sum of their clip_ms, one clip at a time (derived)", "ci95": "bootstrap 95% CI of that median, rounds resampled", "ratio_to_best": "median(engine) / median(best engine) over rounds both completed, with a paired bootstrap " "95% CI resampling rounds (derived); best = lowest clip_ms, highest clips_per_hour, among " - "engines with no failed clip in the task", + "engines with complete successful rounds and no failed clip in the task", "failed_requests": "clips that failed in a round; an engine with any failure in a task is never best there " "and gets no ratio"} return out @@ -272,6 +290,15 @@ def markdown(summary: dict, plan: dict, cells: list[dict]) -> str: f"{ratio_ci} |") if not summary["rows"]: lines.append("| (no successful rounds) | | | | | | | |") + if any(r["metric"] == "peak_rss_mib" for r in summary["rows"]): + lines += ["", "peak_rss_mib is observed host RAM from server-process-group samples; sampling can miss peaks " + "and count shared pages per process. It is not VRAM."] + if summary.get("incomplete_rounds"): + lines += ["", "## Incomplete rounds", "", + "These engines retain their observed values but are excluded from ranking.", "", + "| engine | missing successful rounds |", "|---|---|"] + for item in summary["incomplete_rounds"]: + lines.append(f"| {item['engine']} | {', '.join(map(str, item['missing_rounds']))} |") failed = [c for c in cells if c.get("status") != "ok"] if failed: lines += ["", "## Failed cells", "", "| round | engine | status | detail |", "|---|---|---|---|"] @@ -287,9 +314,11 @@ def run_clip_rounds(manifest_path: str, clips_path: str, rounds: int | None = No run_dir: str | None = None, gpu_lock_path: str | None = None, startup_timeout_s: float = 1800, stop_timeout_s: float = 30, clip_timeout_s: float = 14400, warmup: int = 1, clip_cmd: str | None = None, clip_args: str | None = None, gpu_mem_tolerance_mib: int = 512, - gpu_mem_wait_s: float = 60, poll_s: float = 0.5, gpu_reader=None) -> dict: + gpu_mem_wait_s: float = 60, poll_s: float = 0.5, gpu_reader=None, + resource_mode: str = "gpu", cpu_lock_path: str | None = None) -> dict: """Run every (round, engine) clip cell under the GPU lock; return plan, cells, summary and paths.""" manifest = rr.load_manifest(manifest_path) + resource_mode = round_evidence.resource_mode(manifest, resource_mode) specs = clip_specs(manifest, manifest_path, engines) if not specs: raise OpError("no engines selected") @@ -316,28 +345,45 @@ def run_clip_rounds(manifest_path: str, clips_path: str, rounds: int | None = No "guidance, audio, image), all before the first timed clip, seeds outside " "the timed seed policy and prompts no timed clip has"}, "manifest_sha256": file_sha256(Path(manifest_path))[0]} + plan["resource_mode"] = resource_mode + if resource_mode == "cpu": + plan["cpu"] = rr.cpu_identity() cfg = {"run_dir": rdir, "clips_path": Path(clips_path).resolve(), "startup_timeout_s": startup_timeout_s, "stop_timeout_s": stop_timeout_s, "clip_timeout_s": clip_timeout_s, "warmup": warmup, "clip_cmd": clip_cmd, "clip_args": clip_args, "gpu_mem_tolerance_mib": gpu_mem_tolerance_mib, "gpu_mem_wait_s": gpu_mem_wait_s, "poll_s": poll_s, "gpu_reader": gpu_reader or servers.gpu_memory_used_mib, "model_id": (manifest.get("model") or {}).get("id"), "last_ev": {}} + cfg["resource_mode"] = resource_mode out_dir = rdir / OUT_DIR out_dir.mkdir(parents=True, exist_ok=True) cells: list[dict] = [] - with servers.gpu_lock(gpu_lock_path, owner=f"inferencekit bench rounds (clip) {rdir.name}") as lock: - plan["gpu_lock"] = lock["path"] - cfg["gpu_lock_path"] = lock["path"] + lock_context = (servers.gpu_lock(cpu_lock_path, owner=f"inferencekit bench rounds (clip) {rdir.name}", resource="CPU") + if resource_mode == "cpu" else + servers.gpu_lock(gpu_lock_path, owner=f"inferencekit bench rounds (clip) {rdir.name}")) + with lock_context as lock: + plan[f"{resource_mode}_lock"] = lock["path"] + cfg[f"{resource_mode}_lock_path"] = lock["path"] plan["started"] = time.strftime("%Y-%m-%dT%H:%M:%S%z") (out_dir / "plan.json").write_text(json.dumps(ev.redact(plan), indent=2) + "\n") by_label = {s["label"]: s for s in specs} + halted = None for k, order in enumerate(design["orders"], 1): for pos, label in enumerate(order): - cell = run_clip_cell(by_label[label], k, pos, cfg) + cell = (rr.skipped_cleanup_cell(by_label[label], k, pos, halted) if halted else + run_clip_cell(by_label[label], k, pos, cfg)) + failure = rr.cleanup_failure(cell) + if failure: + halted = {"label": label, "round": k, "detail": failure} + plan["aborted_after_cleanup_failure"] = halted + cell["cleanup_failure"] = failure + if cell.get("status") == "ok": + cell["status"] = ("stop_failed" if (cell.get("stop") or {}).get("group_gone") is False else + "gpu_memory_not_released") cells.append(cell) (out_dir / f"{label}.r{k}.json").write_text(json.dumps(ev.redact(cell), indent=2) + "\n") plan["ended"] = time.strftime("%Y-%m-%dT%H:%M:%S%z") - summary = summarize(cells, labels) - plan["clip_evidence_final"] = dict(cfg["last_ev"]) # each engine's bench clip record with all its rounds + summary = summarize(cells, labels, expected_rounds=plan["rounds"]) + plan["clip_evidence_final"] = rr.finalized_evidence(cfg["last_ev"], cells, plan) (out_dir / "plan.json").write_text(json.dumps(ev.redact(plan), indent=2) + "\n") (out_dir / "summary.json").write_text(json.dumps(ev.redact({"plan": plan, "summary": summary}), indent=2) + "\n") md = markdown(summary, plan, cells) @@ -350,15 +396,17 @@ def bench_rounds_clip(call, *, manifest: str, clips: str, rounds: int | None = N run_dir: str | None = None, gpu_lock: str | None = None, startup_timeout_s: float = 1800, stop_timeout_s: float = 30, clip_timeout_s: float = 14400, warmup: int = 1, clip_args: str | None = None, clip_cmd: str | None = None, gpu_mem_tolerance_mib: int = 512, - gpu_reader=None) -> dict: + gpu_reader=None, resource_mode: str = "gpu", cpu_lock_path: str | None = None) -> dict: """bench rounds in clip mode: run the rounds and store one ``clip_benchmark`` record (the summary).""" from inferencekit.ops.bench_rounds import _limitations # the text runner's cell, order and failure notes - others = gpu_processes() + resource_mode = round_evidence.resource_mode(rr.load_manifest(manifest), resource_mode) + others = [] if resource_mode == "cpu" else gpu_processes() result = run_clip_rounds(manifest, clips, rounds, engines=engines, run_dir=run_dir, gpu_lock_path=gpu_lock, startup_timeout_s=startup_timeout_s, stop_timeout_s=stop_timeout_s, clip_timeout_s=clip_timeout_s, warmup=warmup, clip_cmd=clip_cmd, clip_args=clip_args, - gpu_mem_tolerance_mib=gpu_mem_tolerance_mib, gpu_reader=gpu_reader) + gpu_mem_tolerance_mib=gpu_mem_tolerance_mib, gpu_reader=gpu_reader, + resource_mode=resource_mode, cpu_lock_path=cpu_lock_path) plan, summary, rdir, clipset = result["plan"], result["summary"], Path(result["run_dir"]), result["clipset"] labels = [e["label"] for e in plan["engines"]] limitations, missing = _limitations({**result, "resolved_config": {}}) @@ -390,7 +438,8 @@ def bench_rounds_clip(call, *, manifest: str, clips: str, rounds: int | None = N n=r["n_pairs"], ci95=r["ratio_ci95"], subject=subject)) m = result["manifest"].get("model") or {} kinds = sorted({e["endpoint_kind"] for e in plan["engines"]}) - ident = ev.identity(gpu=gpu_identity(), engine={"name": "+".join(labels), "flags": {"engines": labels}}, + ident = ev.identity(gpu={} if resource_mode == "cpu" else gpu_identity(), cpu=plan.get("cpu"), + engine={"name": "+".join(labels), "flags": {"engines": labels}}, model={"id": m.get("id"), "revision": m.get("revision"), "dtype": m.get("precision")}, workload={"mode": "single", "batch": 1, "endpoint_kind": "+".join(kinds), "clip_set": clipset.identity(), @@ -406,10 +455,11 @@ def bench_rounds_clip(call, *, manifest: str, clips: str, rounds: int | None = N for name, role in (("summary.md", "summary"), ("summary.json", "summary"), ("plan.json", "plan")) if (rdir / OUT_DIR / name).is_file()] artifacts += [ev.artifact(rdir / c["log"], rdir, "server_log") for c in result["cells"] - if (rdir / c["log"]).is_file()] + if c.get("log") and (rdir / c["log"]).is_file()] cells = [{k: c.get(k) for k in ("round", "position", "label", "kind", "endpoint_kind", "status", "ready", "clip", "stop", "gpu_memory", "env", "log", "metrics_source", "claim_valid", - "violations", "cache_allowed_clips", "t_start", "t_end", "detail")} + "violations", "cache_allowed_clips", "t_start", "t_end", "detail", "resources", + "server_env")} for c in result["cells"]] record = ev.build(call, kind=bc.KIND, claim_class="observed", summary=line, identity_=ident, metrics=metrics, artifacts=artifacts, missing=missing, limitations=limitations, diff --git a/src/inferencekit/serve/media_rounds.py b/src/inferencekit/serve/media_rounds.py new file mode 100644 index 0000000..60642c4 --- /dev/null +++ b/src/inferencekit/serve/media_rounds.py @@ -0,0 +1,274 @@ +"""Fresh-process audio/image cells using the existing client timers and Williams order.""" +from __future__ import annotations + +import json +import os +import shlex +import shutil +import sys +import time +from pathlib import Path + +from inferencekit import evidence as ev, proc +from inferencekit.canonical import file_sha256 +from inferencekit.cpuinfo import cpu_identity +from inferencekit.registry import OpError +from inferencekit.serve import resources, round_evidence, rounds as rr, servers + +METRICS = { + "audio": {"audio_ms": "ms", "first_audio_p95_ms": "ms", "rtf": "x"}, + "image": {"image_ms": "ms", "images_per_hour": "images/h"}, +} + + +def mode_for(manifest: dict, requested: str, manifest_path: str) -> str: + if requested != "auto": + return requested + workload = manifest.get("workload") or {} + modes = workload.get("modes") or [] + if len(modes) == 1: + return modes[0] + if len(modes) > 1: + raise OpError("manifest declares multiple workload modes; select one with --mode") + modality = workload.get("modality") + if not modality: + intake = Path(manifest_path).resolve().parent / "intake.normalized.json" + if intake.is_file(): + try: + modality = json.loads(intake.read_text()).get("modality") + except (ValueError, OSError): + raise OpError("cannot read intake.normalized.json for automatic modality routing") from None + if modality: + mode = {"tts": "audio", "audio": "audio", "image": "image", "video": "clip", + "text": "single", "llm": "single"}.get(modality) + if mode is None: + raise OpError(f"no bench rounds client for modality {modality}; select a supported --mode") + return mode + kinds = {e.get("endpoint_kind") for e in manifest.get("engines", [])} - {None} + if kinds == {"audio_http"}: + return "audio" + if kinds == {"image_http"}: + return "image" + if kinds and kinds <= {"clip_http", "comfyui"}: + return "clip" + return "single" + + +def media_argv(mode: str, ctx: dict, template: str | None, extra: str | None) -> list[str]: + # The scheduler owns round indices and endpoint/configuration identity. Extra flags + # may select client timeouts, quality fixtures or cache policy, not replace the plan. + extra_words = shlex.split(extra or "") + protected = {"--base-url", "--fixtures", "--prompts", "--model", "--label", "--rounds", "--first-round", + "--extend", "--endpoint-id", "--device", "--hardware-name", "--gpu-name", "--run-dir", + "--model-revision", "--model-dtype", "--model-quant", "--dtype", "--quant", "--engine-version", + "--seed-base", "--seed-policy", "--seeds", "--sample-rate", "--channels", "--settings", "--warmup", + "--gpu-count"} + if any(w.split("=", 1)[0] in protected for w in extra_words): + raise OpError("media --serve-args cannot override scheduler identity, rounds, seeds or workload settings") + words = shlex.split(template) if template else [sys.executable, "-m", "inferencekit", "bench", mode] + if any("{" in w or "}" in w for w in words): + raise OpError("audio/image --serve-cmd replaces the program only; scheduler arguments are always appended") + argv = words + ["--base-url", ctx["base_url"], "--fixtures" if mode == "audio" else "--prompts", ctx["prompts"], + "--model", ctx["model"], "--label", ctx["label"], "--rounds", "1", + "--first-round", str(ctx["first_round"]), "--warmup", str(ctx["warmup"]), + "--seed-base", str(ctx["seed"]), "--device", ctx["device"]] + if ctx.get("run_dir"): + argv += ["--run-dir", ctx["run_dir"]] + for key, flag in (("revision", "--model-revision"), ("engine_version", "--engine-version"), + ("extend", "--extend"), ("precision", "--model-dtype" if mode == "audio" else "--dtype"), + ("quant", "--model-quant" if mode == "audio" else "--quant")): + if ctx.get(key): + argv += [flag, str(ctx[key])] + argv += ["--hardware-name" if mode == "audio" or ctx["device"] == "cpu" else "--gpu-name", ctx["hardware"]] + if ctx["device"] == "gpu": + argv += ["--gpu-count", str(ctx.get("gpu_count", 1))] + if mode == "audio": + argv += ["--endpoint-id", ctx["label"], "--sample-rate", str(ctx["sample_rate"]), + "--channels", str(ctx["channels"]), "--settings", json.dumps(ctx["settings"])] + return argv + extra_words + + +def run_cell(spec: dict, k: int, position: int, cfg: dict) -> dict: + mode, cpu = cfg["mode"], cfg["device"] == "cpu" + cell = {"round": k, "position": position, "label": spec["label"], "kind": spec["kind"], + "status": None, "t_start": time.time(), "log": f"logs/{spec['label']}.r{k}.log"} + log = cfg["run_dir"] / cell["log"] + env, cell["env"] = servers.server_env(spec["env"]) + cell["server_env"] = resources.runtime_env(env, cwd=spec.get("cwd")) + if servers.port_in_use(spec["host"], spec["port"]): + cell.update(status="port_busy", detail="server port already in use; refusing to measure another process", + t_end=time.time()) + return cell + base = f"http://{spec['host']}:{spec['port']}" + before = None if cpu else cfg["gpu_reader"]() + server = monitor = None + try: + try: + server = servers.launch(spec["launch"], env, log, cwd=spec.get("cwd")) + except OSError as exc: + cell.update(status="launch_failed", detail=ev.redact_text(str(exc))[:300]) + return cell + if cpu: + monitor = resources.ProcessMonitor(server.pgid, env, cwd=spec.get("cwd")).start() + cell["ready"] = servers.wait_ready(server, base + spec["ready_check"], cfg["startup_timeout_s"], cfg["poll_s"]) + if cell["ready"]["status"] != "ready": + cell["status"] = "startup_timeout" if cell["ready"]["status"] == "timeout" else "exited_early" + return cell + if monitor: + monitor.settled() + ctx = {**cfg["client"], "base_url": base, "label": spec["label"], "first_round": k - 1, + "run_dir": str(cfg["run_dir"].resolve()), + "engine_version": spec.get("version"), "extend": cfg["last_ev"].get(spec["label"])} + child_env = dict(os.environ) + child_env.pop("INFERENCEKIT_GPU_LOCK_HELD" if cpu else "INFERENCEKIT_CPU_LOCK_HELD", None) + root = str(ev.default_root().resolve()) + child_env.update(INFERKIT_RUNS_DIR=root, INFERENCEKIT_RUNS_DIR=root, INFERENCEKIT_OUTPUT="json") + child_env[f"INFERENCEKIT_{'CPU' if cpu else 'GPU'}_LOCK_HELD"] = cfg["lock"] + result = proc.run(media_argv(mode, ctx, cfg["serve_cmd"], cfg["serve_args"]), + timeout=cfg["serve_timeout_s"], env=child_env) + record = rr.parse_serve_output(result.stdout) + cell[mode] = {"returncode": result.returncode, "evidence_id": (record or {}).get("evidence_id"), + "stderr_tail": ev.redact_text(result.stderr[-1500:]) if result.returncode else None} + cell["metrics"] = {} + if record and result.returncode == 0 and record.get("kind") == f"{mode}_benchmark": + rows = [r for r in record.get("data", {}).get("rounds", []) if r.get("index") == k - 1] + if len(rows) == 1 and not ev.integrity_problem(record): + row = rows[0] + mets = {name: {"value": row[name], "unit": unit, "stat": "per_round", "n": cfg["n"]} + for name, unit in METRICS[mode].items() if round_evidence.finite(row.get(name))} + mets["failed_requests"] = {"value": row.get("n_failed", 0), "unit": "requests", "n": cfg["n"]} + if all(round_evidence.finite(row.get(name)) for name in METRICS[mode]) or ( + round_evidence.finite(row.get("n_failed")) and row["n_failed"] > 0): + cell["metrics"] = {"all": mets} + cfg["last_ev"][spec["label"]] = record["evidence_id"] + cell["claim_valid"] = record.get("data", {}).get("claim_validity", {}).get("valid") + cell["status"], detail = rr.cell_status(result.returncode, record, cell["metrics"]) + if cell.get("claim_valid") is not True and cell["status"] == "ok": + cell["status"] = "claim_invalid" + if detail: + cell["detail"] = detail + return cell + finally: + if server is not None: + try: + cell["stop"] = servers.stop(server, cfg["stop_timeout_s"]) + finally: + if monitor is not None: + rr.attach_resources(cell, monitor.stop()) + if not cell["stop"]["group_gone"]: + cell["status"] = "stop_failed" + if not cpu: + cell["gpu_memory"] = servers.wait_gpu_memory(before, cfg["gpu_mem_tolerance_mib"], + cfg["gpu_mem_wait_s"], reader=cfg["gpu_reader"]) + cell["t_end"] = time.time() + + +def run_media_rounds(manifest_path, prompts, mode, *, rounds=None, engines=None, run_dir=None, + device="auto", cpu_lock=None, gpu_lock=None, startup_timeout_s=1200, + stop_timeout_s=30, serve_timeout_s=7200, warmup=1, serve_args=None, serve_cmd=None, + gpu_mem_tolerance_mib=512, gpu_mem_wait_s=60, gpu_reader=None, poll_s=0.5): + manifest = rr.load_manifest(manifest_path) + device = round_evidence.resource_mode(manifest, device) + specs = rr.engine_specs(manifest, engines) + if mode not in METRICS or not specs: + raise OpError("media rounds requires audio/image mode and at least one engine") + rounds = rr.default_rounds(len(specs)) if rounds is None else rounds + if not 1 <= rounds <= 10000 or not 0 <= warmup <= 100: + raise OpError("rounds must be 1..10000 and warmup 0..100") + if any(not round_evidence.finite(x) or x <= 0 for x in (startup_timeout_s, serve_timeout_s, poll_s)) or \ + any(not round_evidence.finite(x) or x < 0 for x in (stop_timeout_s, gpu_mem_wait_s, gpu_mem_tolerance_mib)): + raise OpError("timeouts and polling must be finite positive values; stop/memory waits may be zero") + raw = {e.get("label", e.get("name")): e for e in manifest["engines"]} + for spec in specs: + source = raw[spec["label"]] + if source.get("endpoint_kind", f"{mode}_http") != f"{mode}_http": + raise OpError(f"{mode} mode needs {mode}_http endpoints") + if not source.get("ready_check"): + spec["ready_check"] = "/health" + if mode == "audio": + from inferencekit.ops.bench_audio import load_fixtures + fixtures, _ = load_fixtures(prompts) + else: + from inferencekit.ops.bench_image import load_image_set + fixtures = load_image_set(prompts) + workload, model, hardware = (manifest.get(k) or {} for k in ("workload", "model", "hardware")) + cpu = cpu_identity() if device == "cpu" else None + hardware_name = cpu.get("model") if cpu else hardware.get("gpu") + if not hardware_name or not model.get("id"): + raise OpError("media rounds needs model.id and hardware identity") + if mode == "image" and workload.get("settings"): + raise OpError("image settings belong in each image fixture's settings object") + client = {"prompts": str(Path(prompts).resolve()), "model": model["id"], "revision": model.get("revision"), + "precision": model.get("precision"), "quant": workload.get("quant"), "hardware": hardware_name, + "gpu_count": hardware.get("count", 1), + "device": device, "seed": workload.get("seed", 0), "warmup": warmup, + "sample_rate": workload.get("sample_rate", 44100), "channels": workload.get("channels", 1), + "settings": workload.get("settings", {})} + # Validate all scheduler-owned arguments before launching any server. + media_argv(mode, {**client, "base_url": "http://127.0.0.1:1", "label": "check", "first_round": 0}, + serve_cmd, serve_args) + rdir = Path(run_dir) if run_dir else Path(manifest_path).resolve().parent + out = rdir / f"{mode}_rounds" + out.mkdir(parents=True, exist_ok=True) + labels = [s["label"] for s in specs] + plan = {"mode": mode, "resource_mode": device, "rounds": rounds, "design": rr.round_orders(labels, rounds), + "engines": [{k: s.get(k) for k in ("label", "name", "kind", "version", "role")} for s in specs], + "prompts": {"name": Path(prompts).name, "sha256": file_sha256(Path(prompts))[0], "n": len(fixtures)}, + "manifest_sha256": file_sha256(Path(manifest_path))[0], "cpu": cpu} + cfg = {"mode": mode, "device": device, "run_dir": rdir, "client": client, "last_ev": {}, "n": len(fixtures), + "startup_timeout_s": startup_timeout_s, "stop_timeout_s": stop_timeout_s, + "serve_timeout_s": serve_timeout_s, "poll_s": poll_s, "serve_cmd": serve_cmd, "serve_args": serve_args, + "gpu_reader": gpu_reader or servers.gpu_memory_used_mib, "gpu_mem_wait_s": gpu_mem_wait_s, + "gpu_mem_tolerance_mib": gpu_mem_tolerance_mib} + lock = (servers.gpu_lock(cpu_lock, owner=f"inferencekit bench rounds ({mode})", resource="CPU") + if device == "cpu" else servers.gpu_lock(gpu_lock, owner=f"inferencekit bench rounds ({mode})")) + cells, failed, cleanup_failed = [], set(), False + with lock as held: + cfg["lock"] = held["path"] + plan[f"{device}_lock"] = held["path"] + (out / "plan.json").write_text(json.dumps(ev.redact(plan), indent=2) + "\n") + by_label = {s["label"]: s for s in specs} + for k, order in enumerate(plan["design"]["orders"], 1): + for position, label in enumerate(order): + if cleanup_failed: + cell = {"round": k, "position": position, "label": label, "status": "skipped_after_cleanup_failure", + "detail": "previous server cleanup failed; no further measurement can be isolated"} + elif label in failed: + cell = {"round": k, "position": position, "label": label, "status": "skipped_after_failure", + "detail": "previous cell failed; preserving scheduled indices instead of renumbering"} + else: + cell = run_cell(by_label[label], k, position, cfg) + if cell["status"] != "ok": + failed.add(label) + memory = cell.get("gpu_memory") or {} + cleanup_failed = (cell.get("stop", {}).get("group_gone") is False or + memory.get("checked") is True and memory.get("returned") is False) + cells.append(cell) + (out / f"{label}.r{k}.json").write_text(json.dumps(ev.redact(cell), indent=2) + "\n") + plan[f"{mode}_evidence_final"] = round_evidence.finalize_arm_records(cfg["last_ev"], cells, plan, cpu=cpu) + store = ev.Store() + evidence_dir = rdir / "evidence" + evidence_dir.mkdir(exist_ok=True) + for reference in plan[f"{mode}_evidence_final"].values(): + # Standalone per-arm bundles retain hash-matching request artifacts; shared + # outputs/metrics support review build, whose top-level evidence scan needs JSON copies. + shutil.copytree(store.run_dir(reference), rdir / "artifacts" / reference, dirs_exist_ok=True) + (evidence_dir / f"{reference}.json").write_text(json.dumps(store.load(reference), indent=2) + "\n") + summary = rr.summarize(cells, labels, expected_rounds=rounds) + summary["definitions"]["median"] = "Median of client-timed per-round values; memory is sampled server process-group RSS." + lines = [f"# {mode.title()} benchmark", "", "| Engine | Metric | Median | 95% CI | Rounds |", + "|---|---|---:|---|---:|"] + for row in summary["rows"]: + lines.append(f"| {row['engine']} | {row['metric']} | {rr._fmt(row['median'])} {row['unit']} | " + f"{rr._fmt(row['ci95'][0])}–{rr._fmt(row['ci95'][1])} | {row['n']}/{rounds} |") + for label in labels: + missing = [c["round"] for c in cells if c["label"] == label and c["status"] != "ok"] + if missing: + lines += ["", f"{label}: incomplete rounds {missing}; excluded from ranking."] + lines += ["", "Timing does not establish output quality or sustained reliability."] + markdown = "\n".join(lines) + "\n" + (out / "plan.json").write_text(json.dumps(ev.redact(plan), indent=2) + "\n") + (out / "summary.json").write_text(json.dumps(ev.redact({"plan": plan, "summary": summary}), indent=2) + "\n") + (out / "summary.md").write_text(markdown) + return {"plan": plan, "cells": cells, "summary": summary, "run_dir": rdir, "manifest": manifest, + "markdown": markdown, "resolved_config": {}} diff --git a/src/inferencekit/serve/resources.py b/src/inferencekit/serve/resources.py new file mode 100644 index 0000000..2d32c81 --- /dev/null +++ b/src/inferencekit/serve/resources.py @@ -0,0 +1,296 @@ +"""Bounded, read-only CPU/process telemetry for a launched server process group. + +RSS is sampled concurrently across the group's live processes. Per-process VmHWM +is retained separately: adding those high-water marks does not give a group peak. +""" + +from __future__ import annotations + +import hashlib +import math +import os +import re +import stat +import sys +import threading +import time +from pathlib import Path + +from inferencekit.serve import servers + +PROC_ROOT = Path("/proc") +MAX_PROCESSES = 128 +MAX_RECORDED_PROCESSES = 1024 +MAX_TASKS = 256 +MAX_STATUS_BYTES = 64 * 1024 +MAX_PRELOAD_BYTES = 64 * 1024 * 1024 + +_THREAD_VARS = frozenset(("OMP_NUM_THREADS", "OMP_THREAD_LIMIT", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", + "GOTO_NUM_THREADS", "BLIS_NUM_THREADS", "NUMEXPR_NUM_THREADS", "NUMEXPR_MAX_THREADS", + "VECLIB_MAXIMUM_THREADS", "TF_NUM_INTRAOP_THREADS", "TF_NUM_INTEROP_THREADS", + "RAYON_NUM_THREADS", "TBB_NUM_THREADS")) +_CHOICES = {"OMP_DYNAMIC": {"true", "false", "1", "0"}, "MKL_DYNAMIC": {"true", "false", "1", "0"}, + "OMP_WAIT_POLICY": {"active", "passive"}, + "MKL_THREADING_LAYER": {"intel", "sequential", "gnu", "tbb", "pgi"}, + "MKL_INTERFACE_LAYER": {"lp64", "ilp64", "gnu,lp64", "gnu,ilp64"}, + "MKL_ENABLE_INSTRUCTIONS": {"sse4_2", "avx", "avx2", "avx512", "avx512_mic", "avx512_mic_e1", "avx512_e1"}, + "MKL_VERBOSE": {"0", "1"}, "DNNL_VERBOSE": {"0", "1", "2"}, "ONEDNN_VERBOSE": {"0", "1", "2"}} +_ISA_VALUES = {"default", "sse41", "avx", "avx2", "avx2_vnni", "avx2_vnni_2", "avx512_core", + "avx512_core_vnni", "avx512_core_bf16", "avx512_core_fp16", "avx512_core_amx", + "avx512_core_amx_fp16", "avx10_1_512", "avx10_1_512_amx"} +_CHOICES.update({"DNNL_MAX_CPU_ISA": _ISA_VALUES, "ONEDNN_MAX_CPU_ISA": _ISA_VALUES}) +_CPU_LIST = re.compile(r"\d+(?:-\d+)?(?:,\d+(?:-\d+)?)*") + + +def _control_value(name: str, value: str) -> str | None: + if len(value) > 512: + return None + value = value.strip() + if name in _THREAD_VARS: + if re.fullmatch(r"[1-9]\d{0,6}(?:,[1-9]\d{0,6})*", value): + return value + elif name in _CHOICES: + if value.lower() in _CHOICES[name]: + return value.lower() + elif name == "OMP_PROC_BIND": + if all(x in {"true", "false", "master", "primary", "close", "spread"} + for x in value.lower().split(",")): + return value.lower() + elif name == "OMP_PLACES": + if re.fullmatch(r"(?:threads|cores|sockets|ll_caches|numa_domains)(?:\(\d+\))?", value.lower()) or \ + re.fullmatch(r"[\d{},:!\s-]+", value): + return value.lower() + elif name == "GOMP_CPU_AFFINITY": + if re.fullmatch(r"\d+(?:-\d+(?::\d+)?)?(?:\s+\d+(?:-\d+(?::\d+)?)?)*", value): + return value + elif name == "KMP_BLOCKTIME": + if value.lower() == "infinite" or re.fullmatch(r"\d{1,9}(?:ms|us)?", value): + return value.lower() + elif name == "KMP_AFFINITY": + # No arbitrary text or paths. These are the documented tuning tokens and numeric masks. + words = re.findall(r"[A-Za-z_]+", value.lower()) + if re.fullmatch(r"[A-Za-z_0-9,={}\[\]:@.\s-]+", value) and all( + w in {"verbose", "noverbose", "warnings", "nowarnings", "respect", "norespect", "granularity", + "fine", "thread", "core", "package", "node", "group", "compact", "scatter", "balanced", + "explicit", "none", "disabled", "proclist", "duplicates", "noduplicates"} for w in words): + return value.lower() + return None + + +def _preload(path: str, *, cwd: str | os.PathLike | None = None) -> dict: + p = Path(path) + name = p.name + if not re.fullmatch(r"lib[A-Za-z0-9_.+-]{1,120}\.(?:so(?:\.\d+)*|dylib)", name): + return {"name": None, "sha256": None, "status": "invalid_or_redacted"} + result = {"name": name, "sha256": None, "status": "unavailable"} + # A bare soname uses the dynamic loader's search, not the launch directory. + # Do not fingerprint a same-named cwd file that the loader may never load. + if not p.is_absolute() and "/" not in path: + return result + if not p.is_absolute() and cwd is not None: + p = Path(cwd) / p + try: + fd = os.open(p, os.O_RDONLY | os.O_NONBLOCK) + with os.fdopen(fd, "rb") as source: + info = os.fstat(source.fileno()) + if not stat.S_ISREG(info.st_mode): + return {**result, "status": "not_regular"} + if info.st_size > MAX_PRELOAD_BYTES: + return {**result, "status": "size_limit"} + digest, size = hashlib.sha256(), 0 + while chunk := source.read(64 * 1024): + size += len(chunk) + if size > MAX_PRELOAD_BYTES: + return {**result, "status": "size_limit"} + digest.update(chunk) + return {**result, "sha256": digest.hexdigest(), "status": "hashed"} + except OSError: + return result # Never put exception text (which contains private paths) into evidence. + + +def runtime_env(env: dict, *, cwd: str | os.PathLike | None = None) -> dict: + """Only recognized CPU controls and preload basenames/content hashes; never arbitrary env values. + + Controls are requested values, not evidence that a runtime honored them. Invalid + values are null and named in ``omitted``. Relative preload paths use the server's + launch directory (the caller's by default); loader-only sonames remain unavailable. + """ + names = _THREAD_VARS | _CHOICES.keys() | {"OMP_PROC_BIND", "OMP_PLACES", "GOMP_CPU_AFFINITY", + "KMP_BLOCKTIME", "KMP_AFFINITY"} + controls, omitted = {}, [] + for name in sorted(names & env.keys()): + raw = env[name] + value = _control_value(name, str(raw)) if isinstance(raw, (str, int)) and not isinstance(raw, bool) else None + controls[name] = value + if value is None: + omitted.append(name) + preload = [] + raw = env.get("LD_PRELOAD") + if isinstance(raw, str) and raw: + if len(raw) <= 8192: + paths = [p for p in re.split(r"[:\s]+", raw) if p] + preload = [_preload(p, cwd=cwd) for p in paths[:16]] + if len(paths) > 16: + omitted.append("LD_PRELOAD (entry limit)") + else: + omitted.append("LD_PRELOAD (length limit)") + return {"controls": controls, "ld_preload": preload, "omitted": omitted} + + +def _status(path: Path) -> dict | None: + try: + with path.open("rb") as source: + raw = source.read(MAX_STATUS_BYTES + 1) + if len(raw) > MAX_STATUS_BYTES: + return None + except OSError: + return None + fields = {} + for line in raw.decode("ascii", errors="replace").splitlines(): + name, sep, value = line.partition(":") + if sep and name in {"VmRSS", "VmHWM", "Threads", "Cpus_allowed_list"}: + fields[name] = value.strip() + out = {} + for source, target in (("VmRSS", "rss_mib"), ("VmHWM", "vm_hwm_mib")): + match = re.fullmatch(r"(\d+)\s+kB", fields.get(source, "")) + out[target] = int(match[1]) / 1024 if match else None + threads = fields.get("Threads", "") + out["threads"] = int(threads) if threads.isdigit() else None + affinity = fields.get("Cpus_allowed_list", "") + out["affinity"] = affinity if len(affinity) <= 4096 and _CPU_LIST.fullmatch(affinity) else None + return out + + +class ProcessMonitor: + """Sample only ``pgid``; no signals, mutation, or inclusion of the separate benchmark client. + + ``start`` captures launch, ``settled`` captures readiness/idle state when called, + and ``stop`` returns the complete JSON-safe result. All three are idempotent. + """ + + def __init__(self, pgid: int, requested_env: dict, poll_s: float = 0.05, + *, cwd: str | os.PathLike | None = None): + if type(pgid) is not int or pgid <= 0 or not math.isfinite(poll_s) or poll_s <= 0: + raise ValueError("pgid and finite poll_s must be positive") + self.pgid, self.poll_s = pgid, max(0.001, poll_s) + self.requested = runtime_env(requested_env, cwd=cwd) + self._event, self._lock = threading.Event(), threading.Lock() + self._thread = None + self._started = self._stopped = False + self._samples = 0 + self._peak_rss = self._peak_threads = None + self._processes, self._observations, self._affinity = {}, {}, set() + self._notes = set() + self._partial = False + self._available = sys.platform.startswith("linux") and PROC_ROOT.is_dir() + + def _sample(self, phase: str | None = None): + with self._lock: + if not self._available: + return + try: + members = servers.group_members(self.pgid) + except OSError: + members = None + if members is None: + self._notes.add("Process-group discovery is unavailable.") + self._partial = True + return + if len(members) > MAX_PROCESSES: + self._notes.add("Process count exceeded the sampling limit; group measurements are partial.") + self._partial = True + rows = [] + for pid in members[:MAX_PROCESSES]: + row = _status(PROC_ROOT / str(pid) / "status") + if row is None: + self._notes.add("A process status was unreadable or exited during sampling; brief peaks may be missed.") + self._partial = True + continue + rows.append({"pid": pid, **row}) + if pid not in self._processes: + if len(self._processes) >= MAX_RECORDED_PROCESSES: + self._notes.add("Per-process diagnostic limit reached.") + continue + self._processes[pid] = {"pid": pid, "peak_rss_mib": None, "vm_hwm_mib": None, + "peak_threads": None, "affinity": []} + saved = self._processes[pid] + for source, target in (("rss_mib", "peak_rss_mib"), ("vm_hwm_mib", "vm_hwm_mib"), + ("threads", "peak_threads")): + if row[source] is not None: + saved[target] = max(saved[target] or 0, row[source]) + if row["affinity"] is not None: + if len(self._affinity) < 32: + self._affinity.add(row["affinity"]) + saved["affinity"] = sorted(set(saved["affinity"]) | {row["affinity"]})[:32] + rss = sum(r["rss_mib"] for r in rows) if rows and all(r["rss_mib"] is not None for r in rows) else None + threads = sum(r["threads"] for r in rows) if rows and all(r["threads"] is not None for r in rows) else None + if rss is not None: + self._peak_rss = max(self._peak_rss or 0, rss) + if threads is not None: + self._peak_threads = max(self._peak_threads or 0, threads) + self._samples += 1 + if phase: + # Thread affinities can differ from the process leader. Inspect them only at + # lifecycle boundaries, under one total bound, to keep the timing sampler small. + masks, count = set(), 0 + for row in rows: + try: + with os.scandir(PROC_ROOT / str(row["pid"]) / "task") as entries: + for entry in entries: + if not entry.name.isdigit(): + continue + if count >= MAX_TASKS: + self._notes.add("Thread affinity inspection reached its limit; masks are partial.") + break + count += 1 + task = _status(Path(entry.path) / "status") + if task and task["affinity"]: + masks.add(task["affinity"]) + except OSError: + self._notes.add("Per-thread affinity was unavailable for one or more processes.") + self._observations[phase] = {"t_observed": time.time(), "processes": len(rows), "rss_mib": rss, + "threads": threads, "affinity": sorted(masks), "tasks_checked": count} + + def _run(self): + while not self._event.wait(self.poll_s): + self._sample() + + def start(self): + if not self._started and not self._stopped: + self._started = True + self._sample("launch") + if self._available: + self._thread = threading.Thread(target=self._run, name="inferencekit-process-monitor", daemon=True) + self._thread.start() + return self + + def settled(self): + if self._started and not self._stopped and "settled" not in self._observations: + self._sample("settled") + return self + + def stop(self) -> dict: + if not self._stopped: + self._event.set() + if self._thread is not None: + self._thread.join() + if self._started: + self._sample("final") + self._stopped = True + limitations = ["Sampled RSS can miss peaks between observations; it is not an allocation high-water mark.", + "Group RSS sums resident pages and can count shared pages more than once.", + "Per-process VmHWM values are individual lifetime peaks; their sum is not a simultaneous group peak.", + "Observed threads include idle and runtime helper threads; requested controls do not prove effective pool sizes.", + "Processes that leave the server process group are outside this measurement."] + if not self._available: + limitations.append("Linux /proc telemetry is unavailable on this host.") + if "settled" not in self._observations: + limitations.append("No settled observation was requested after readiness.") + return {"status": ("partial" if self._partial else "observed") if self._peak_rss is not None else "unavailable", + "pgid": self.pgid, + "method": "sampled_process_group_rss", "peak_rss_mib": self._peak_rss, + "peak_threads": self._peak_threads, "affinity": sorted(self._affinity)[:32], + "requested": self.requested, "observations": dict(self._observations), + "processes": [self._processes[p] for p in sorted(self._processes)], + "samples": self._samples, "poll_s": self.poll_s, + "limitations": limitations + sorted(self._notes)} diff --git a/src/inferencekit/serve/round_evidence.py b/src/inferencekit/serve/round_evidence.py new file mode 100644 index 0000000..473144e --- /dev/null +++ b/src/inferencekit/serve/round_evidence.py @@ -0,0 +1,203 @@ +"""Attach scheduler-observed execution context without changing client timers or source records.""" +from __future__ import annotations + +import copy +import json +import math +from collections import Counter +from pathlib import Path + +from inferencekit import evidence as ev +from inferencekit.canonical import file_sha256 +from inferencekit.registry import OpError + +MAX_ROUNDS = 100_000 +MAX_ROUND_INDEX = 100_000 + + +def resource_mode(manifest: dict, requested: str = "auto") -> str: + if requested not in ("auto", "cpu", "gpu"): + raise OpError("device must be auto, cpu or gpu") + hardware = manifest.get("hardware") or {} + declared = hardware.get("backend") + if declared == "cpu" and hardware.get("gpu") or declared in ("cuda", "rocm", "metal") and hardware.get("cpu"): + raise OpError("manifest hardware has conflicting CPU and accelerator declarations") + if requested == "auto": + return "cpu" if declared == "cpu" else "gpu" + if declared == "cpu" and requested == "gpu": + raise OpError("--device gpu conflicts with the manifest's CPU hardware") + if requested == "cpu" and declared in ("cuda", "rocm", "metal"): + raise OpError("--device cpu conflicts with the manifest's accelerator hardware") + return requested + + +def finite(value): + return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) + + +def memory_of(record: dict) -> dict: + """Known memory measurements, with their measurement method retained for comparison.""" + data = record.get("data") or {} + memory = data.get("memory") or {} + result = {} + for key in ("peak_rss_mib", "peak_vram_mib"): + value = memory.get(key) + if finite(value) and value >= 0: + result[key] = {"value": value, "unit": "MiB", "method": memory.get("method") or memory.get("source"), + "complete": memory.get("complete"), "verdict": memory.get("verdict"), + "limitations": memory.get("limitations", [])} + for metric in record.get("metrics") or []: + key, value = metric.get("name"), metric.get("value") + if key in ("peak_rss_mib", "peak_vram_mib") and key not in result and finite(value) and value >= 0: + result[key] = {"value": value, "unit": "MiB", "method": memory.get("method") or "recorded metric", + "complete": memory.get("complete"), "verdict": memory.get("verdict"), + "limitations": memory.get("limitations", [])} + return result + + +def round_coverage(record: dict, expected_rounds: int | None = None) -> dict: + rows = record.get("data", {}).get("rounds") or [] + oversized_rows = isinstance(rows, list) and len(rows) > MAX_ROUNDS + invalid_indices = not isinstance(rows, list) + rows = rows[:MAX_ROUNDS] if isinstance(rows, list) else [] + indices = [r.get("index") for r in rows if isinstance(r, dict)] + invalid_indices = invalid_indices or len(indices) != len(rows) or any( + type(i) is not int or not 0 <= i <= MAX_ROUND_INDEX for i in indices) + counts = Counter(i for i in indices if type(i) is int and 0 <= i <= MAX_ROUND_INDEX) + actual = sorted(counts) + plan = record.get("data", {}).get("round_plan") or {} + requested = plan.get("expected_indices") if isinstance(plan, dict) else None + invalid_plan = not isinstance(plan, dict) + oversized_plan = isinstance(requested, list) and (len(requested) > MAX_ROUNDS or any( + type(i) is int and i > MAX_ROUND_INDEX for i in requested[:MAX_ROUNDS])) + if requested is not None and (not isinstance(requested, list) or oversized_plan or any( + type(i) is not int or not 0 <= i <= MAX_ROUND_INDEX for i in requested)): + invalid_plan = True + requested = None + if requested is not None and (not requested or len(set(requested)) != len(requested)): + invalid_plan = True + requested = None + if expected_rounds is not None: + if type(expected_rounds) is not int or not 1 <= expected_rounds <= MAX_ROUNDS: + invalid_plan = True + oversized_plan = type(expected_rounds) is int and expected_rounds > MAX_ROUNDS + requested = None + else: + requested = list(range(expected_rounds)) + # Gaps are visible even when a manually chained run never recorded its intended size. + gaps = sorted(set(range(min(actual), max(actual) + 1)) - set(actual)) if actual else [] + missing = sorted(set(requested or []) - set(actual)) + unexpected = sorted(set(actual) - set(requested)) if requested is not None else [] + duplicates = [i for i in actual if counts[i] > 1] + failed = [r["index"] for r in rows if isinstance(r, dict) and r.get("n_failed") + and type(r.get("index")) is int and 0 <= r["index"] <= MAX_ROUND_INDEX] + invalid = invalid_indices or invalid_plan or oversized_rows + return {"actual_indices": actual, "expected_indices": requested, "missing_indices": missing, + "unexpected_indices": unexpected, "gaps": gaps, "duplicate_indices": duplicates, "failed_indices": failed, + "invalid_indices": invalid_indices, "invalid_plan": invalid_plan, "oversized_plan": oversized_plan, + "oversized_rows": oversized_rows, + "complete": False if invalid else not (missing or unexpected or gaps or duplicates or failed) + if requested is not None else None} + + +def finalize_arm_records(last_ev: dict, cells: list[dict], plan: dict, *, cpu: dict | None = None) -> dict: + """Save immutable enriched copies after every cell has stopped; never alter the raw client record. + + These records are final scheduler results, not inputs for a later manual --extend. + Their source_evidence remains available for inspecting the original client capture. + """ + store, final = ev.Store(), {} + expected = list(range(plan["rounds"])) + for label, reference in last_ev.items(): + source = store.load(reference) + if ev.integrity_problem(source): + raise OpError(f"{label}: client evidence integrity failed") + record = copy.deepcopy(source) + data = record["data"] + mine = [c for c in cells if c["label"] == label] + contexts = [{"index": c["round"] - 1, "status": c.get("status"), + "position": c.get("position"), "ready": c.get("ready"), "stop": c.get("stop"), + "resources": c.get("resources"), "server_env": c.get("server_env"), + "t_start": c.get("t_start"), "t_end": c.get("t_end")} for c in mine] + envs = [c.get("server_env") for c in mine if c.get("server_env") is not None] + stable_env = len(envs) == len(mine) and bool(envs) and all(e == envs[0] for e in envs) + data["source_evidence"] = source["evidence_id"] + data["scheduler_final"] = True + data["round_plan"] = {"expected_indices": expected, "requested_rounds": plan["rounds"], + "design": plan["design"]["design"], "balanced": plan["design"]["balanced"]} + data["runtime_rounds"] = contexts + cell_indices = [c["index"] for c in contexts] + source_rows = data.get("rounds") or [] + windows = {c["index"]: (c.get("t_start"), c.get("t_end")) for c in contexts} + timing_bound = len(source_rows) == len(expected) and all( + type(row.get("index")) is int and row["index"] in windows and + all(finite(v) for v in (*windows[row["index"]], row.get("t_start"), row.get("t_end"))) and + windows[row["index"]][0] <= row["t_start"] < row["t_end"] <= windows[row["index"]][1] + for row in source_rows) + valid = (sorted(cell_indices) == expected and timing_bound and stable_env and + all(c.get("status") == "ok" and (c.get("stop") or {}).get("group_gone") is True + and (c.get("ready") or {}).get("status") == "ready" for c in mine)) + data["lifecycle"] = {"source": "bench rounds", "fresh_process_per_cell": True, + "valid": bool(valid), "verdict": "PASS" if valid else "UNPROVEN", + "scope": "server launch/readiness/process-group cleanup, not sustained reliability"} + flags = record["identity"]["engine"].setdefault("flags", {}) + flags["server_env"] = envs[0] if stable_env else {"varied_between_rounds": envs} + if cpu is not None: + record["identity"]["cpu"] = cpu + record["identity"]["gpu"] = ev.identity()["gpu"] + by_index = {c["index"]: c for c in contexts} + peaks = [] + for row in data.get("rounds") or []: + context = by_index.get(row["index"], {}) + resource = context.get("resources") or {} + peak = resource.get("peak_rss_mib") + if finite(peak) and peak >= 0: + row["peak_rss_mib"] = peak + peaks.append(peak) + if peaks: + complete_memory = len(peaks) == len(expected) and all( + (c.get("resources") or {}).get("status") == "observed" for c in mine) + data["memory"] = {"peak_rss_mib": max(peaks), "method": "sampled_process_group_rss", + "verdict": "PASS" if complete_memory else "UNPROVEN", + "complete": complete_memory, "rounds_measured": len(peaks), + "limitations": ["Sampled RSS is a lower bound on the true peak and includes shared pages per process.", + "This is host RAM, not VRAM, swap, or sustained leak testing."]} + record["metrics"].append(ev.metric("peak_rss_mib", max(peaks), "MiB", "max", n=len(peaks))) + record["limitations"] = [s for s in record.get("limitations", []) + if not s.startswith("Rounds share one server process:") + and not (peaks and s.startswith("Memory, swap and sustained"))] + if peaks: + record["limitations"].append("Host RSS was sampled; swap, device memory and sustained memory growth were not measured.") + if data["memory"]["complete"]: + record["coverage"]["missing"] = [x for x in record["coverage"].get("missing", []) + if x != "external memory telemetry"] + record["limitations"].append("Scheduler execution context is recorded separately from unchanged client timing; CPU locks are advisory.") + if not stable_env: + record["limitations"].append("Server tuning environment was unavailable or changed between rounds.") + coverage = round_coverage(record) + data["round_coverage"] = coverage + problems = [] + if coverage["missing_indices"]: + problems.append(f"missing requested rounds: {coverage['missing_indices']}") + if not valid: + problems.append("complete fresh-process lifecycle") + record["coverage"]["missing"] = sorted(set(record["coverage"].get("missing", []) + problems)) + record["coverage"]["complete"] = not record["coverage"]["missing"] + old_root = store.run_dir(source["evidence_id"]).resolve() + with store.staging_dir() as staging: + for artifact in source.get("artifacts", []): + relative = Path(artifact["path"]) + path = (old_root / relative).resolve() + if relative.is_absolute() or not path.is_relative_to(old_root): + raise OpError("client artifact escapes its evidence directory") + if not path.is_file() or file_sha256(path)[0] != artifact["sha256"]: + raise OpError("client artifact missing or changed before scheduler finalization") + dest = staging / relative + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_bytes(path.read_bytes()) + proof = staging / "scheduler-context.json" + proof.write_text(json.dumps(ev.redact({"plan": data["round_plan"], "cells": contexts}), indent=2) + "\n") + record["artifacts"].append(ev.artifact(proof, staging, "scheduler_context")) + saved = store.save(record, staging) + final[label] = saved["evidence_id"] + return final diff --git a/src/inferencekit/serve/rounds.py b/src/inferencekit/serve/rounds.py index edd4186..a30942e 100644 --- a/src/inferencekit/serve/rounds.py +++ b/src/inferencekit/serve/rounds.py @@ -57,8 +57,9 @@ from inferencekit import evidence as ev from inferencekit import proc, stats +from inferencekit.cpuinfo import cpu_identity from inferencekit.registry import OpError -from inferencekit.serve import servers +from inferencekit.serve import resources, round_evidence, servers SERVE_CMD_ENV = "INFERENCEKIT_BENCH_SERVE_CMD" STATUSES_OK = ("ok",) @@ -542,6 +543,8 @@ def higher_is_better(metric: str) -> bool | None: latency (lower is better) although it mentions N*. """ m = base_metric(metric) + if m in ("peak_rss_mib", "peak_vram_mib", "rtf"): + return False if any(k in m for k in ("usd", "cost", "fail", "error")): return False if any(k in m for k in ("ttft", "tpot", "itl", "e2e", "latency")) or m.endswith(("_ms", "_seconds")): @@ -594,24 +597,30 @@ def run_cell(spec: dict, k: int, position: int, cfg: dict, prompts: dict) -> dic log_path.unlink() # a rerun of the same round must not append to an old log env, env_info = servers.server_env(spec["env"]) cell["env"] = env_info + cell["server_env"] = resources.runtime_env(env, cwd=spec.get("cwd")) + cpu = cfg.get("resource_mode") == "cpu" if servers.port_in_use(spec["host"], spec["port"]): cell.update(status="port_busy", detail=f"something already listens on {spec['host']}:{spec['port']}; " "refusing to benchmark it in place of the engine") cell["t_end"] = time.time() return cell - gpu_before = cfg["gpu_reader"]() - server = None + gpu_before = None if cpu else cfg["gpu_reader"]() + server = monitor = None try: try: server = servers.launch(spec["launch"], env, log_path, cwd=spec.get("cwd")) except OSError as e: cell.update(status="launch_failed", detail=str(e)[:300]) return cell + if cpu: + monitor = resources.ProcessMonitor(server.pgid, env, cwd=spec.get("cwd")).start() ready = servers.wait_ready(server, base + spec["ready_check"], cfg["startup_timeout_s"], cfg["poll_s"]) cell["ready"] = ready if ready["status"] != "ready": cell["status"] = "startup_timeout" if ready["status"] == "timeout" else "exited_early" return cell + if monitor is not None: + monitor.settled() model = servers.served_model(base) cell["served_model"] = model model = model or cfg.get("model_id") @@ -640,12 +649,14 @@ def run_cell(spec: dict, k: int, position: int, cfg: dict, prompts: dict) -> dic "source": sched["source"] if shared else "this engine's own warm-up"} argv = serve_argv(cfg["serve_cmd"], ctx, cfg["serve_args"]) child_env = dict(os.environ) + child_env.pop("INFERENCEKIT_GPU_LOCK_HELD" if cpu else "INFERENCEKIT_CPU_LOCK_HELD", None) # Same evidence store as this run; and tell a lock-aware client that the GPU lock is already held # (by this process) so it must not try to take it again. root = str(ev.default_root().resolve()) # The child's record is read from its stdout (parse_serve_output): full JSON, never the terse view. child_env.update({"INFERKIT_RUNS_DIR": root, "INFERENCEKIT_RUNS_DIR": root, - "INFERENCEKIT_GPU_LOCK_HELD": str(cfg.get("gpu_lock_path") or ""), + "INFERENCEKIT_CPU_LOCK_HELD" if cpu else "INFERENCEKIT_GPU_LOCK_HELD": + str(cfg.get("cpu_lock_path" if cpu else "gpu_lock_path") or ""), "INFERENCEKIT_OUTPUT": "json"}) t0 = time.monotonic() res = proc.run(argv, timeout=cfg["serve_timeout_s"], env=child_env) # a timeout kills its process group @@ -688,11 +699,16 @@ def run_cell(spec: dict, k: int, position: int, cfg: dict, prompts: dict) -> dic return cell finally: if server is not None: - cell["stop"] = servers.stop(server, cfg["stop_timeout_s"]) + try: + cell["stop"] = servers.stop(server, cfg["stop_timeout_s"]) + finally: + if monitor is not None: + attach_resources(cell, monitor.stop()) if not cell["stop"]["group_gone"] and cell["status"] == "ok": cell["status"] = "stop_failed" - cell["gpu_memory"] = servers.wait_gpu_memory(gpu_before, cfg["gpu_mem_tolerance_mib"], - cfg["gpu_mem_wait_s"], reader=cfg["gpu_reader"]) + if not cpu: + cell["gpu_memory"] = servers.wait_gpu_memory(gpu_before, cfg["gpu_mem_tolerance_mib"], + cfg["gpu_mem_wait_s"], reader=cfg["gpu_reader"]) resolved = servers.parse_resolved_config(spec["kind"], log_path.read_text(errors="replace") if log_path.exists() else "") rc_dir = run_dir / "resolved_config" @@ -709,10 +725,46 @@ def run_cell(spec: dict, k: int, position: int, cfg: dict, prompts: dict) -> dic cell["t_end"] = time.time() +def attach_resources(cell: dict, telemetry: dict) -> None: + """Keep the raw method/limitations and expose a per-round observed host-memory metric.""" + cell["resources"] = telemetry + peak = telemetry.get("peak_rss_mib") + if peak is not None and telemetry.get("status") == "observed": + cell.setdefault("metrics", {}).setdefault("all", {})["peak_rss_mib"] = { + "value": peak, "unit": "MiB", "stat": "sampled_peak", "n": telemetry.get("samples")} + + +def cleanup_failure(cell: dict) -> str | None: + """A failed cleanup invalidates the host for every later cell, not only this arm.""" + stop, memory = cell.get("stop") or {}, cell.get("gpu_memory") or {} + if stop.get("group_gone") is False: + return "server processes survived cleanup" + if memory.get("checked") is True and memory.get("returned") is not True: + return "GPU memory did not return to its pre-launch level" + return None + + +def skipped_cleanup_cell(spec: dict, k: int, position: int, halted: dict) -> dict: + now = time.time() + return {"round": k, "position": position, "label": spec["label"], "kind": spec["kind"], + "status": "skipped_after_cleanup_failure", "t_start": now, "t_end": now, + "detail": f"cleanup failed after {halted['label']} r{halted['round']}: {halted['detail']}"} + + +def finalized_evidence(last_ev: dict, cells: list[dict], plan: dict) -> dict: + """Custom clients may print a record without storing it; retain that limitation explicitly.""" + store = ev.Store() + stored = {label: ref for label, ref in last_ev.items() if (store.run_dir(ref) / "evidence.json").is_file()} + absent = sorted(set(last_ev) - set(stored)) + if absent: + plan["unfinalized_evidence"] = absent + return {**last_ev, **round_evidence.finalize_arm_records(stored, cells, plan, cpu=plan.get("cpu"))} + + # --------------------------------------------------------------------------- summary def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAULT_RESAMPLES, - seed: int = 0, mode: str = "single") -> dict: + seed: int = 0, mode: str = "single", expected_rounds: int | None = None) -> dict: """Medians over rounds with bootstrap CIs, and ratios to the best engine with paired CIs. An engine with any failed request in a bucket (over all its rounds) is not eligible as @@ -723,6 +775,9 @@ def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAU values: dict[tuple[str, str, str], dict[int, float]] = {} units: dict[str, str | None] = {} requests: dict[tuple[str, str], float] = {} + expected = set(range(1, expected_rounds + 1)) if expected_rounds is not None else {c["round"] for c in cells} + completed = {label: {c["round"] for c in cells if c["label"] == label and c.get("status") in STATUSES_OK} + for label in labels} for c in cells: if c.get("status") not in STATUSES_OK: continue @@ -747,7 +802,8 @@ def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAU per[label] = {"median": statistics.median(xs), "ci95": list(ci), "n": len(xs), "by_round": v} if not per: continue - eligible = [lb for lb in per if not failures.get((lb, bucket))] + eligible = [lb for lb in per if not failures.get((lb, bucket)) and + set(per[lb]["by_round"]) == expected and completed[lb] == expected] best = None if direction is not None and eligible: pick = max if direction else min @@ -758,11 +814,16 @@ def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAU "higher_is_better": direction, "median": info["median"], "ci95": info["ci95"], "n": info["n"], "best": best, "ratio_to_best": None, "ratio_ci95": [None, None], "n_pairs": 0, "failed_requests": failed} + absent = sorted(expected - set(info["by_round"])) + row["missing_rounds"] = absent if name in COUNT_METRICS: row["total"] = sum(info["by_round"].values()) if failed and direction is not None: row["excluded_from_best"] = (f"{failed:g} failed request(s) over {info['n']} round(s): latencies " "cover only the answered requests") + elif (absent or completed[label] != expected) and direction is not None: + row["excluded_from_best"] = ("incomplete successful rounds: missing " + f"{absent or sorted(expected - completed[label])}") elif best is not None: b, c = per[best]["by_round"], info["by_round"] paired = sorted(set(b) & set(c)) @@ -782,12 +843,16 @@ def summarize(cells: list[dict], labels: list[str], resamples: int = stats.DEFAU else: median_def = ("median over rounds of the per-round value (median over that round's requests in the " "bucket, from bench serve's requests artifact) (observed)") - return {"rows": rows, "failures": fail_list, "definitions": { + return {"rows": rows, "failures": fail_list, + "incomplete_rounds": [{"engine": label, "missing_rounds": sorted(expected - completed[label])} + for label in labels if completed[label] != expected], "definitions": { "median": median_def, + "peak_rss_mib": "median over rounds of sampled peak server-process-group RSS (host RAM, observed); " + "sampling can miss peaks and shared pages can be counted per process", "ci95": f"bootstrap 95% CI of that median, rounds resampled ({'derived' if mode == 'load' else 'observed'})", "ratio_to_best": "median(engine) / median(best engine) over rounds both completed, with a paired " "bootstrap 95% CI resampling rounds (derived); best = lowest for latencies, highest " - "for rates, among engines with no failed request in the bucket", + "for rates, among engines with complete successful rounds and no failed request in the bucket", "failed_requests": "requests that failed in a round (median over rounds; 'total' sums them); an engine " "with any failure in a bucket is never best there and gets no ratio"}} @@ -907,6 +972,9 @@ def markdown(summary: dict, plan: dict, cells: list[dict], resolved: dict) -> st f"Arrival schedule: {same}{ref}; {sched.get('source') or 'unknown'}.", ""] else: lines += ["Medians and CIs are [observed] (client-side timer in `bench serve`); ratios are [derived].", ""] + if any(r["metric"] == "peak_rss_mib" for r in summary["rows"]): + lines += ["peak_rss_mib is observed host RAM from server-process-group samples across launch, requests and " + "cleanup; sampling can miss peaks and count shared pages per process. It is not client timing or VRAM.", ""] lines += ["An engine marked [FAILURES] had failed requests in that bucket: its latencies cover only the " "requests it answered, so it is never named best there and gets no ratio.", "", f"| engine | bucket | metric | median [{cls}] | 95% CI | rounds | ratio to best [derived] | " @@ -927,6 +995,12 @@ def markdown(summary: dict, plan: dict, cells: list[dict], resolved: dict) -> st f"{ratio_ci} |") if not summary["rows"]: lines.append("| (no successful rounds) | | | | | | | |") + if summary.get("incomplete_rounds"): + lines += ["", "## Incomplete rounds", "", + "These engines retain their observed values but are excluded from ranking.", "", + "| engine | missing successful rounds |", "|---|---|"] + for item in summary["incomplete_rounds"]: + lines.append(f"| {item['engine']} | {', '.join(map(str, item['missing_rounds']))} |") if summary.get("failures"): lines += ["", "## Failed requests", "", "| engine | bucket | failed | requests | rounds |", "|---|---|---|---|---|"] @@ -984,7 +1058,8 @@ def run_rounds(manifest_path: str, prompts_path: str, rounds: int | None, mode: gpu_mem_tolerance_mib: int = 512, gpu_mem_wait_s: float = 60, poll_s: float = 0.5, gpu_reader=None, ref_e2e_ms: float | None = None, chat_template_kwargs: str | dict | None = None, ci_target: float | None = None, min_rounds: int | None = None, - extra_body: str | dict | None = None) -> dict: + extra_body: str | dict | None = None, resource_mode: str = "gpu", + cpu_lock_path: str | None = None) -> dict: """Run every (round, engine) cell under the GPU lock; return plan, cells, summary and paths. ``rounds`` None: the smallest multiple of the Williams period that is at least 3. @@ -998,6 +1073,9 @@ def run_rounds(manifest_path: str, prompts_path: str, rounds: int | None, mode: if mode not in ("single", "load"): raise OpError("mode must be 'single' or 'load'") manifest = load_manifest(manifest_path) + resource_mode = round_evidence.resource_mode(manifest, resource_mode) + if resource_mode == "cpu" and mode == "load": + raise OpError("CPU load mode is not supported: capacity metrics currently use GPU normalization; use --mode single") specs = engine_specs(manifest, engines) if not specs: raise OpError("no engines selected") @@ -1042,6 +1120,9 @@ def run_rounds(manifest_path: str, prompts_path: str, rounds: int | None, mode: "serve_args": serve_args, "serve_cmd_custom": bool(serve_cmd or os.environ.get(SERVE_CMD_ENV)), "warmup_kind": servers.WARMUP_KIND}, "manifest_sha256": hashlib.sha256(Path(manifest_path).read_bytes()).hexdigest()} + plan["resource_mode"] = resource_mode + if resource_mode == "cpu": + plan["cpu"] = cpu_identity() if mode == "load": plan["load_schedule"] = schedule plan["chat_template_kwargs"] = {"value": ctk or None, "source": ctk_source} @@ -1060,21 +1141,35 @@ def run_rounds(manifest_path: str, prompts_path: str, rounds: int | None, mode: "model_id": (manifest.get("model") or {}).get("id"), "last_ev": {}, "ref_e2e_ms": ref_e2e_ms, "schedule": schedule, "chat_template_kwargs": ctk or None, "extra_body": body or None} + cfg["resource_mode"] = resource_mode out_dir = rdir / "rounds" out_dir.mkdir(parents=True, exist_ok=True) cells: list[dict] = [] - with servers.gpu_lock(gpu_lock_path, owner=f"inferencekit bench rounds {rdir.name}") as lock: - plan["gpu_lock"] = lock["path"] - cfg["gpu_lock_path"] = lock["path"] + lock_context = (servers.gpu_lock(cpu_lock_path, owner=f"inferencekit bench rounds {rdir.name}", resource="CPU") + if resource_mode == "cpu" else + servers.gpu_lock(gpu_lock_path, owner=f"inferencekit bench rounds {rdir.name}")) + with lock_context as lock: + plan[f"{resource_mode}_lock"] = lock["path"] + cfg[f"{resource_mode}_lock_path"] = lock["path"] plan["started"] = time.strftime("%Y-%m-%dT%H:%M:%S%z") (out_dir / "plan.json").write_text(json.dumps(ev.redact(plan), indent=2) + "\n") by_label = {s["label"]: s for s in specs} + halted = None for k, order in enumerate(design["orders"], 1): for pos, label in enumerate(order): - cell = run_cell(by_label[label], k, pos, cfg, prompts) + cell = (skipped_cleanup_cell(by_label[label], k, pos, halted) if halted else + run_cell(by_label[label], k, pos, cfg, prompts)) + failure = cleanup_failure(cell) + if failure: + halted = {"label": label, "round": k, "detail": failure} + plan["aborted_after_cleanup_failure"] = halted + cell["cleanup_failure"] = failure + if cell.get("status") == "ok": + cell["status"] = ("stop_failed" if (cell.get("stop") or {}).get("group_gone") is False else + "gpu_memory_not_released") cells.append(cell) (out_dir / f"{label}.r{k}.json").write_text(json.dumps(ev.redact(cell), indent=2) + "\n") - if adaptive is None: + if adaptive is None or halted: continue whole = k % adaptive["period"] == 0 and k >= adaptive["first_check_round"] if whole or k == rounds: # the last round is checked too, to record whether the target was met @@ -1099,10 +1194,10 @@ def run_rounds(manifest_path: str, prompts_path: str, rounds: int | None, mode: schedule["source"] = ("no reference latency reported by bench serve: each engine's concurrency levels " "were turned into rates with its own warm-up e2e") plan["load_schedule"] = schedule + plan["serve_evidence_final"] = finalized_evidence(cfg["last_ev"], cells, plan) (out_dir / "plan.json").write_text(json.dumps(ev.redact(plan), indent=2) + "\n") - summary = summarize(cells, labels, mode=mode) + summary = summarize(cells, labels, mode=mode, expected_rounds=plan["rounds"]) resolved = resolved_summary(cells, labels) - plan["serve_evidence_final"] = dict(cfg["last_ev"]) # single mode: each engine's record with all its rounds (out_dir / "summary.json").write_text(json.dumps(ev.redact({"plan": plan, "summary": summary, "resolved_config": resolved}), indent=2) + "\n") md = markdown(summary, plan, cells, resolved) diff --git a/src/inferencekit/serve/servers.py b/src/inferencekit/serve/servers.py index f47f320..8cfe660 100644 --- a/src/inferencekit/serve/servers.py +++ b/src/inferencekit/serve/servers.py @@ -420,22 +420,24 @@ def wait_gpu_memory(baseline: list[int] | None, tolerance_mib: int, timeout_s: f # --------------------------------------------------------------------------- GPU lock -def default_lock_path() -> Path: - env = os.environ.get("INFERENCEKIT_GPU_LOCK") +def default_lock_path(resource: str = "GPU") -> Path: + env = os.environ.get(f"INFERENCEKIT_{resource}_LOCK") if env: return Path(env) - return Path(os.path.expanduser("~")) / ".cache" / "inferencekit" / "gpu.lock" + return Path(os.path.expanduser("~")) / ".cache" / "inferencekit" / f"{resource.lower()}.lock" @contextlib.contextmanager -def gpu_lock(path: str | os.PathLike | None, owner: str): +def gpu_lock(path: str | os.PathLike | None, owner: str, *, resource: str = "GPU"): """Hold an exclusive, non-blocking ``flock`` on ``path`` for the duration of the block. The kernel drops the lock if this process dies, so a crashed run never leaves a stale lock behind. The file holds the current holder's pid and owner string, for the error message another run prints. """ - p = Path(path) if path else default_lock_path() + if resource not in ("GPU", "CPU"): + raise ValueError("resource must be GPU or CPU") + p = Path(path) if path else default_lock_path(resource) p.parent.mkdir(parents=True, exist_ok=True) fd = os.open(p, os.O_RDWR | os.O_CREAT, 0o644) try: @@ -445,8 +447,10 @@ def gpu_lock(path: str | os.PathLike | None, owner: str): holder = os.pread(fd, 4096, 0).decode(errors="replace").strip() or "unknown holder" finally: os.close(fd) - raise OpError(f"GPU lock {p} is held by {holder}: another run is using the GPU. Wait for it, or pass " - "a different --gpu-lock only if the runs use different GPUs.") from None + advice = ("a different --gpu-lock only if the runs use different GPUs." if resource == "GPU" else + "a different --cpu-lock only if the runs use independent CPU resources.") + raise OpError(f"{resource} lock {p} is held by {holder}: another run is using the {resource}. Wait for it, " + f"or pass {advice}") from None info = {"pid": os.getpid(), "owner": owner, "since": time.strftime("%Y-%m-%dT%H:%M:%S%z")} try: os.ftruncate(fd, 0) diff --git a/tests/fakes_rounds/fake_media_server.py b/tests/fakes_rounds/fake_media_server.py new file mode 100644 index 0000000..9a21166 --- /dev/null +++ b/tests/fakes_rounds/fake_media_server.py @@ -0,0 +1,68 @@ +"""Synthetic fresh-process media endpoint; no model or accelerator dependencies.""" +import argparse +import base64 +import hashlib +import json +import os +import struct +import time +import zlib +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + +def png(width, height, seed): + def chunk(kind, value): + return struct.pack(">I", len(value)) + kind + value + struct.pack(">I", zlib.crc32(kind + value) & 0xffffffff) + color = hashlib.sha256(str(seed).encode()).digest()[:3] + raw = (b"\x00" + color * width) * height + return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)) + + chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b"")) + + +parser = argparse.ArgumentParser() +parser.add_argument("--port", type=int, required=True) +parser.add_argument("--mode", choices=("audio", "image"), required=True) +parser.add_argument("--fail", action="store_true") +args = parser.parse_args() + + +class Handler(BaseHTTPRequestHandler): + def log_message(self, *_): + pass + + def do_GET(self): + self.send_response(200) + self.end_headers() + self.wfile.write(b"ready") + + def do_POST(self): + body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + if args.fail: + self.send_response(500) + self.end_headers() + return + time.sleep(0.01) + self.send_response(200) + if args.mode == "audio": + self.send_header("Content-Type", "application/x-ndjson") + self.end_headers() + pcm = struct.pack(" 0 and cell["stop"]["group_gone"] + final = store.load(rec["data"]["plan"]["clip_evidence_final"]["cpu-video"]) + assert final["data"]["rounds"][0]["peak_rss_mib"] > 0 + assert final["identity"]["cpu"] == rec["identity"]["cpu"] + assert not any("gpu identity" in item for item in final["coverage"]["missing"]) + assert final["identity"]["engine"]["flags"]["server_env"]["controls"]["OMP_NUM_THREADS"] == "2" + + def test_bench_rounds_mode_clip_runs_clip_rounds_with_the_default_bench_clip(mods, store, tmp_path, monkeypatch): """bench rounds --mode clip hands the run to serve/clip_rounds.py, and each cell runs the registered 'inferencekit bench clip' (no command template).""" diff --git a/tests/test_bench_rounds.py b/tests/test_bench_rounds.py index 1c3876a..e60dde2 100644 --- a/tests/test_bench_rounds.py +++ b/tests/test_bench_rounds.py @@ -181,6 +181,19 @@ def test_gpu_lock_is_exclusive(tmp_path): pass +def test_cpu_lock_uses_its_own_default_and_is_exclusive(tmp_path, monkeypatch): + cpu, gpu = tmp_path / "cpu.lock", tmp_path / "gpu.lock" + monkeypatch.setenv("INFERENCEKIT_CPU_LOCK", str(cpu)) + monkeypatch.setenv("INFERENCEKIT_GPU_LOCK", str(gpu)) + with servers.gpu_lock(None, "cpu-first", resource="CPU") as held: + assert held["path"] == str(cpu) + with pytest.raises(OpError, match="CPU lock"): + with servers.gpu_lock(None, "cpu-second", resource="CPU"): + pass + with servers.gpu_lock(None, "independent-gpu") as held_gpu: + assert held_gpu["path"] == str(gpu) + + def test_gpu_memory_wait_detects_a_leak(): readings = iter([[9000], [5000], [1100]]) ok = servers.wait_gpu_memory([1000], 256, 5, reader=lambda: next(readings)) @@ -443,8 +456,73 @@ def test_a_cell_whose_requests_all_failed_is_serve_failed(): assert rr.cell_status(0, {"x": 1}, {})[0] == "serve_no_metrics" +def test_missing_or_failed_cells_never_make_an_engine_best(): + cells = [_rows_cell(label, k, 1 if label == "incomplete" else 100, 0) + for label in ("complete", "incomplete") for k in (1, 2)] + cells[-1] = {"label": "incomplete", "round": 2, "status": "startup_timeout"} + summary = rr.summarize(cells, ["complete", "incomplete"], resamples=100, expected_rounds=2) + row = next(r for r in summary["rows"] if r["engine"] == "incomplete" and r["metric"] == "ttft_ms") + assert row["median"] == 1 and row["best"] == "complete" and row["ratio_to_best"] is None + assert row["missing_rounds"] == [2] and "incomplete" in row["excluded_from_best"] + assert summary["incomplete_rounds"] == [{"engine": "incomplete", "missing_rounds": [2]}] + assert rr.higher_is_better("rtf") is False + + +def test_cpu_load_is_refused_before_any_launch(tmp_path): + manifest, prompts = write_run(tmp_path, [fake_engine("a", free_port())]) + with pytest.raises(OpError, match="CPU load mode.*single"): + rr.run_rounds(str(manifest), str(prompts), 2, "load", resource_mode="cpu") + + +@pytest.mark.parametrize("cleanup", [ + {"stop": {"group_gone": False}, "status": "startup_timeout"}, + {"stop": {"group_gone": True}, "gpu_memory": {"checked": True, "returned": False}, "status": "ok"}, +]) +def test_cleanup_failure_skips_every_remaining_text_cell(tmp_path, monkeypatch, cleanup): + manifest, prompts = write_run(tmp_path, [fake_engine("a", free_port()), fake_engine("b", free_port())]) + launched = [] + + def fake_cell(spec, k, position, cfg, prompts): + launched.append(spec["label"]) + return {"round": k, "position": position, "label": spec["label"], "kind": "generic", **cleanup} + + monkeypatch.setattr(rr, "run_cell", fake_cell) + result = rr.run_rounds(str(manifest), str(prompts), 4, "single", gpu_lock_path=str(tmp_path / "lock")) + assert launched == ["a"] + assert len(result["cells"]) == 8 + assert all(c["status"] == "skipped_after_cleanup_failure" for c in result["cells"][1:]) + assert result["plan"]["aborted_after_cleanup_failure"]["label"] == "a" + assert result["plan"]["rounds"] == 4 + assert result["cells"][0]["status"] != "ok" + assert "Incomplete rounds" in result["markdown"] + + # --------------------------------------------------------------------------- end to end + +@linux_only +def test_cpu_rounds_measure_server_group_without_gpu_probes(tmp_path, store, monkeypatch): + engines = [fake_engine("a", free_port(), "--ttft-ms", "5", env={"OMP_NUM_THREADS": "2"}), + fake_engine("b", free_port(), "--ttft-ms", "10", env={"OMP_NUM_THREADS": "4"})] + manifest, prompts = write_run(tmp_path, engines) + monkeypatch.setattr(rr, "cpu_identity", lambda: {"model": "Test CPU", "architecture": "test"}) + result = rr.run_rounds(str(manifest), str(prompts), 2, "single", resource_mode="cpu", + cpu_lock_path=str(tmp_path / "cpu.lock"), startup_timeout_s=20, stop_timeout_s=3, + warmup=0, poll_s=.05, serve_cmd=f"{sys.executable} {STUB_SERVE}", + gpu_reader=lambda: pytest.fail("CPU runs must not read GPU memory")) + assert result["plan"]["resource_mode"] == "cpu" and "gpu_lock" not in result["plan"] + assert result["plan"]["cpu"]["model"] == "Test CPU" + assert result["plan"]["unfinalized_evidence"] == ["a", "b"] # deliberately unstored test-client records + for cell in result["cells"]: + assert cell["status"] == "ok" and cell["stop"]["group_gone"] + assert "gpu_memory" not in cell + assert cell["resources"]["peak_rss_mib"] > 0 + assert cell["resources"]["peak_threads"] >= 1 + assert cell["resources"]["observations"]["settled"]["processes"] >= 1 + assert cell["server_env"]["controls"]["OMP_NUM_THREADS"] == ("2" if cell["label"] == "a" else "4") + memory = [r for r in result["summary"]["rows"] if r["metric"] == "peak_rss_mib"] + assert len(memory) == 2 and all(r["higher_is_better"] is False and r["n"] == 2 for r in memory) + def test_end_to_end_two_engines_via_cli(tmp_path, capsys, monkeypatch): p_fast, p_slow = free_port(), free_port() env_dump = tmp_path / "env.json" diff --git a/tests/test_bench_serve.py b/tests/test_bench_serve.py index f83ac29..4b4b73b 100644 --- a/tests/test_bench_serve.py +++ b/tests/test_bench_serve.py @@ -197,6 +197,12 @@ def test_single_mode_record(store, tmp_path): assert metric(rec, "e2e_ms", "p95")["value"] >= metric(rec, "e2e_ms")["value"] assert len(rec["data"]["rounds"]) == 2 and {"decode_tok_s", "t_start", "total_ms"} <= set(rec["data"]["rounds"][0]) assert rec["data"]["token_count_methods"] == {"stream_token_ids:choice_token_ids": 8} + accounting = rec["data"]["request_accounting"] + assert accounting == {"timed_requests": 8, "successful_requests": 8, "failed_requests": 0, + "selected_token_counts": 8, "unknown_token_counts": 0, "potentially_estimated_counts": 0, + "selected_output_tokens_total": 128, "selected_output_tokens_min": 16, + "selected_output_tokens_max": 16, "successful_count_mismatches": 0} + assert registry.call("inspect_evidence", {"id": rec["evidence_id"], "view": "summary"})["request_accounting"] == accounting assert rec["data"]["request_profile"]["asked"]["exact_length"] == ["ignore_eos", "min_tokens"] # outputs: text, token ids, finish reason, usage, timings for every prompt out = json.loads((tmp_path / "run/outputs/fake-vllm/short/short-0.json").read_text()) @@ -391,3 +397,32 @@ def test_cli_parses_bench_serve(): "p.jsonl", "--mode", "load", "--concurrency", "1", "--concurrency", "8", "--slo-ttft-p95-ms", "500", "--gpu-price-usd-h", "0.6"]) assert ns._op == "bench_serve" and ns.concurrency == [1, 8] and ns.slo_ttft_p95_ms == 500.0 + + +def test_request_accounting_does_not_count_failed_or_unknown_tokens(): + from inferencekit.ops.bench_serve import _request_accounting + + rows = [{"ok": True, "n_tokens": 8, "token_count_method": "usage", "count_mismatch": None}, + {"ok": True, "n_tokens": None, "count_mismatch": None}, + {"ok": True, "n_tokens": 4, "token_count_method": "min(stream_token_ids:choice_token_ids,usage)", "count_mismatch": "ids differ from usage"}, + {"ok": False, "n_tokens": 30, "count_mismatch": None}] + assert _request_accounting(rows) == { + "timed_requests": 4, "successful_requests": 3, "failed_requests": 1, + "selected_token_counts": 2, "unknown_token_counts": 1, "potentially_estimated_counts": 1, + "selected_output_tokens_total": 12, "selected_output_tokens_min": 4, + "selected_output_tokens_max": 8, "successful_count_mismatches": 1} + assert _request_accounting([{"ok": True, "n_tokens": None}])["selected_output_tokens_total"] is None + + +def test_sse_event_accounting_is_labeled_estimated(store, tmp_path): + prompts = write_prompts(tmp_path / "dev.jsonl", buckets=("short",), per_bucket=2, max_tokens=8) + with FakeServer(FakeConfig(ids="none", usage=False, tokens_per_event=4, ttft_s=0.001, itl_s=0.001)) as srv: + rec = run_single(srv, prompts, rounds=1, warmup=0) + accounting = rec["data"]["request_accounting"] + assert accounting["successful_requests"] == 2 + assert accounting["potentially_estimated_counts"] == 2 + assert accounting["selected_output_tokens_total"] < 16 + assert rec["data"]["token_count_methods"] == {"sse_events": 2} + view = registry.call("inspect_evidence", {"id": rec["evidence_id"], "view": "summary"}) + assert view["request_accounting"] == accounting + assert any("SSE events" in limitation for limitation in view["limitations"]) diff --git a/tests/test_bench_serve_evidence.py b/tests/test_bench_serve_evidence.py index 15f74ca..d612c5e 100644 --- a/tests/test_bench_serve_evidence.py +++ b/tests/test_bench_serve_evidence.py @@ -87,3 +87,14 @@ def test_schema_files_are_valid_json_schema(name): import jsonschema jsonschema.Draft202012Validator.check_schema(json.loads((REPO / "schemas" / name).read_text())) + + +def test_local_cpu_scheduler_identity_does_not_probe_gpu(monkeypatch): + from inferencekit.ops import bench_serve + from inferencekit.serve.client import Endpoint + monkeypatch.setenv("INFERENCEKIT_CPU_LOCK_HELD", "/test/cpu.lock") + monkeypatch.setattr(bench_serve, "gpu_identity", lambda: pytest.fail("CPU child probed GPU identity")) + monkeypatch.setattr(bench_serve, "cpu_identity", lambda: {"model": "test CPU", "architecture": "test"}) + identity, missing = bench_serve._identity(Endpoint.parse("http://127.0.0.1:1"), None, 1, "custom", "v1", "m", {}) + assert identity["cpu"]["model"] == "test CPU" and identity["gpu"]["name"] is None + assert missing == [] diff --git a/tests/test_cli_terse.py b/tests/test_cli_terse.py index 7ea8bd9..59ef259 100644 --- a/tests/test_cli_terse.py +++ b/tests/test_cli_terse.py @@ -151,7 +151,8 @@ def test_every_command_accepts_json_after_it(): required = [a for a in op.args if a.required and not (a.name == "argv" and op.passthrough)] argv = list(op.cli) for a in required: - value = a.choices[0] if a.choices else "1" if a.type in ("integer", "number") else "x" + scalar_type = a.items if a.type == "array" else a.type + value = a.choices[0] if a.choices else "1" if scalar_type in ("integer", "number") else "x" argv += [value] if a.positional else [cli._flag(a.name), value] ns = parser.parse_args(argv + ["--json"]) assert ns.json_output is True, op.cli diff --git a/tests/test_compare_coverage.py b/tests/test_compare_coverage.py new file mode 100644 index 0000000..976d334 --- /dev/null +++ b/tests/test_compare_coverage.py @@ -0,0 +1,216 @@ +"""Incomplete rounds cannot disappear through pair intersection; memory stays per arm.""" +import copy + +import pytest + +from inferencekit import evidence as ev, registry +from inferencekit.registry import OpError +from inferencekit.serve.round_evidence import MAX_ROUNDS, round_coverage +from conftest import make_benchmark +from test_compare_media import media_record + + +def compare(a, b, **kwargs): + return registry.call("compare", {"baseline": a["evidence_id"], "candidate": b["evidence_id"], + "resamples": 100, **kwargs}) + + +def arms(store): + return (make_benchmark(store, [100] * 4, t0=0), make_benchmark(store, [200] * 4, t0=5)) + + +@pytest.mark.parametrize("mutate,reason", [ + (lambda rows: rows.pop(), "unmatched metric rounds"), + (lambda rows: rows.pop(1), "gaps in observed rounds"), + (lambda rows: rows.append(copy.deepcopy(rows[0])), "duplicate round indices"), + (lambda rows: rows[1].pop("decode_tok_s"), "missing finite decode_tok_s"), + (lambda rows: rows[1].update(index="1"), "invalid round indices"), +]) +def test_partial_or_duplicate_metric_rounds_cannot_win(store, mutate, reason): + a, b = arms(store) + mutate(b["data"]["rounds"]) + b = store.save(b) + result = compare(a, b) + assert result["data"]["verdict"] == "inconclusive" + assert reason in result["data"]["decision"]["reason"] + + +def test_matching_truncated_tails_need_a_plan_or_explicit_count(store): + a, b = arms(store) + observed = compare(a, b) + assert observed["data"]["verdict"] == "improved" + assert observed["data"]["arm_round_coverage"]["baseline"]["complete"] is None + assert any("planned round count unknown" in x for x in observed["limitations"]) + checked = compare(a, b, expected_rounds=5) + assert checked["data"]["verdict"] == "inconclusive" + assert checked["data"]["arm_round_coverage"]["baseline"]["missing_indices"] == [4] + a["data"]["round_plan"] = {"expected_indices": [0, 1, 2, 3, 4]} + planned = compare(store.save(a), b) + assert planned["data"]["verdict"] == "inconclusive" + + +def test_explicit_count_cannot_weaken_a_recorded_plan(store): + a, b = arms(store) + a["data"]["round_plan"] = {"expected_indices": [0, 1, 2, 3, 4]} + result = compare(store.save(a), b, expected_rounds=4) + assert result["data"]["verdict"] == "inconclusive" + assert "conflicts with its recorded round plan" in result["data"]["decision"]["reason"] + with pytest.raises(OpError, match="positive"): + compare(a, b, expected_rounds=0) + + +@pytest.mark.parametrize("recorded", [False, True]) +def test_rounds_outside_the_expected_plan_are_inconclusive(store, recorded): + a, b = arms(store) + if recorded: + for rec in (a, b): + rec["data"]["round_plan"] = {"expected_indices": [0, 1, 2]} + a, b = store.save(a), store.save(b) + result = compare(a, b, **({} if recorded else {"expected_rounds": 3})) + assert result["data"]["verdict"] == "inconclusive" + assert "rounds outside the requested plan" in result["data"]["decision"]["reason"] + assert result["data"]["arm_round_coverage"]["baseline"]["unexpected_indices"] == [3] + assert result["data"]["arm_round_coverage"]["baseline"]["complete"] is False + + +@pytest.mark.parametrize("mutate,flag", [ + (lambda data: data["rounds"][0].update(index=10**30), "invalid_indices"), + (lambda data: data.update(round_plan={"expected_indices": [10**30]}), "oversized_plan"), + (lambda data: data.update(round_plan={"expected_indices": [0, 0]}), "invalid_plan"), + (lambda data: data.update(round_plan=[0, 1]), "invalid_plan"), +]) +def test_unbounded_or_invalid_indices_and_plans_are_refused_safely(store, mutate, flag): + a, b = arms(store) + mutate(b["data"]) + result = compare(a, store.save(b)) + assert result["data"]["verdict"] == "inconclusive" + assert result["data"]["arm_round_coverage"]["candidate"][flag] is True + + +def test_huge_expected_count_is_bounded_before_allocation(store): + a, b = arms(store) + with pytest.raises(OpError, match="expected_rounds"): + compare(a, b, expected_rounds=10**30) + coverage = round_coverage(a, expected_rounds=10**30) + assert coverage["oversized_plan"] is True and coverage["expected_indices"] is None + excessive = {"data": {"rounds": [{"index": 0}] * (MAX_ROUNDS + 1)}} + coverage = round_coverage(excessive) + assert coverage["oversized_rows"] is True and coverage["complete"] is False + + +def test_memory_measurement_and_method_retained_per_arm(store): + a, b = arms(store) + a["data"]["memory"] = {"peak_rss_mib": 80, "method": "sampled_process_group_rss"} + b["metrics"].append(ev.metric("peak_rss_mib", 120, "MiB", "max")) + result = compare(store.save(a), store.save(b)) + assert result["data"]["verdict"] == "improved" + assert result["data"]["memory"]["baseline"]["peak_rss_mib"]["method"] == "sampled_process_group_rss" + assert result["data"]["memory"]["candidate"]["peak_rss_mib"]["value"] == 120 + assert "peak RSS baseline 80.0 MiB (sampled, coverage unknown)" in result["summary"] + assert "candidate 120.0 MiB (coverage unknown)" in result["summary"] + assert {m["subject"] for m in result["metrics"] if m["name"] == "peak_rss_mib"} == {"baseline", "candidate"} + assert not any(m["name"] == "peak_rss_mib_ratio" for m in result["metrics"]) + + +def test_partial_memory_remains_qualified_without_invalidating_timing(store): + a, b = arms(store) + a["data"]["memory"] = {"peak_rss_mib": 80, "method": "sampled_process_group_rss", "complete": False, + "verdict": "UNPROVEN", "limitations": ["One requested round has no RSS samples."]} + result = compare(store.save(a), b) + assert result["data"]["verdict"] == "improved" + memory = result["data"]["memory"]["baseline"]["peak_rss_mib"] + assert memory["complete"] is False and memory["verdict"] == "UNPROVEN" + assert "80.0 MiB (sampled, partial)" in result["summary"] and "candidate unknown" in result["summary"] + assert any("One requested round has no RSS samples" in line for line in result["limitations"]) + assert any("lower bound on the true peak" in line for line in result["limitations"]) + + +def test_mixed_server_tuning_within_an_arm_is_inconclusive(store): + a, b = arms(store) + b["identity"]["engine"]["flags"]["server_env"] = {"varied_between_rounds": [ + {"controls": {"OMP_NUM_THREADS": "1"}}, {"controls": {"OMP_NUM_THREADS": "2"}}]} + result = compare(a, store.save(b)) + assert result["data"]["verdict"] == "inconclusive" + assert "server tuning environment varied" in result["data"]["decision"]["reason"] + + +def test_different_stable_tuning_between_arms_can_improve(store): + a, b = arms(store) + a["identity"]["engine"]["flags"]["server_env"] = {"controls": {"OMP_NUM_THREADS": "1"}} + b["identity"]["engine"]["flags"]["server_env"] = {"controls": {"OMP_NUM_THREADS": "2"}} + a["identity"]["engine"]["flags"]["private_path"] = "/private/unused" + b["identity"]["engine"]["flags"]["server_env"]["ld_preload"] = [ + {"name": "libmkl.so", "sha256": "a" * 64, "status": "hashed"}] + result = compare(store.save(a), store.save(b)) + assert result["data"]["verdict"] == "improved" + assert result["data"]["arm_environment"]["baseline"]["controls"]["OMP_NUM_THREADS"] == "1" + assert result["data"]["arm_environment"]["candidate"]["controls"]["OMP_NUM_THREADS"] == "2" + assert "private_path" not in result["data"]["arm_environment"]["baseline"] + assert "server env differs: OMP_NUM_THREADS, LD_PRELOAD" in result["summary"] + assert "/private/unused" not in result["summary"] and "libmkl.so" not in result["summary"] + + +def test_unknown_requested_controls_and_preload_are_visible_limitations(store): + a, b = arms(store) + b["identity"]["engine"]["flags"]["server_env"] = {"controls": {"OMP_NUM_THREADS": None}, + "omitted": ["OMP_NUM_THREADS"], "ld_preload": [{"name": "libfoo.so", "sha256": None, "status": "unavailable"}]} + result = compare(a, store.save(b)) + assert result["data"]["verdict"] == "improved" + assert any("requested CPU controls were omitted or invalid" in x for x in result["limitations"]) + assert any("preload library identity" in x for x in result["limitations"]) + + +def cpu_record(store, *, offset=0, scale=1, model="Fixture CPU", topology="a" * 64, allowed=None): + rec = media_record(store, offset=offset, scale=scale) + rec["identity"]["gpu"] = ev.identity()["gpu"] + rec["identity"]["cpu"] = ev.identity(cpu={"model": model, "architecture": "x86_64", + "topology_sha256": topology, "allowed_cpus": allowed or [0, 1]})["cpu"] + return store.save(rec) + + +def test_cpu_media_identity_allows_thread_and_affinity_tuning(store): + a = cpu_record(store) + b = cpu_record(store, offset=3, scale=.5, allowed=[0]) + result = compare(a, b, metric="image_ms") + assert result["data"]["verdict"] == "improved" + + +@pytest.mark.parametrize("kwargs", [{"model": "Another CPU"}, {"topology": "b" * 64}]) +def test_cpu_model_or_topology_mismatch_is_inconclusive(store, kwargs): + result = compare(cpu_record(store), cpu_record(store, offset=3, scale=.5, **kwargs), metric="image_ms") + assert result["data"]["verdict"] == "inconclusive" + assert "different CPU hardware" in result["data"]["decision"]["reason"] + + +def test_cpu_vs_gpu_media_is_inconclusive(store): + result = compare(cpu_record(store), media_record(store, offset=3, scale=.5), metric="image_ms") + assert result["data"]["verdict"] == "inconclusive" + assert "hardware backends differ" in result["data"]["decision"]["reason"] + + +def scheduler_proof(rec): + rec["data"].update(scheduler_final=True, + round_plan={"expected_indices": list(range(4))}, + lifecycle={"source": "bench rounds", "fresh_process_per_cell": True, "valid": True, "verdict": "PASS"}, + runtime_rounds=[{"index": i, "status": "ok", "ready": {"status": "ready"}, + "stop": {"group_gone": True}} for i in range(4)]) + return rec + + +def test_scheduler_lifecycle_proof_does_not_claim_quality_or_reliability(store): + a = store.save(scheduler_proof(cpu_record(store))) + b = store.save(scheduler_proof(cpu_record(store, offset=3, scale=.5))) + result = compare(a, b, metric="image_ms") + assert result["data"]["required_proofs"] == {"media quality": "UNPROVEN", "fresh-process lifecycle": "PASS", + "repeated-run reliability": "UNPROVEN"} + b["data"]["runtime_rounds"][0]["stop"]["group_gone"] = False + result = compare(a, store.save(b), metric="image_ms") + assert result["data"]["required_proofs"]["fresh-process lifecycle"] == "UNPROVEN" + + +def test_lifecycle_flag_alone_does_not_pass(store): + a, b = cpu_record(store), cpu_record(store, offset=3, scale=.5) + for rec in (a, b): + rec["data"]["lifecycle"] = {"source": "bench rounds", "fresh_process_per_cell": True, "valid": True, "verdict": "PASS"} + result = compare(store.save(a), store.save(b), metric="image_ms") + assert result["data"]["required_proofs"]["fresh-process lifecycle"] == "UNPROVEN" diff --git a/tests/test_cpu_screen.py b/tests/test_cpu_screen.py new file mode 100644 index 0000000..d8738d4 --- /dev/null +++ b/tests/test_cpu_screen.py @@ -0,0 +1,85 @@ +"""Manifest preparation is GPU/CPU-workload free and never changes the source.""" +import copy +import json + +import pytest + +from inferencekit import cli +from inferencekit.ops import cpu +from inferencekit.registry import OpError +from test_manifest import good_manifest + + +def facts(): + return {"model": "Fixture CPU", "architecture": "x86_64", "logical_cores": 8, "physical_cores": 4, + "allowed_cpus": list(range(8)), + "topology": [{"cpu": n, "core": n % 4, "package": 0, "core_type": None} for n in range(8)], + "l3_groups": [{"cpus": [0, 1, 4, 5]}, {"cpus": [2, 3, 6, 7]}], "numa_nodes": []} + + +def baseline(): + m = good_manifest() + e = m["engines"][0] + e["launch"][-1] = "{port}" + e["launch"] += ["--threads", "{threads}"] + m["engines"] = [e] + return m + + +def test_grid_is_deterministic_explicit_and_does_not_reuse_resolved_config(): + source = baseline() + before = copy.deepcopy(source) + result, plan = cpu.screen_manifest(source, facts=facts(), threads=[1, 2], pinning=["none", "compact", "spread"]) + assert source == before + assert result["model"] == source["model"] and result["workload"] == source["workload"] + assert result["hardware"]["backend"] == "cpu" and "gpu" not in result["hardware"] + assert len(result["engines"]) == 6 + assert [p["cpus"] for p in plan if p["threads"] == 2] == [list(range(8)), [0, 1], [0, 2]] + assert len({e["port"] for e in result["engines"]}) == 6 + assert all("resolved_config" not in e and "{" not in " ".join(e["launch"]) for e in result["engines"]) + assert result["engines"][-1]["env"]["OMP_PROC_BIND"] == "spread" + + +@pytest.mark.parametrize("kwargs,needle", [ + ({"threads": [0]}, "positive"), ({"threads": [5]}, "physical"), + ({"cpus": [9]}, "allowed"), ({"thread_env": ["HF_TOKEN"]}, "thread_env"), + ({"pinning": ["invented"]}, "pinning"), +]) +def test_invalid_screen_is_rejected(kwargs, needle): + args = {"facts": facts(), "threads": [2], "pinning": ["compact"], **kwargs} + with pytest.raises(OpError, match=needle): + cpu.screen_manifest(baseline(), **args) + + +def test_smt_is_explicit_and_client_cores_remain_outside_pool(): + _, plan = cpu.screen_manifest(baseline(), facts=facts(), threads=[4], pinning=["compact"], + cpus=[0, 1, 4, 5], include_smt=True) + assert plan[0]["cpus"] == [0, 1, 4, 5] + + +def test_placeholders_or_explicit_count_env_required(): + m = baseline() + m["engines"][0]["launch"] = m["engines"][0]["launch"][:-2] + with pytest.raises(OpError, match="threads"): + cpu.screen_manifest(m, facts=facts(), threads=[2], pinning=["none"]) + result, _ = cpu.screen_manifest(m, facts=facts(), threads=[2], pinning=["none"], thread_env=["OMP_NUM_THREADS"]) + assert result["engines"][0]["env"]["OMP_NUM_THREADS"] == "2" + + +def test_cli_writes_only_a_manifest_and_refuses_overwrite(tmp_path, monkeypatch, capsys): + source, out = tmp_path / "base.json", tmp_path / "screen.json" + source.write_text(json.dumps(baseline())) + monkeypatch.setattr(cpu, "cpu_facts", facts) + args = ["cpu", "screen-manifest", str(source), "--out", str(out), "--threads", "1", "--threads", "2"] + assert cli.main(args) == 0 + result = json.loads(capsys.readouterr().out) + assert result["ran"] is False and len(result["variants"]) == 6 + assert out.is_file() and json.loads(source.read_text()) == baseline() + assert cli.main(args) == 1 + + +def test_secret_literal_env_fails_before_output(): + m = baseline() + m["engines"][0]["env"]["HF_TOKEN"] = "secret-value" + with pytest.raises(OpError, match="secret"): + cpu.screen_manifest(m, facts=facts(), threads=[1], pinning=["none"]) diff --git a/tests/test_cpuinfo.py b/tests/test_cpuinfo.py new file mode 100644 index 0000000..3bd0fbf --- /dev/null +++ b/tests/test_cpuinfo.py @@ -0,0 +1,110 @@ +"""CPU discovery uses fixture OS files; no workloads or counter probes.""" +import json + +import pytest + +from inferencekit import cpuinfo, evidence, hardware, registry +from inferencekit.ops import doctor + + +def put(path, value): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(str(value)) + + +@pytest.fixture +def cpu_card(tmp_path, monkeypatch): + proc, cpu, node = tmp_path / "proc", tmp_path / "cpu", tmp_path / "node" + put(proc / "cpuinfo", "\n\n".join(f"processor : {i}\nmodel name : Fixture Hybrid\n" + "flags : avx2 avx512f avx512_vnni avx_vnni avx512_bf16\n" + "Serial : SECRET_SERIAL\n" for i in range(8))) + put(cpu / "online", "0-7") + for i in range(8): + root = cpu / f"cpu{i}" + put(root / "topology/physical_package_id", 0) + put(root / "topology/core_id", i % 4) + put(root / "topology/thread_siblings_list", f"{i % 4},{i % 4 + 4}") + put(root / "topology/core_type", 1 if i % 4 < 2 else 2) + put(root / "cache/index3/level", 3) + put(root / "cache/index3/shared_cpu_list", "0-1,4-5" if i % 4 < 2 else "2-3,6-7") + put(root / "cache/index3/size", "16384K") + put(node / "node0/cpulist", "0-1,4-5") + put(node / "node1/cpulist", "2-3,6-7") + put(cpu / "cpufreq/policy0/scaling_governor", "powersave") + put(proc / "sys/kernel/perf_event_paranoid", 4) + put(proc / "self/status", "Name:\tSECRET_PROCESS\nCapEff:\t0000000000000000\n") + put(proc / "loadavg", "30.00 20.00 10.00 1/2 12345") + monkeypatch.setattr(cpuinfo, "which", lambda _: "/fixture/perf") + monkeypatch.setattr(cpuinfo.os, "environ", {"HF_TOKEN": "SECRET_TOKEN", "LD_LIBRARY_PATH": "/SECRET_PATH", + "OMP_NUM_THREADS": "4", "MKL_ENABLE_INSTRUCTIONS": "AVX2", + "DNNL_MAX_CPU_ISA": "SECRET_VALUE"}) + return cpuinfo.cpu_facts(proc_root=proc, cpu_root=cpu, node_root=node, allowed_cpus=[0, 1, 2, 3]) + + +def test_cpu_card_has_observed_topology_and_no_raw_or_secret_data(cpu_card): + assert cpu_card["physical_cores"] == 4 and cpu_card["logical_cores"] == 8 + assert cpu_card["allowed_cpus"] == [0, 1, 2, 3] + assert len(cpu_card["l3_groups"]) == len(cpu_card["numa_nodes"]) == 2 + assert {r["core_type"] for r in cpu_card["topology"]} == {1, 2} + assert all(cpu_card["isa"][k] for k in ("avx2", "avx512", "avx512_vnni", "avx_vnni", "avx512_bf16")) + assert cpu_card["governors"] == ["powersave"] + assert cpu_card["perf"]["access"] == "restricted" and not cpu_card["perf"]["verified"] + assert any("contention" in w for w in cpu_card["limitations"]) + assert cpu_card["tuning_env"] == {"OMP_NUM_THREADS": "4", "MKL_ENABLE_INSTRUCTIONS": "AVX2"} + assert cpu_card["library_dispatch"]["status"] == "unverified" + assert "SECRET" not in json.dumps(cpu_card) + + +def test_identity_excludes_volatile_load_and_dispatch(cpu_card): + before = cpuinfo.cpu_identity(cpu_card) + cpu_card["load_average"] = [0, 0, 0] + cpu_card["governors"] = ["performance"] + cpu_card["tuning_env"] = {} + assert cpuinfo.cpu_identity(cpu_card) == before + assert "cpu" not in evidence.identity() + assert evidence.identity(cpu=before)["cpu"] == before + cpu_card["topology"][0]["core"] = 99 + assert cpuinfo.cpu_identity(cpu_card)["topology_sha256"] != before["topology_sha256"] + + +def test_missing_topology_and_isa_stay_unknown(tmp_path, monkeypatch): + monkeypatch.setattr(cpuinfo, "which", lambda _: None) + card = cpuinfo.cpu_facts(proc_root=tmp_path, cpu_root=tmp_path, node_root=tmp_path, allowed_cpus=[]) + assert card["physical_cores"] is None and card["topology"] == [] + assert all(v is None for v in card["isa"].values()) + assert card["load_average"] is None and card["perf"]["access"] == "unknown" + + +@pytest.mark.parametrize("paranoid,caps,expected", [(2, 0, "policy_allows_user"), (4, 0, "restricted"), + (4, 1 << 38, "policy_allows_user")]) +def test_perf_policy_is_not_a_permission_probe(tmp_path, monkeypatch, paranoid, caps, expected): + put(tmp_path / "sys/kernel/perf_event_paranoid", paranoid) + put(tmp_path / "self/status", f"CapEff:\t{caps:016x}\n") + monkeypatch.setattr(cpuinfo, "which", lambda _: "/fixture/perf") + card = cpuinfo.cpu_facts(proc_root=tmp_path, cpu_root=tmp_path, node_root=tmp_path, allowed_cpus=[]) + assert card["perf"]["access"] == expected and card["perf"]["verified"] is False + route = hardware.profiler_route("cpu", card) + assert ("perf (" in route["timeline"]) == (expected == "policy_allows_user") + assert "torch.profiler" in route["fallback"] + + +def test_cpu_doctor_never_calls_cuda_permissions_or_probes(cpu_card, monkeypatch, store): + monkeypatch.setattr(doctor, "query_gpus", lambda: []) + monkeypatch.setattr(doctor, "discover_non_nvidia", lambda: {"platform": "cpu", "devices": [], "cpu": cpu_card}) + monkeypatch.setattr(doctor, "counter_permission", lambda: pytest.fail("NVIDIA permission queried")) + monkeypatch.setattr(doctor, "profiler_probe", lambda *_: pytest.fail("CUDA workload launched")) + monkeypatch.setattr(doctor, "engine_versions", lambda _: {}) + monkeypatch.setattr(doctor, "llamacpp_version", lambda: None) + monkeypatch.setattr(doctor, "tool_version", lambda _: "fixture") + result = registry.call("doctor", {"probe_profilers": True}) + assert result["identity"]["cpu"] == cpuinfo.cpu_identity(cpu_card) + assert result["identity"]["gpu"]["name"] is None + assert result["data"]["profiler_probe"]["ncu"]["ran"] is False + assert "torch.profiler" in result["data"]["profiling_route"]["timeline"] + assert evidence.validate(result) == [] + assert "SECRET" not in json.dumps(result) + + +@pytest.mark.parametrize("value", ["1-0", "1-x", "1000001", "0-100001", "1,,3"]) +def test_invalid_cpu_lists_are_not_partially_used(value): + assert cpuinfo.cpu_list(value) == [] diff --git a/tests/test_doctor_counters.py b/tests/test_doctor_counters.py index 6cb3c58..6013bfe 100644 --- a/tests/test_doctor_counters.py +++ b/tests/test_doctor_counters.py @@ -219,7 +219,7 @@ def no_space(self): def _doctor(monkeypatch, store, fake, perm_status="unknown"): use(monkeypatch, fake) monkeypatch.setattr(dr, "counter_permission", lambda: {"status": perm_status, "reason": "static", "euid": 0}) - monkeypatch.setattr(dr, "query_gpus", lambda: []) + monkeypatch.setattr(dr, "query_gpus", lambda: [{"name": "Fixture NVIDIA GPU"}]) monkeypatch.setattr(dr, "tool_version", lambda name: "2026.1" if name in fake.tools else None) return registry.call("doctor", {"probe_profilers": True}) @@ -237,7 +237,7 @@ def test_doctor_probe_decides_the_counter_status(monkeypatch, store): def test_doctor_without_probe_says_probe_needed(monkeypatch, store): monkeypatch.setattr(dr, "counter_permission", lambda: {"status": "unknown", "reason": "no params (probe needed)"}) - monkeypatch.setattr(dr, "query_gpus", lambda: []) + monkeypatch.setattr(dr, "query_gpus", lambda: [{"name": "Fixture NVIDIA GPU"}]) rec = registry.call("doctor", {}) assert "counters unknown (probe needed)" in rec["summary"] and "profiler_probe" not in rec["data"] assert any(x.startswith("profiling counter access unknown") for x in rec["limitations"]) @@ -246,7 +246,7 @@ def test_doctor_without_probe_says_probe_needed(monkeypatch, store): def test_cli_flag(monkeypatch, tmp_path, capsys): calls = [] monkeypatch.setattr(dr, "profiler_probe", lambda python: calls.append(python) or {"ncu": None, "nsys": None}) - monkeypatch.setattr(dr, "query_gpus", lambda: []) + monkeypatch.setattr(dr, "query_gpus", lambda: [{"name": "Fixture NVIDIA GPU"}]) try: assert cli.main(["--runs-dir", str(tmp_path / "runs"), "doctor", "--probe-profilers"]) == 0 finally: diff --git a/tests/test_evidence_summary.py b/tests/test_evidence_summary.py new file mode 100644 index 0000000..9f13263 --- /dev/null +++ b/tests/test_evidence_summary.py @@ -0,0 +1,267 @@ +"""Compact inspection retains decision evidence and routes back to exact stored values.""" + +import json + +import pytest + +from inferencekit import cli, evidence as ev, registry +from inferencekit.mcp_server import fit +from inferencekit.ops.store import get_path +from inferencekit.registry import OpError + +from conftest import fake_call + + +def large_record(store): + checks = [{"gate": f"check_{i}", "verdict": "PASS", "applicable": True, "reason": "tokens agree", + "samples": [{"token": j, "logprobs": [-0.1, -2.3], "text": "sample " * 40} + for j in range(30)]} for i in range(80)] + checks[-2].update(verdict="FAIL", reason="late token mismatch") + checks[-1].update(verdict="NOT RUN", reason="reference unavailable") + return store.save(ev.build(fake_call("run_gates", ("gates", "run")), kind="gates", claim_class="observed", + summary="FAIL: one mismatch; quality remains UNPROVEN", + identity_=ev.identity(engine={"name": "candidate"}, + workload={"prompt_len": 128, "gen_len": 64}), + metrics=[ev.metric("decode_tok_s_ratio", 1.25, "x", "ratio_of_medians", n=6, + ci95=[1.2, 1.3])], + missing=["reference quality proof"], limitations=["screening workload only"], + data={"verdict": "FAIL", "checks": checks, + "claim_validity": {"valid": False, "violations": ["quality proof absent"]}, + "required_proofs": {"quality": "UNPROVEN"}})) + + +def inspect(rec, **kwargs): + return registry.call("inspect_evidence", {"id": rec["evidence_id"], **kwargs}) + + +def test_summary_preserves_decision_inputs_and_full_record(store): + rec = large_record(store) + path = store.run_dir(rec["evidence_id"]) / "evidence.json" + before = path.read_bytes() + full = inspect(rec) + summary = inspect(rec, view="summary") + assert full == {"integrity": ev.verify(rec), "record": rec} + assert inspect(rec, view="full") == full + assert summary["integrity"] == {"id_ok": True, "schema_ok": True, "schema_errors": []} + for key in ("evidence_id", "kind", "claim_class", "summary", "identity", "provenance", "metrics", + "coverage", "limitations"): + assert summary[key] == rec[key] + assert summary["finding_counts"]["verdict:FAIL"] == 2 + assert summary["finding_counts"]["verdict:NOT RUN"] == 1 + assert summary["finding_counts"]["verdict:UNPROVEN"] == 1 + assert summary["finding_counts"]["valid:False"] == 1 + assert summary["data_index"]["checks"] == {"path": "data.checks", "type": "array", "count": 80} + assert len(json.dumps(summary)) < len(json.dumps(full)) / 10 + assert path.read_bytes() == before and store.load(rec["evidence_id"]) == rec + + +def test_late_failures_first_and_all_findings_recoverable_without_skips(store): + rec = large_record(store) + summary = inspect(rec, view="summary", limit=6) + first = {x["path"]: x for x in summary["items"]} + assert first["data.checks.78.verdict"]["reason"] == "late token mismatch" + assert first["data.checks.79.verdict"]["applicable"] is True + assert first["data.checks.79.verdict"]["value"] == "NOT RUN" + assert "view='summary', offset=6, limit=6" in summary["more"] + found = [] + while True: + for item in summary["items"]: + assert get_path(rec, item["path"]) == item["value"] + found.append(item["path"]) + if summary["next_offset"] is None: + break + summary = inspect(rec, view="summary", offset=summary["next_offset"], limit=6) + assert len(found) == len(set(found)) == summary["total"] == 83 + exact = inspect(rec, path="data.checks", offset=78, limit=2) + assert exact["items"] == rec["data"]["checks"][78:] and exact["next_offset"] is None + assert inspect(rec, path="data.checks.78")['value'] == rec["data"]["checks"][78] + + +def test_summary_does_not_invent_a_verdict_and_retains_bad_integrity(store): + rec = store.save(ev.build(fake_call(), kind="benchmark", claim_class="observed", summary="no checks", + data={"samples": [1, 2, 3]})) + summary = inspect(rec, view="summary") + assert summary["finding_counts"] == {} and summary["items"] == [] + assert "No overall verdict is inferred" in summary["note"] + path = store.run_dir(rec["evidence_id"]) / "evidence.json" + rec["summary"] = "changed after capture" + path.write_text(json.dumps(rec)) + assert inspect(rec, view="summary")["integrity"]["id_ok"] is False + + +def test_cli_summary_and_path_combination(store, capsys, monkeypatch): + rec = large_record(store) + monkeypatch.delenv("INFERENCEKIT_OUTPUT", raising=False) + assert cli.main(["evidence", "show", rec["evidence_id"], "--view", "summary", "--limit", "3"]) == 0 + view = json.loads(capsys.readouterr().out) + assert view["view"] == "summary" and len(view["items"]) == 3 + with pytest.raises(OpError, match="cannot be combined"): + inspect(rec, view="summary", path="metrics") + + +def test_mcp_summary_page_hint_keeps_view_and_does_not_skip(store): + rec = large_record(store) + summary = inspect(rec, view="summary") + view = fit(summary, max_chars=6000) + assert len(json.dumps(view)) <= 6000 + assert 1 <= len(view["items"]) < len(summary["items"]) + assert view["next_offset"] == len(view["items"]) + assert f"view='summary', offset={view['next_offset']}" in view["more"] + assert view["finding_counts"] == summary["finding_counts"] and view["integrity"] == summary["integrity"] + + +@pytest.mark.parametrize("width", [725, 735, 745]) +def test_summary_page_hint_overhead_never_skips_findings(store, width): + rec = store.save(ev.build(fake_call(), kind="gates", claim_class="observed", summary="50 checks", + data={"checks": [{"verdict": "PASS", "reason": "x" * width} + for _ in range(50)]})) + offset, seen = 0, [] + while offset is not None: + source = inspect(rec, view="summary", offset=offset, limit=50) + before = json.dumps(source) + shown = fit(source) + assert json.dumps(source) == before + assert len(json.dumps(shown, ensure_ascii=False)) <= 32000 + assert isinstance(shown["items"], list) and shown["items"] + seen.extend(row["path"] for row in shown["items"]) + if shown["next_offset"] is not None: + assert shown["next_offset"] == offset + len(shown["items"]) + assert f"view='summary', offset={shown['next_offset']}" in shown["more"] + offset = shown["next_offset"] + assert seen == [f"data.checks.{i}.verdict" for i in range(50)] + + +def test_mcp_summary_qualifier_hint_points_to_original_record(store): + rec = store.save(ev.build(fake_call(), kind="benchmark", claim_class="observed", summary="invalid", + data={"claim_validity": {"valid": False, + "violations": [f"problem {i}: " + "x" * 200 for i in range(100)]}})) + view = fit(inspect(rec, view="summary"), max_chars=8000) + cut = view["items"][0]["violations"] + assert cut["truncated"] + assert "path='data.claim_validity.violations', offset=10" in cut["more"] + assert "items.0" not in cut["more"] + + +def test_mcp_summary_last_resort_retains_failure_counts_and_integrity(store): + rec = large_record(store) + summary = inspect(rec, view="summary") + summary["identity"] = {f"field{i}": "x" * 200 for i in range(100)} + view = fit(summary, max_chars=3000) + assert view["truncated"] and view["view"] == "summary" + assert view["finding_counts"]["verdict:FAIL"] == 2 and view["integrity"]["id_ok"] is True + assert view["limitation_count"] == view["missing_count"] == 1 + assert "omitted details cannot establish PASS" in view["more"] + + +def test_ambiguous_engine_keys_use_exact_json_pointers(store): + rec = store.save(ev.build(fake_call(), kind="gates", claim_class="observed", summary="FAIL", + data={"verdicts": {"engine.v1": "FAIL", "engine": {"v1": "PASS"}, + "a./~b": "UNPROVEN", "": "NOT RUN"}, + "checks.v1": [{"verdict": "FAIL", "reason": "exact path"}]})) + summary = inspect(rec, view="summary") + values = {f["path"]: f["value"] for f in summary["items"]} + assert values["/data/verdicts/engine.v1"] == "FAIL" + assert values["data.verdicts.engine.v1"] == "PASS" + assert values["/data/verdicts/a.~1~0b"] == "UNPROVEN" + assert values["/data/verdicts/"] == "NOT RUN" + assert values["/data/checks.v1/0/verdict"] == "FAIL" + assert summary["data_index"]["checks.v1"]["path"] == "/data/checks.v1" + for path, value in values.items(): + assert inspect(rec, path=path)["value"] == value + assert inspect(rec, path="data.verdicts.engine.v1")["value"] == "PASS" # unchanged dotted semantics + + +def test_mcp_qualifier_hint_keeps_json_pointer_escaping(store): + rec = store.save(ev.build(fake_call(), kind="gates", claim_class="observed", summary="FAIL", + data={"checks.v1": {"a/b~c": {"verdict": "FAIL", + "violations": [f"problem {i}: " + "x" * 200 for i in range(100)]}}})) + view = fit(inspect(rec, view="summary"), max_chars=8000) + cut = view["items"][0]["violations"] + assert "path='/data/checks.v1/a~1b~0c/violations', offset=10" in cut["more"] + assert len(inspect(rec, path="/data/checks.v1/a~1b~0c/violations")["items"]) == 50 + + +def test_mcp_nested_list_hints_follow_pointer_pages_and_object_wrappers(store): + rec = store.save(ev.build(fake_call(), kind="gates", claim_class="observed", summary="FAIL", + data={"checks.v1": [{"verdict": "FAIL", "probe.details": ["x" * 200] * 100}] + * 10})) + for source_path, kwargs, wrapper in [("/data/checks.v1", {"offset": 7, "limit": 1}, "items"), + ("/data/checks.v1/7", {}, "value")]: + view = fit(inspect(rec, path=source_path, **kwargs), max_chars=4000) + item = view["items"][0] if wrapper == "items" else view["value"] + cut = item["probe.details"] + assert "path='/data/checks.v1/7/probe.details', offset=10" in cut["more"] + exact = inspect(rec, path="/data/checks.v1/7/probe.details", offset=10, limit=50) + assert exact["items"] == rec["data"]["checks.v1"][7]["probe.details"][10:60] + + +def test_mcp_full_record_hint_escapes_ambiguous_keys(store): + rec = store.save(ev.build(fake_call(), kind="gates", claim_class="observed", summary="FAIL", + data={"engine.v1": {"probe/details": ["x" * 200] * 100}})) + view = fit(inspect(rec), max_chars=5000) + cut = view["record"]["data"]["engine.v1"]["probe/details"] + assert "path='/data/engine.v1/probe~1details', offset=10" in cut["more"] + assert inspect(rec, path="/data/engine.v1/probe~1details", offset=10)["total"] == 100 + + +@pytest.mark.parametrize("status", ["launch_failed", "serve_failed", "startup_timeout", "warmup_failed", + "stop_failed", "exited_early", "port_busy", "serve_no_metrics"]) +def test_runtime_failures_precede_unproven_and_keep_detail(store, status): + rec = store.save(ev.build(fake_call(), kind="serve_benchmark", claim_class="observed", summary="failed cell", + data={"checks": [{"verdict": "UNPROVEN"}] * 50, + "cells": [{"status": status, "detail": "engine failed to finish", "round": 1}]})) + first = inspect(rec, view="summary", limit=1)["items"][0] + assert first["value"] == status and first["detail"] == "engine failed to finish" + assert inspect(rec, path=first["path"])["value"] == status + + +@pytest.mark.parametrize("statuses", [["X" * 50_000], [f"unique_{i}" for i in range(3000)]]) +def test_unbounded_status_values_keep_exact_total_without_unbounded_count_keys(store, statuses): + rec = store.save(ev.build(fake_call(), kind="serve_benchmark", claim_class="observed", summary="status fixture", + data={"cells": [{"status": status} for status in statuses]})) + summary = inspect(rec, view="summary") + assert summary["finding_counts"] == {"status:": len(statuses)} + assert summary["total"] == len(statuses) + assert summary["items"][0]["value"] == statuses[0] + view = fit(summary, max_chars=3000) + assert len(json.dumps(view, ensure_ascii=False)) <= 3000 + assert view["finding_counts"] == summary["finding_counts"] + assert view["integrity"]["id_ok"] is True + + +def test_mcp_summary_bounds_malformed_integrity_errors_and_counts_all(store): + rec = large_record(store) + path = store.run_dir(rec["evidence_id"]) / "evidence.json" + rec["metrics"] = [{"name": f"bad_{i}", "value": "x" * 50_000, + "unit": "x", "stat": "x", "n": 1} for i in range(25)] + path.write_text(json.dumps(rec)) + error_count = len(ev.validate(rec)) + assert error_count > 20 # verify exposes at most 20 messages, but the total must remain exact + summary = inspect(rec, view="summary") + view = fit(summary) + assert len(json.dumps(view, ensure_ascii=False)) <= 32_000 + assert view["truncated"] + assert view["integrity"] == {"id_ok": False, "schema_ok": False, + "schema_error_count": error_count, "schema_errors_omitted": True} + assert view["finding_counts"]["verdict:FAIL"] == 2 + assert "inferencekit evidence show" in view["more"] and "original id or path" in view["more"] + assert "inferencekit.evidence.validate" in view["more"] + assert inspect(rec)["integrity"]["schema_errors"] # ordinary full inspection is unchanged + + +@pytest.mark.parametrize("field,value", [("artifacts", 7), ("limitations", 7), + ("coverage", 7), ("coverage", {"missing": 7})]) +def test_invalid_top_level_counts_remain_unknown(store, field, value): + rec = large_record(store) + path = store.run_dir(rec["evidence_id"]) / "evidence.json" + rec[field] = value + # Force the bounded fallback without relying on the malformed field to make it large. + rec["identity"] = {f"field{i}": "x" * 200 for i in range(100)} + path.write_text(json.dumps(rec)) + summary = inspect(rec, view="summary") + assert summary["integrity"]["schema_ok"] is False + assert summary["artifact_count"] == (None if field == "artifacts" else len(rec["artifacts"])) + view = fit(summary, max_chars=3000) + assert view["truncated"] and view["integrity"]["schema_ok"] is False + assert view["limitation_count"] == (None if field == "limitations" else len(rec["limitations"])) + assert view["missing_count"] == (None if field == "coverage" else len(rec["coverage"]["missing"])) diff --git a/tests/test_intake.py b/tests/test_intake.py index d94f2ac..20fe5ce 100644 --- a/tests/test_intake.py +++ b/tests/test_intake.py @@ -54,6 +54,59 @@ def test_defaults_are_filled_and_precision_rule_is_equal_precision(): assert r["derived"]["claim_class"] == "derived" and r["derived"]["modes"] == ["single"] +@pytest.mark.parametrize("modality", ["tts", "stt", "image", "video"]) +def test_media_has_no_llm_token_or_prompt_defaults(modality): + result = intake.check(base(modality=modality)) + assert "max_tokens" not in result["intake"]["workload"] + assert "prompt_set" not in result["intake"]["workload"] + legacy = intake.check(base(modality=modality, workload={"max_tokens": 1})) + assert legacy["intake"]["workload"]["max_tokens"] == 1 + assert any("unused" in w for w in legacy["warnings"]) + + +def test_llm_still_rejects_one_output_token_in_schema_and_validator(): + data = base(workload={"max_tokens": 1}) + assert not jsonschema.Draft202012Validator(intake.load_schema()).is_valid(data) + with pytest.raises(OpError, match="max_tokens"): + intake.check(data) + + +def test_tts_workload_preserves_pins_and_recognizes_audio_client(): + speech = {"voice": "speaker-a", "sample_rate": 24000, "lead_metric": "first_audio_ms", + "text_length_buckets": [{"name": "short", "min_chars": 1, "max_chars": 80}, + {"name": "long", "min_chars": 81, "max_chars": 800}]} + result = intake.check(base(modality="tts", workload={"client": "bench_audio", "tts": speech}, + io_modes={"output": "complete"}, goals={"targets": {"rtf": 0.5}})) + assert result["intake"]["workload"]["tts"] == speech + assert result["intake"]["io_modes"]["output"] == "complete" + assert result["intake"]["buckets"] == ["short", "long"] + assert result["derived"]["modes"] == ["audio"] + assert not any("no audio timing client" in w for w in result["warnings"]) + assert any("quality" in w and "UNPROVEN" in w for w in result["warnings"]) + again = intake.check(result["intake"]) + assert again["intake"] == result["intake"] and again["derived"] == result["derived"] + + +@pytest.mark.parametrize("workload", [ + {"client": "bench_serve"}, {"tts": {"sample_rate": 0}}, + {"tts": {"lead_metric": "decode_tok_s"}}, + {"tts": {"text_length_buckets": [{"name": "short", "min_chars": 20, "max_chars": 2}]}}, + {"tts": {"text_length_buckets": [{"name": "short", "min_chars": 1, "max_chars": 2}] * 2}}, +]) +def test_bad_tts_workload_is_rejected(workload): + with pytest.raises(OpError): + intake.check(base(modality="tts", workload=workload)) + + +def test_audio_client_does_not_claim_stt_support_or_allow_tts_fields_for_llm(): + with pytest.raises(OpError, match="does not support modality stt"): + intake.check(base(modality="stt", workload={"client": "bench_audio"})) + with pytest.raises(OpError, match="only to modality tts"): + intake.check(base(workload={"tts": {"voice": "speaker"}})) + external = intake.check(base(modality="stt", workload={"client": "external"})) + assert not any("no audio timing client" in w for w in external["warnings"]) + + def test_no_target_interactive_means_the_stop_point_asks(): r = intake.check(base()) stop = r["derived"]["stop_rule"] diff --git a/tests/test_interfaces.py b/tests/test_interfaces.py index 2d3f8bc..e7160b0 100644 --- a/tests/test_interfaces.py +++ b/tests/test_interfaces.py @@ -13,7 +13,7 @@ CONTRACT_TOOLS = {"doctor", "roofline", "measure_bandwidth", "bench_decode", "capture_nsys", "analyze_nsys", "capture_ncu", "summarize_ncu", "compare", "check_correctness", "list_evidence", "inspect_evidence", "report"} -CONTRACT_TOOLS |= {"intake_template", "intake_validate", "intake_budget"} # intake +CONTRACT_TOOLS |= {"intake_template", "intake_validate", "intake_budget", "intake_resume"} # intake CONTRACT_TOOLS |= {"build_review", "ledger_trailers", "history"} # review page and ledger CONTRACT_TOOLS |= {"quality_trajectory", "quality_score", "quality_gsm8k", "quality_niah"} # quality checks CONTRACT_TOOLS |= {"quality_sampling"} # first-token sampler check (draws vs the server's own logprobs) @@ -26,6 +26,8 @@ CONTRACT_TOOLS |= {"bench_image", "bench_audio"} # client-timed image and speech outputs CONTRACT_TOOLS |= {"gates_clip", "quality_clip", "quality_clip_steps"} # video (gates, quality, per-step) CONTRACT_TOOLS |= {"capture_rocprof", "analyze_rocprof"} # AMD profiling +CONTRACT_TOOLS |= {"cpu_screen_manifest", "quality_audio_policy", "quality_audio"} +CONTRACT_TOOLS |= {"quality_audio_scorers", "quality_audio_score"} def test_registry_matches_contract_tool_names(): @@ -125,3 +127,47 @@ def test_mcp_hint_inside_a_page_item_points_at_the_real_path(): view = fit(result, max_chars=8_000) cut = view["items"][0]["stalls"] assert cut["truncated"] and "path='data.kernels.7.stalls'" in cut["more"] + + +def test_mcp_preserves_operation_and_source_arguments_when_shortening_a_page(): + from inferencekit.ops.store import page + + args = {"run_dir": "runs/experiment", "file": "alternate-intake.yaml", "now": "2026-11-05T21:00:00Z"} + rows = [{"name": f"row_{i}", "detail": "x" * 600} for i in range(100)] + source = {"pagination": {"tool": "intake_resume", "args": args}, **page(rows, 10, 50)} + view = fit(source, max_chars=6000) + k = len(view["items"]) + assert len(json.dumps(view)) <= 6000 and 1 <= k < 50 + assert view["next_call"] == {"tool": "intake_resume", "arguments": {**args, "offset": 10 + k, "limit": k}} + assert view["next_offset"] == 10 + k + assert source["pagination"]["args"] == args and "next_call" not in source + + +@pytest.mark.parametrize("width,max_chars", [(600, 9802), (823, 32000)]) +def test_resume_page_hint_overhead_cannot_skip_unshown_items(width, max_chars): + from inferencekit.ops.store import page + + args = {"run_dir": "runs/experiment", "file": "original.json", "now": "2026-11-05T21:00:00Z"} + rows = [{"index": i, "summary": "x" * width} for i in range(100)] + offset, limit, seen = 0, 50, [] + while offset is not None: + source = {"pagination": {"tool": "intake_resume", "args": args}, **page(rows, offset, limit)} + if source["next_offset"] is not None: + source["next_call"] = {"tool": "intake_resume", "arguments": { + **args, "offset": source["next_offset"], "limit": limit}} + before = json.dumps(source) + shown = fit(source, max_chars=max_chars) + assert json.dumps(source) == before + assert len(json.dumps(shown)) <= max_chars + assert isinstance(shown["items"], list) and shown["items"] + assert shown["items"][0]["index"] == offset + seen.extend(row["index"] for row in shown["items"]) + offset = shown["next_offset"] + if offset is not None: + assert offset == shown["items"][-1]["index"] + 1 + call = shown["next_call"] + assert call["tool"] == "intake_resume" + assert {k: call["arguments"][k] for k in args} == args + assert call["arguments"]["offset"] == offset + limit = call["arguments"]["limit"] + assert seen == list(range(100)) diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 9c2f716..743aa7d 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -48,6 +48,23 @@ def test_good_manifest_is_valid_without_warnings(): assert res == {"valid": True, "errors": [], "warnings": []} +def test_cpu_manifest_has_explicit_identity_without_gpu_label(): + m = good_manifest() + m["hardware"] = {"backend": "cpu", "count": 1, "cpu": {"model": "Fixture CPU", "architecture": "x86_64"}} + assert mf.check_manifest(m)["valid"] + m["hardware"]["gpu"] = "pretend GPU" + assert not mf.check_manifest(m)["valid"] + m["hardware"].pop("gpu") + m["hardware"].pop("cpu") + assert not mf.check_manifest(m)["valid"] + + +def test_unmarked_cpu_metadata_does_not_bypass_legacy_gpu_requirement(): + m = good_manifest() + m["hardware"] = {"count": 1, "cpu": {"model": "Fixture CPU"}} + assert not mf.check_manifest(m)["valid"] + + def test_full_example_manifest_passes_the_schema(): # A full example manifest, with placeholders filled. m = good_manifest() diff --git a/tests/test_media_rounds.py b/tests/test_media_rounds.py new file mode 100644 index 0000000..335460b --- /dev/null +++ b/tests/test_media_rounds.py @@ -0,0 +1,139 @@ +"""Real local server/client subprocesses exercise CPU media orchestration, never a model.""" +import json +import socket +import sys +from pathlib import Path + +import pytest + +from inferencekit import evidence as ev, registry +from inferencekit.registry import OpError +from inferencekit.serve import media_rounds as mr, servers + + +def free_port(): + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + return sock.getsockname()[1] + + +def write_run(tmp_path, mode, fail=False): + script = Path(__file__).parent / "fakes_rounds" / "fake_media_server.py" + engines = [] + for label, threads in (("base", "1"), ("candidate", "2")): + port = free_port() + engines.append({"label": label, "name": "custom", "version": "pinned", "port": port, + "env": {"OMP_NUM_THREADS": threads, "OPENBLAS_NUM_THREADS": threads}, + "launch": [sys.executable, str(script), "--port", str(port), "--mode", mode] + + (["--fail"] if fail and label == "candidate" else [])}) + manifest = {"model": {"id": "fake/model", "revision": "a" * 40, "precision": "fp32"}, + "hardware": {"backend": "cpu", "cpu": {"model": "local CPU"}, "count": 1}, + "engines": engines, "workload": {"modes": [mode], "sample_rate": 8000, "quant": "none"}} + path = tmp_path / "manifest.json" + path.write_text(json.dumps(manifest)) + fixtures = tmp_path / "fixtures.jsonl" + item = {"id": "one", "text": "Synthetic test."} if mode == "audio" else { + "id": "one", "prompt": "a fox", "width": 4, "height": 3, "steps": 5, "guidance_scale": 3.0} + fixtures.write_text(json.dumps(item) + "\n") + return path, fixtures + + +@pytest.mark.parametrize("mode", ["audio", "image"]) +def test_fresh_process_cpu_media_evidence(tmp_path, store, monkeypatch, mode): + manifest, fixtures = write_run(tmp_path, mode) + monkeypatch.setenv("INFERENCEKIT_ENDPOINT_HMAC_KEY", "synthetic-test-key") + monkeypatch.setattr(servers, "gpu_memory_used_mib", lambda: pytest.fail("CPU route probed GPU")) + result = mr.run_media_rounds(str(manifest), str(fixtures), mode, rounds=4, warmup=1, + startup_timeout_s=5, stop_timeout_s=1, poll_s=.02, + cpu_lock=str(tmp_path / "cpu.lock"), serve_args="--resamples 10") + assert all(c["status"] == "ok" for c in result["cells"]), result["cells"] + logs = [(tmp_path / c["log"]).read_text() for c in result["cells"]] + assert len({text.split("pid=")[1].splitlines()[0] for text in logs}) == 8 + assert all(c["stop"]["group_gone"] for c in result["cells"]) + final = result["plan"][f"{mode}_evidence_final"] + assert len(final) == 2 + for label, reference in final.items(): + record = store.load(reference) + assert ev.integrity_problem(record) is None + assert record["identity"]["cpu"]["model"] + assert not record["identity"]["gpu"]["name"] + assert record["data"]["round_coverage"]["complete"] is True + assert record["data"]["lifecycle"]["verdict"] == "PASS" + assert record["data"]["memory"]["peak_rss_mib"] > 0 + assert record["identity"]["engine"]["flags"]["server_env"]["controls"]["OMP_NUM_THREADS"] == ( + "1" if label == "base" else "2") + assert len(record["data"]["rounds"]) == 4 + assert all((store.run_dir(reference) / a["path"]).is_file() for a in record["artifacts"]) + assert (tmp_path / "artifacts" / reference / "evidence.json").is_file() + compared = registry.call("compare", {"baseline": final["base"], "candidate": final["candidate"], + "metric": "audio_ms" if mode == "audio" else "image_ms", "resamples": 20}) + assert compared["data"]["interleaved"] is True + review = registry.call("build_review", {"run_dir": str(tmp_path), "prompts": str(fixtures), "reference": "base"}) + assert review["engines"] == ["base", "candidate"] + assert review["inputs"] == 4 + + +def test_failed_media_cells_remain_in_plan_and_are_not_ranked(tmp_path, store): + manifest, fixtures = write_run(tmp_path, "audio", fail=True) + result = mr.run_media_rounds(str(manifest), str(fixtures), "audio", rounds=3, warmup=0, + startup_timeout_s=5, stop_timeout_s=1, poll_s=.02, + cpu_lock=str(tmp_path / "cpu.lock"), serve_args="--resamples 10") + bad = [c for c in result["cells"] if c["label"] == "candidate"] + assert bad[0]["status"] == "serve_failed" + assert [c["status"] for c in bad[1:]] == ["skipped_after_failure"] * 2 + assert all(r["best"] != "candidate" for r in result["summary"]["rows"]) + record = store.load(result["plan"]["audio_evidence_final"]["candidate"]) + assert record["data"]["round_coverage"]["missing_indices"] == [1, 2] + assert record["data"]["lifecycle"]["verdict"] == "UNPROVEN" + + +def test_auto_routing_and_protected_arguments(tmp_path): + assert mr.mode_for({"workload": {"modality": "tts"}}, "auto", str(tmp_path / "m.json")) == "audio" + (tmp_path / "intake.normalized.json").write_text('{"modality":"image"}') + assert mr.mode_for({}, "auto", str(tmp_path / "m.json")) == "image" + (tmp_path / "intake.normalized.json").write_text('{"modality":"llm"}') + assert mr.mode_for({}, "auto", str(tmp_path / "m.json")) == "single" + with pytest.raises(OpError, match="multiple"): + mr.mode_for({"workload": {"modes": ["single", "load"]}}, "auto", "m.json") + with pytest.raises(OpError, match="override"): + mr.media_argv("audio", {}, None, "--rounds=2") + + +def test_operation_auto_routes_cpu_audio(tmp_path, store, monkeypatch): + from inferencekit.ops import bench_rounds + manifest, fixtures = write_run(tmp_path, "audio") + monkeypatch.setattr(bench_rounds, "gpu_processes", lambda: pytest.fail("CPU mode probed GPU")) + monkeypatch.setattr(bench_rounds, "gpu_identity", lambda: pytest.fail("CPU mode probed GPU")) + rec = registry.call("bench_rounds", {"manifest": str(manifest), "prompts": str(fixtures), "rounds": 1, + "warmup": 0, "startup_timeout_s": 5, "stop_timeout_s": 1, + "cpu_lock": str(tmp_path / "cpu.lock"), "serve_args": "--resamples 10"}) + assert rec["kind"] == "audio_benchmark" + assert rec["data"]["plan"]["resource_mode"] == "cpu" + assert rec["data"]["plan"]["audio_evidence_final"] + + +def test_cleanup_failure_stops_all_later_cells_and_retains_plan(tmp_path, store, monkeypatch): + manifest, fixtures = write_run(tmp_path, "audio") + calls = [] + def failed_cleanup(spec, k, position, cfg): + assert (tmp_path / "audio_rounds" / "plan.json").is_file() + calls.append(spec["label"]) + return {"label": spec["label"], "round": k, "position": position, "status": "stop_failed", + "stop": {"group_gone": False}} + monkeypatch.setattr(mr, "run_cell", failed_cleanup) + result = mr.run_media_rounds(str(manifest), str(fixtures), "audio", rounds=3, + cpu_lock=str(tmp_path / "cpu.lock")) + assert calls == ["base"] + assert all(c["status"] == "skipped_after_cleanup_failure" for c in result["cells"][1:]) + + +def test_conflicting_hardware_refused_and_gpu_count_forwarded(tmp_path): + from inferencekit.serve.round_evidence import resource_mode + with pytest.raises(OpError, match="conflicting"): + resource_mode({"hardware": {"backend": "cpu", "gpu": "GPU"}}) + ctx = {"base_url": "http://127.0.0.1:1", "prompts": "p", "model": "m", "label": "base", "first_round": 0, + "warmup": 0, "seed": 0, "device": "gpu", "gpu_count": 4, "hardware": "GPU"} + argv = mr.media_argv("image", ctx, None, None) + assert argv[argv.index("--gpu-count") + 1] == "4" + with pytest.raises(OpError, match="override"): + mr.media_argv("image", ctx, None, "--gpu-count 2") diff --git a/tests/test_quality_audio.py b/tests/test_quality_audio.py new file mode 100644 index 0000000..aa16771 --- /dev/null +++ b/tests/test_quality_audio.py @@ -0,0 +1,541 @@ +"""Speech quality uses synthetic PCM and imported observations only, never models.""" + +import copy +import json +import math +import shutil +import struct +import time + +import pytest + +from inferencekit import evidence as ev +from inferencekit.ops import quality_audio as op +from inferencekit.ops.bench_audio import load_fixtures, round_seed +from inferencekit.ops.report import build_report +from inferencekit.quality import audio as qa +from inferencekit.registry import Call, OpError, REGISTRY +from conftest import fake_call + + +def invoke(name, **kwargs): + operation = REGISTRY[name] + args = operation.validate(kwargs) + return operation.handler(Call(operation, args), **args) + + +def fixture_file(tmp_path, n=3, reference=False, texts=None): + path = tmp_path / "fixtures.jsonl" + texts = texts if texts is not None else ["The meeting starts tomorrow morning."] * n + rows = [{"id": f"f{i}", "text": text, "unseen": True} for i, text in enumerate(texts)] + if reference: + (tmp_path / "voice.wav").write_bytes(b"synthetic voice reference") + for row in rows: + row.update(reference_audio="voice.wav", reference_text="Reference voice.") + path.write_text("".join(json.dumps(row) + "\n" for row in rows)) + return path + + +def scorer(name): + value = {"name": name, "version": "test-1", "model": "synthetic-scorer", "revision": "test-pinned", + "config": {"device": "cpu", "vad_filter": True, "condition_on_previous_text": False}} + if name in ("naturalness", "speaker"): + value["scale"] = [1, 5] if name == "naturalness" else [-1, 1] + return value + + +def policy_file(tmp_path, acceptance="exact", retry=False, listening=False, speaker=False, text_metrics=("wer",)): + doc = {"acceptance": acceptance, "resamples": 100} + if acceptance != "exact": + doc.update(metrics={**{name: {"margin": 0.02, f"outlier_{name}": 1} for name in text_metrics}, + "naturalness": {"margin": 0.1}}, + scorers={"asr": scorer("asr"), "naturalness": scorer("naturalness")}) + if retry: + doc["scorers"]["asr_retry"] = scorer("asr_retry") + if speaker: + doc["metrics"]["speaker"] = {"margin": 0.02} + doc["scorers"]["speaker"] = scorer("speaker") + if listening: + doc["metrics"]["listening"] = {"margin": 0.1, "min_raters": 2} + doc["scorers"]["listening"] = {"name": "blinded_panel", "version": "1", + "protocol_sha256": "a" * 64, "scale": [1, 5], + "config": {"blinded": True}} + path = tmp_path / "policy.json" + path.write_text(json.dumps(doc)) + return path + + +def bench(store, fixtures, label, *, different=False, frames=16, rounds=2, keep=True, failed=False, start=None): + fixture_rows, fixture_sha = load_fixtures(fixtures) + config = {"sample_rate": 8000, "channels": 1, "seed_base": 0, "settings": {"temperature": 0.7}} + rows, outputs = [], [] + now = time.time() + 1 if start is None else start + round_rows = [{"index": r, "seed": round_seed(0, r), "t_start": now + r * 5, "t_end": now + r * 5 + 1} + for r in range(rounds)] + with store.staging_dir() as staging: + for r in range(rounds): + for fixture in fixture_rows: + pcm = struct.pack("<" + "h" * frames, *([101 if different else 100] * frames)) + row = {**fixture.identity(), "round": r, "seed": round_seed(0, r), "ok": not failed, + "sample_count": frames, "sample_rate": 8000, "channels": 1, + "pcm_sha256": qa.sha(pcm) if not failed else None, "pcm_bytes": len(pcm) if not failed else None} + rows.append(row) + if keep and not failed: + rel = f"outputs/{label}/r{r}/{fixture.id}.pcm" + target = staging / rel + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(pcm) + outputs.append({"path": rel, "sha256": qa.sha(pcm), "bytes": len(pcm), + "sample_rate": 8000, "channels": 1, "encoding": "pcm_s16le"}) + requests = staging / "requests.jsonl" + requests.write_text("".join(json.dumps(row) + "\n" for row in rows)) + index = staging / f"outputs/{label}/index.json" + index.parent.mkdir(parents=True, exist_ok=True) + index.write_text(json.dumps(outputs)) + artifacts = [ev.artifact(requests, staging, "requests"), ev.artifact(index, staging, "outputs_index")] + artifacts.extend(ev.artifact(staging / x["path"], staging, "audio_pcm") for x in outputs) + rec = ev.build(fake_call(), kind="audio_benchmark", claim_class="observed", summary="synthetic speech", + identity_=ev.identity(engine={"name": label}, model={"id": "test", "revision": "abc", "dtype": "fp32", "quant": "none"}), + artifacts=artifacts, data={"source": "bench audio", "label": label, + "fixture_set": {"sha256": fixture_sha, "fixtures": [f.identity() for f in fixture_rows]}, + "rounds": round_rows, "audio_config": config, "total_requests": len(rows), + "structural_verdict": "FAIL" if failed else "PASS", "quality_verdict": "UNPROVEN"}) + return store.save(rec, staging) + + +def setup_run(tmp_path, store, acceptance="exact", *, retry=False, listening=False, reference=False, + text_metrics=("wer",), texts=None, **candidate_args): + fixtures = fixture_file(tmp_path, reference=reference, texts=texts) + protocol = invoke("quality_audio_policy", policy=str(policy_file(tmp_path, acceptance, retry, listening, reference, text_metrics)), + fixtures=str(fixtures)) + baseline = bench(store, fixtures, "baseline") + candidate = bench(store, fixtures, "candidate", **candidate_args) + return fixtures, protocol, baseline, candidate + + +def score_file(tmp_path, store, run, *, candidate_text=None, retry_text=None, mos=4.0, listening=False, speaker=False): + fixtures, protocol, baseline, candidate = run + texts = {f.id: f.text for f in load_fixtures(fixtures)[0]} + rows = [] + for arm, source in (("baseline", baseline), ("candidate", candidate)): + _, requests = qa.load_benchmark(store, source["evidence_id"]) + for (fid, round_index), row in requests.items(): + asr = {"text": candidate_text if arm == "candidate" and candidate_text is not None else texts[fid]} + if retry_text is not None: + asr["retry_text"] = retry_text if arm == "candidate" else texts[fid] + item = {"arm": arm, "id": fid, "round": round_index, "seed": row["seed"], + "pcm_sha256": row["pcm_sha256"], "text_sha256": row["text_sha256"], "asr": asr, + "naturalness": {"windows": [{"start_frame": 0, "end_frame": row["sample_count"], + "score": mos if arm == "candidate" else 4.0}]}} + if listening: + item["listening"] = {"ratings": [{"rater": "anon-1", "score": 4.0}, {"rater": "anon-2", "score": 4.0}]} + if speaker: + item["speaker"] = {"score": 0.8, "reference_sha256": row["reference_sha256"]} + rows.append(item) + doc = {"format": qa.SCORES_FORMAT, "protocol": protocol["evidence_id"], "baseline": baseline["evidence_id"], + "candidate": candidate["evidence_id"], "scorers": protocol["data"]["policy"]["scorers"], "rows": rows} + path = tmp_path / "scores.json" + path.write_text(json.dumps(doc)) + return path + + +def quality(run, scores=None): + fixtures, protocol, baseline, candidate = run + return invoke("quality_audio", baseline=baseline["evidence_id"], candidate=candidate["evidence_id"], + fixtures=str(fixtures), protocol=protocol["evidence_id"], scores=str(scores) if scores else None) + + +def test_parity_signed_stereo_and_no_truncation(): + a = struct.pack("<4h", -1000, 1000, -2000, 2000) + b = struct.pack("<4h", -1001, 1001, -2002, 2002) + result = qa.parity(a, b, 2) + assert result["baseline_frames"] == 2 + assert result["maxabs"] == 2 / 32768 + assert result["snr_db"] == pytest.approx(60) + assert qa.parity(a, a, 2)["snr_status"] == "identical" + assert qa.parity(a, b[:-4], 2)["snr_db"] is None + assert qa.parity(a, b[:-4], 2)["maxabs"] is None + + +def test_wer_is_recomputed_not_capped_or_silently_normalized(): + assert qa.word_error("Hello, WORLD!", "hello world", "nfkc_casefold_punctuation_v1")["errors"] == 0 + assert qa.word_error("hello", "hello extra extra extra", "whitespace_v1")["wer"] == 3 + assert qa.word_error("a b c", "a x", "whitespace_v1")["errors"] == 2 + with pytest.raises(OpError, match="no words"): + qa.word_error("!!!", "speech", "nfkc_casefold_punctuation_v1") + + +@pytest.mark.parametrize("reference,hypothesis,count", [ + ("你好世界", "你好世间", 4), + ("こんにちは", "こんにちわ", 5), + ("แมว", "แม", 3), + ("سلام", "سلا", 4), + ("कि", "क", 2), # A remaining combining mark counts as a code point, not a grapheme. +]) +def test_cer_uses_unicode_code_points_without_word_boundaries(reference, hypothesis, count): + result = qa.character_error(reference, hypothesis, "nfkc_casefold_punctuation_v1") + assert result == {"errors": 1, "characters": count, "cer": 1 / count, + "hypothesis_characters": len(hypothesis)} + + +def test_cer_normalization_whitespace_punctuation_and_uncapped_insertions(): + normalized = qa.character_error("ＣＡＦＥ\u0301，Straße!", "café strasse", "nfkc_casefold_punctuation_v1") + assert normalized["cer"] == 0 + assert normalized["characters"] == 11 + assert qa.character_error(" A\tB\u00a0C\n", "ABC", "whitespace_v1")["cer"] == 0 + assert qa.character_error("a!", "A", "whitespace_v1")["errors"] == 2 + assert qa.character_error("é", "e\u0301", "whitespace_v1")["cer"] == 2 + assert qa.character_error("你", "你好世界", "whitespace_v1")["cer"] == 3 + assert qa.character_error("你好", "", "whitespace_v1")["cer"] == 1 + with pytest.raises(OpError, match="no characters"): + qa.character_error("！\t", "你好", "nfkc_casefold_punctuation_v1") + + +@pytest.mark.parametrize("text_metrics", [("cer",), ("wer", "cer")]) +def test_cer_policy_and_report_preserve_independent_fixture_clustered_metrics(tmp_path, store, text_metrics): + run = setup_run(tmp_path, store, "task_noninferiority", text_metrics=text_metrics, + texts=["你好", "你好世界", "今天阳光灿烂"]) + path = score_file(tmp_path, store, run) + good = quality(run, path) + assert good["data"]["verdict"] == "PASS" + assert build_report([good], "improved", "quality")["verdict"] == "PASS" + assert set(good["data"]["scorer_provenance"]) == {"asr", "naturalness"} + check = good["data"]["checks"]["cer"] + assert check["normalization"] == {"name": "nfkc_casefold_punctuation_v1", "unit": "unicode_code_point", + "whitespace": "removed"} + assert check["n_prompts"] == 3 + assert check["ci95"] == [0.0, 0.0] + assert all(m["unit"] == "fraction" for m in good["metrics"] if m["name"].startswith("audio_cer")) + + doc = json.loads(path.read_text()) + for row in doc["rows"]: + if row["arm"] == "candidate" and row["round"] == 0: + row["asr"]["text"] = row["asr"]["text"][:-1] + "错" + path.write_text(json.dumps(doc)) + bad = quality(run, path) + check = bad["data"]["checks"]["cer"] + expected = op.qm.paired_delta_ci([(1, 4), (1, 8), (1, 12)], [(0, 4), (0, 8), (0, 12)], 100, 0) + assert check["candidate"] == pytest.approx(3 / 24) + assert check["baseline"] == 0 + assert check["n_prompts"] == 3 + assert check["ci95"] == list(expected["ci95"]) + assert check["worse_ci95"] == check["ci95"] # Higher error is worse. + assert bad["data"]["verdict"] == "FAIL" + assert build_report([bad], "improved", "quality")["verdict"] == "FAIL" + if "wer" in text_metrics: + assert bad["data"]["checks"]["wer"]["candidate"] == 0.5 + + +@pytest.mark.parametrize("failure", ["missing_row", "failed_asr", "all_asr_failed", "empty_transcript"]) +def test_cer_incomplete_and_failed_observations_never_become_zero_or_pass(tmp_path, store, failure): + run = setup_run(tmp_path, store, "task_noninferiority", text_metrics=("cer",)) + path = score_file(tmp_path, store, run) + doc = json.loads(path.read_text()) + if failure == "missing_row": + doc["rows"].pop() + else: + rows = doc["rows"] if failure in ("all_asr_failed", "empty_transcript") else doc["rows"][-1:] + for row in rows: + if failure == "empty_transcript": + if row["arm"] == "candidate": + row["asr"]["text"] = "" + else: + del row["asr"] + row["error"] = "ASR failed" + path.write_text(json.dumps(doc)) + result = quality(run, path) + expected = "FAIL" if failure == "empty_transcript" else "UNPROVEN" + assert result["data"]["checks"]["cer"]["verdict"] == expected + assert result["data"]["verdict"] == expected + assert build_report([result], "improved", "quality")["verdict"] == expected + if failure == "all_asr_failed": + assert result["data"]["checks"]["cer"]["candidate"] is None + assert all(m["value"] is None for m in result["metrics"] if m["name"].startswith("audio_cer")) + elif failure == "empty_transcript": + assert result["data"]["checks"]["cer"]["candidate"] == 1 + + +def test_cer_and_wer_outliers_are_corroborated_independently_without_selecting_best(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority", text_metrics=("wer", "cer"), retry=True) + hallucination = "错" * 200 # One word: WER stays below its outlier threshold; CER does not. + path = score_file(tmp_path, store, run, candidate_text=hallucination) + result = quality(run, path) + assert not result["data"]["checks"]["wer"]["outliers"] + assert result["data"]["checks"]["wer"]["verdict"] == "FAIL" + cer = result["data"]["checks"]["cer"] + assert cer["primary_verdict"] == "FAIL" + assert cer["verdict"] == "UNPROVEN" + assert len(cer["retry_missing"]) == 6 + assert cer["candidate"] > 1 + primary = cer["candidate"] + + path = score_file(tmp_path, store, run, candidate_text=hallucination, + retry_text="The meeting starts tomorrow morning.") + result = quality(run, path) + cer = result["data"]["checks"]["cer"] + assert cer["candidate"] == primary + assert cer["retry"]["candidate"] == 0 + assert cer["verdict"] == "UNPROVEN" + assert "retry" not in result["data"]["checks"]["wer"] + + path = score_file(tmp_path, store, run, candidate_text=hallucination, retry_text=hallucination) + assert quality(run, path)["data"]["checks"]["cer"]["verdict"] == "FAIL" + + +@pytest.mark.parametrize("mutation", ["missing_asr", "missing_task_metric", "missing_naturalness", + "missing_outlier", "zero_outlier", "negative_margin", "wrong_outlier"]) +def test_cer_requires_registered_task_margin_outliers_and_asr(tmp_path, mutation): + doc = json.loads(policy_file(tmp_path, "task_noninferiority", text_metrics=("cer",)).read_text()) + if mutation == "missing_asr": + del doc["scorers"]["asr"] + elif mutation == "missing_task_metric": + del doc["metrics"]["cer"] + elif mutation == "missing_naturalness": + del doc["metrics"]["naturalness"] + elif mutation == "missing_outlier": + del doc["metrics"]["cer"]["outlier_cer"] + elif mutation == "zero_outlier": + doc["metrics"]["cer"]["outlier_cer"] = 0 + elif mutation == "negative_margin": + doc["metrics"]["cer"]["margin"] = -0.1 + else: + doc["metrics"]["cer"]["outlier_wer"] = doc["metrics"]["cer"].pop("outlier_cer") + with pytest.raises(OpError): + qa.validate_policy(doc) + + +def test_exact_preservation_and_changed_pcm_fail(tmp_path, store): + run = setup_run(tmp_path, store) + good = quality(run) + assert good["data"]["verdict"] == "PASS" + assert "baseline outputs" in good["data"]["scope"] + assert ev.validate(good) == [] + assert build_report([good], "improved", "quality")["verdict"] == "PASS" + changed = bench(store, run[0], "candidate", different=True) + bad = quality((*run[:3], changed)) + assert bad["data"]["verdict"] == "FAIL" + assert bad["data"]["pairs"][0]["snr_db"] == pytest.approx(40) + + +def test_missing_pcm_is_unproven_and_never_zero(tmp_path, store): + run = setup_run(tmp_path, store, keep=False) + result = quality(run) + assert result["data"]["verdict"] == "UNPROVEN" + assert next(m for m in result["metrics"] if m["name"] == "bit_exact_pct")["value"] is None + + +def test_failed_source_cannot_become_quality_pass(tmp_path, store): + run = setup_run(tmp_path, store, failed=True) + assert quality(run)["data"]["verdict"] == "FAIL" + + +def test_posthoc_policy_cannot_pass(tmp_path, store): + run = setup_run(tmp_path, store, start=0) + result = quality(run) + assert result["data"]["verdict"] == "UNPROVEN" + assert result["data"]["checks"]["parity"]["verdict"] == "PASS" + assert not result["data"]["preregistered"] + + +def test_proxy_policy_allows_nonidentical_pcm_without_claiming_listening(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority", different=True) + result = quality(run, score_file(tmp_path, store, run)) + assert result["data"]["verdict"] == "PASS" + assert result["data"]["checks"]["parity"]["verdict"] == "FAIL" + assert result["data"]["checks"]["parity"]["required"] is False + assert "not human listening" in result["data"]["scope"] + assert result["data"]["checks"]["wer"]["n_prompts"] == 3 # not six fixture-rounds + assert result["data"]["checks"]["wer"]["ci95"] == [0.0, 0.0] + + +def test_missing_and_failed_metrics_preserve_verdicts(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority") + assert quality(run)["data"]["verdict"] == "UNPROVEN" + path = score_file(tmp_path, store, run, mos=3.0) + assert quality(run, path)["data"]["verdict"] == "FAIL" + path = score_file(tmp_path, store, run) + doc = json.loads(path.read_text()) + doc["rows"].pop() + path.write_text(json.dumps(doc)) + assert quality(run, path)["data"]["verdict"] == "UNPROVEN" + + +@pytest.mark.parametrize("mutation", ["duplicate", "pcm", "text", "scorer", "window_gap", "window_overlap", "nan"]) +def test_invalid_import_rejected(tmp_path, store, mutation): + run = setup_run(tmp_path, store, "task_noninferiority") + path = score_file(tmp_path, store, run) + doc = json.loads(path.read_text()) + if mutation == "duplicate": + doc["rows"].append(doc["rows"][0]) + elif mutation in ("pcm", "text"): + doc["rows"][0][f"{mutation}_sha256"] = "0" * 64 + elif mutation == "scorer": + doc["scorers"]["asr"]["revision"] = "other" + elif mutation.startswith("window"): + window = doc["rows"][0]["naturalness"]["windows"][0] + if mutation == "window_gap": + window["end_frame"] -= 1 + else: + doc["rows"][0]["naturalness"]["windows"].append(window) + else: + doc["rows"][0]["naturalness"]["windows"][0]["score"] = math.nan + path.write_text(json.dumps(doc)) + with pytest.raises(OpError): + quality(run, path) + + +def test_pcm_tampering_is_refused(tmp_path, store): + run = setup_run(tmp_path, store) + candidate = run[-1] + item = next(a for a in candidate["artifacts"] if a["role"] == "audio_pcm") + (store.run_dir(candidate["evidence_id"]) / item["path"]).write_bytes(b"changed") + with pytest.raises(OpError, match="missing, changed"): + quality(run) + + +def test_asr_outliers_never_choose_best_decode_or_use_snr_to_waive_wer(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority", retry=True) + hallucination = "invented " * 20 + path = score_file(tmp_path, store, run, candidate_text=hallucination) + result = quality(run, path) + assert result["data"]["checks"]["wer"]["primary_verdict"] == "FAIL" + assert result["data"]["verdict"] == "UNPROVEN" + path = score_file(tmp_path, store, run, candidate_text=hallucination, retry_text="The meeting starts tomorrow morning.") + assert quality(run, path)["data"]["verdict"] == "UNPROVEN" + path = score_file(tmp_path, store, run, candidate_text=hallucination, retry_text=hallucination) + assert quality(run, path)["data"]["verdict"] == "FAIL" + + +def test_length_weighting_and_complete_tail(): + value = {"windows": [{"start_frame": 0, "end_frame": 75, "score": 4}, + {"start_frame": 75, "end_frame": 100, "score": 2}]} + assert qa.window_score(value, 100, scorer("naturalness")) == 3.5 + + +def test_perceptual_policy_requires_blinded_paired_listeners(tmp_path, store): + run = setup_run(tmp_path, store, "perceptual_and_task_noninferiority", listening=True) + assert quality(run, score_file(tmp_path, store, run))["data"]["verdict"] == "UNPROVEN" + path = score_file(tmp_path, store, run, listening=True) + assert quality(run, path)["data"]["verdict"] == "PASS" + doc = json.loads(path.read_text()) + doc["rows"][-1]["listening"]["ratings"][0]["rater"] = "unpaired" + path.write_text(json.dumps(doc)) + assert quality(run, path)["data"]["verdict"] == "UNPROVEN" + + +def test_speaker_score_binds_reference_audio(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority", reference=True) + path = score_file(tmp_path, store, run, speaker=True) + assert quality(run, path)["data"]["verdict"] == "PASS" + doc = json.loads(path.read_text()) + doc["rows"][0]["speaker"]["reference_sha256"] = "0" * 64 + path.write_text(json.dumps(doc)) + with pytest.raises(OpError, match="voice reference"): + quality(run, path) + + +def test_report_retains_missing_quality_observations(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority") + result = quality(run, score_file(tmp_path, store, run)) + root = store.run_dir(result["evidence_id"]) + artifact = next(a for a in result["artifacts"] if a["role"] == "audio_quality_observations") + (root / artifact["path"]).unlink() + assert build_report([result], "improved", "quality")["verdict"] == "UNPROVEN" + + +def test_run_directory_retains_policy_and_observations_without_store(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority") + path = score_file(tmp_path, store, run) + out = tmp_path / "portable" + result = invoke("quality_audio", baseline=run[2]["evidence_id"], candidate=run[3]["evidence_id"], + fixtures=str(run[0]), protocol=run[1]["evidence_id"], scores=str(path), run_dir=str(out)) + assert all((out / a["path"]).is_file() for a in result["artifacts"]) + shutil.rmtree(store.run_dir(result["evidence_id"])) + assert op.report_verdict(result)[0] == "PASS" + policy = invoke("quality_audio_policy", policy=str(policy_file(tmp_path)), fixtures=str(run[0]), run_dir=str(out)) + loaded, root = qa.load_record(store, str(out / "evidence" / f"{policy['evidence_id']}.json"), "quality_audio") + assert qa.artifact_bytes(root, qa.one_artifact(loaded, "audio_quality_policy")) + + +def test_longer_output_parity_has_no_truncated_score(tmp_path, store): + run = setup_run(tmp_path, store, "task_noninferiority", frames=20) + result = quality(run, score_file(tmp_path, store, run)) + assert result["data"]["verdict"] == "PASS" + assert all(pair["snr_status"] == "length_mismatch" for pair in result["data"]["pairs"]) + assert all(pair["maxabs"] is None for pair in result["data"]["pairs"]) + + +def comparison(store, baseline, candidate): + return store.save(ev.build(fake_call(), kind="comparison", claim_class="derived", summary="synthetic timing", + data={"source_kind": "audio_benchmark", "baseline": baseline["evidence_id"], + "candidate": candidate["evidence_id"], "verdict": "improved", "interleaved": True, + "required_proofs": {"media quality": "UNPROVEN", "fresh-process lifecycle": "UNPROVEN"}})) + + +def finalize_sources(store, run, planned_rounds): + from inferencekit.serve.round_evidence import finalize_arm_records + + cells = [{"label": arm, "round": r, "status": "ok" if r <= 2 else "skipped_after_cleanup_failure", + "ready": {"status": "ready"}, "stop": {"group_gone": True}, "server_env": {}} + for arm in ("baseline", "candidate") for r in range(1, planned_rounds + 1)] + final = finalize_arm_records({"baseline": run[2]["evidence_id"], "candidate": run[3]["evidence_id"]}, + cells, {"rounds": planned_rounds, "design": {"design": "Williams", "balanced": True}}) + return (*run[:2], store.load(final["baseline"]), store.load(final["candidate"])) + + +def test_matching_scheduler_prefix_cannot_pass_quality(tmp_path, store): + run = finalize_sources(store, setup_run(tmp_path, store), planned_rounds=4) + result = quality(run) + assert result["data"]["verdict"] == "UNPROVEN" + assert result["coverage"]["complete"] is False + assert result["data"]["checks"]["round_coverage"]["verdict"] == "UNPROVEN" + assert result["data"]["checks"]["round_coverage"]["arms"]["baseline"]["missing_indices"] == [2, 3] + assert all(pair["bit_exact"] for pair in result["data"]["pairs"]) + compared = comparison(store, run[2], run[3]) + report = build_report([compared, result], "improved", "quality") + assert all(c["verdict"] == "UNPROVEN" for c in report["claims"] + if c["claim"] == "speech quality preservation against baseline") + + +def test_final_source_ids_and_original_round_times_bind_quality(tmp_path, store): + raw = setup_run(tmp_path, store) + raw_quality = quality(raw) + final = finalize_sources(store, raw, planned_rounds=2) + compared = comparison(store, final[2], final[3]) + unbound = build_report([compared, raw_quality], "improved", "quality") + assert next(c for c in unbound["claims"] if c["claim"] == "media quality")["verdict"] == "UNPROVEN" + final_quality = quality(final) + assert final_quality["data"]["preregistered"] is True + assert final_quality["data"]["verdict"] == "PASS" + bound = build_report([compared, final_quality], "improved", "quality") + assert next(c for c in bound["claims"] if c["source"] == compared["evidence_id"] and + c["claim"] == "speech quality preservation against baseline")["verdict"] == "PASS" + assert final[3]["data"]["quality_verdict"] == "UNPROVEN" # source timing record is immutable + late = finalize_sources(store, (*raw[:3], bench(store, raw[0], "candidate", start=0)), planned_rounds=2) + assert quality(late)["data"]["preregistered"] is False + assert quality(late)["data"]["verdict"] == "UNPROVEN" + + +def test_report_attaches_exact_pair_and_retains_lifecycle_and_failure(tmp_path, store): + run = setup_run(tmp_path, store) + good = quality(run) + compare = comparison(store, run[2], run[3]) + report = build_report([compare, good], "improved", "quality") + assert report["verdict"] == "UNPROVEN" + linked = [c for c in report["claims"] if c["source"] == compare["evidence_id"]] + assert next(c for c in linked if c["claim"] == "speech quality preservation against baseline")["verdict"] == "PASS" + assert next(c for c in linked if c["claim"] == "fresh-process lifecycle")["verdict"] == "UNPROVEN" + unrelated = copy.deepcopy(compare) + unrelated["data"]["candidate"] = "ev_" + "0" * 64 + unrelated = store.save(unrelated) + report = build_report([unrelated, good], "improved", "quality") + assert next(c for c in report["claims"] if c["claim"] == "media quality")["verdict"] == "UNPROVEN" + changed = bench(store, run[0], "candidate", different=True) + failed = quality((*run[:3], changed)) + assert build_report([compare, good, failed], "improved", "quality")["verdict"] == "FAIL" + + +@pytest.mark.parametrize("field,value", [("resamples", 0), ("min_fixtures", 1), ("seed", -1), ("min_rounds", True)]) +def test_invalid_registered_plan_is_rejected(field, value): + with pytest.raises(OpError): + qa.validate_policy({"acceptance": "exact", field: value}) diff --git a/tests/test_report_resources.py b/tests/test_report_resources.py new file mode 100644 index 0000000..b4cb5a2 --- /dev/null +++ b/tests/test_report_resources.py @@ -0,0 +1,29 @@ +"""Both arms' observed memory and requested CPU controls remain readable in a report.""" +from inferencekit import evidence as ev, registry +from inferencekit.ops.report import build_report, to_markdown +from conftest import make_benchmark + + +def test_comparison_environment_card_has_both_cpu_arms_and_memory(store): + arms = [] + for label, threads, memory, start in (("base", "2", 80, 0), ("candidate", "4", 160, 5)): + record = make_benchmark(store, [100] * 4, t0=start) + record["identity"]["gpu"] = ev.identity()["gpu"] + record["identity"]["cpu"] = {"model": "Fixture CPU"} + record["identity"]["engine"].update(name=label, flags={"server_env": { + "controls": {"OMP_NUM_THREADS": threads}, "ld_preload": [], "omitted": []}}) + record["data"]["memory"] = {"peak_rss_mib": memory, "method": "sampled_process_group_rss", + "complete": label == "base", "verdict": "PASS" if label == "base" else "UNPROVEN"} + arms.append(store.save(record)) + compared = registry.call("compare", {"baseline": arms[0]["evidence_id"], "candidate": arms[1]["evidence_id"], + "resamples": 100}) + report = build_report([compared], "improved", "CPU report") + rows = {r["engine"]: r for r in report["environment"]} + assert rows["base"]["hardware"] == rows["candidate"]["hardware"] == "CPU: Fixture CPU" + assert rows["base"]["server tuning (requested)"] == "OMP_NUM_THREADS=2" + assert rows["candidate"]["server tuning (requested)"] == "OMP_NUM_THREADS=4" + assert "80" in rows["base"]["peak memory"] and "160" in rows["candidate"]["peak memory"] + assert "partial" in rows["candidate"]["peak memory"] + markdown = to_markdown(report) + assert "peak memory" in markdown and "server tuning (requested)" in markdown + assert "sampled_process_group_rss" in markdown diff --git a/tests/test_resources.py b/tests/test_resources.py new file mode 100644 index 0000000..6e8be3c --- /dev/null +++ b/tests/test_resources.py @@ -0,0 +1,119 @@ +"""CPU telemetry uses the server group, bounded proc reads, and safe tuning metadata.""" + +import hashlib +import json + +import pytest + +from inferencekit.serve import resources + + +def write_process(root, pid, *, rss=1024, hwm=2048, threads=2, affinity="0-3", task_affinity=None): + directory = root / str(pid) + directory.mkdir(exist_ok=True) + text = f"Name:\ttest\nVmRSS:\t{rss} kB\nVmHWM:\t{hwm} kB\nThreads:\t{threads}\nCpus_allowed_list:\t{affinity}\n" + (directory / "status").write_text(text) + task = directory / "task" / str(pid) + task.mkdir(parents=True, exist_ok=True) + (task / "status").write_text(text.replace(affinity, task_affinity or affinity)) + + +def test_runtime_env_only_persists_valid_controls_and_library_identity(tmp_path): + library = tmp_path / "private-directory" / "libcpu.so.1" + library.parent.mkdir() + library.write_bytes(b"synthetic library") + result = resources.runtime_env({"OMP_NUM_THREADS": "8", "MKL_NUM_THREADS": "4", "OMP_PROC_BIND": "close", + "GOMP_CPU_AFFINITY": "0-7:2 9", "OMP_PLACES": "{0:4},{8:4}", + "LD_PRELOAD": str(library), "HF_TOKEN": "secret-value", + "PRIVATE_TOKEN": "secret-value", "LD_LIBRARY_PATH": "/private/library/path", + "KMP_AFFINITY": "granularity=fine,compact,1,0"}) + assert result["controls"]["OMP_NUM_THREADS"] == "8" + assert result["ld_preload"] == [{"name": "libcpu.so.1", "status": "hashed", + "sha256": hashlib.sha256(b"synthetic library").hexdigest()}] + encoded = json.dumps(result) + assert "secret-value" not in encoded and "private-directory" not in encoded and "/private" not in encoded + assert "HF_TOKEN" not in encoded and "LD_LIBRARY_PATH" not in encoded + + +def test_bad_control_values_and_preload_paths_are_not_retained(): + result = resources.runtime_env({"OMP_NUM_THREADS": "some-secret-value", "KMP_AFFINITY": "/private/key", + "LD_PRELOAD": "/private/key.txt", "OMP_DYNAMIC": "FALSE"}) + assert result["controls"] == {"KMP_AFFINITY": None, "OMP_DYNAMIC": "false", "OMP_NUM_THREADS": None} + assert result["omitted"] == ["KMP_AFFINITY", "OMP_NUM_THREADS"] + assert result["ld_preload"][0]["name"] is None + assert "some-secret-value" not in json.dumps(result) and "/private" not in json.dumps(result) + + +def test_preload_file_read_is_bounded(tmp_path, monkeypatch): + library = tmp_path / "liblarge.so" + library.write_bytes(b"large") + monkeypatch.setattr(resources, "MAX_PRELOAD_BYTES", 2) + assert resources.runtime_env({"LD_PRELOAD": str(library)})["ld_preload"][0]["status"] == "size_limit" + + +def test_relative_preload_uses_actual_server_cwd_without_exposing_paths(tmp_path, monkeypatch): + for name in ("first", "second"): + directory = tmp_path / name + directory.mkdir() + (directory / "libcpu.so").write_bytes(name.encode()) + monkeypatch.chdir(tmp_path) + first = resources.runtime_env({"LD_PRELOAD": "./libcpu.so"}, cwd="first") + second = resources.runtime_env({"LD_PRELOAD": "./libcpu.so"}, cwd=tmp_path / "second") + assert first["ld_preload"][0]["sha256"] == hashlib.sha256(b"first").hexdigest() + assert second["ld_preload"][0]["sha256"] == hashlib.sha256(b"second").hexdigest() + assert first["ld_preload"][0]["name"] == second["ld_preload"][0]["name"] == "libcpu.so" + assert "first" not in json.dumps(first) and "second" not in json.dumps(second) + result = resources.ProcessMonitor(11, {"LD_PRELOAD": "./libcpu.so"}, cwd="first").stop() + assert result["requested"] == first + # A bare library name follows loader search paths, not cwd; stay unknown. + soname = resources.runtime_env({"LD_PRELOAD": "libcpu.so"}, cwd="first") + assert soname["ld_preload"][0]["sha256"] is None + + +def test_group_peak_is_not_sum_of_individual_high_water_marks(tmp_path, monkeypatch): + monkeypatch.setattr(resources, "PROC_ROOT", tmp_path) + monkeypatch.setattr(resources.sys, "platform", "linux") + monkeypatch.setattr(resources.servers, "group_members", lambda pgid: [11, 12]) + write_process(tmp_path, 11, rss=10 * 1024, hwm=15 * 1024, threads=3) + write_process(tmp_path, 12, rss=20 * 1024, hwm=24 * 1024, threads=4, affinity="4-7") + monitor = resources.ProcessMonitor(11, {"OMP_NUM_THREADS": "3"}, poll_s=60).start() + write_process(tmp_path, 11, rss=12 * 1024, hwm=15 * 1024, threads=5, affinity="1-2", task_affinity="2") + write_process(tmp_path, 12, rss=1024, hwm=24 * 1024, threads=4, affinity="4-7") + monitor.settled() + result = monitor.stop() + assert result["peak_rss_mib"] == 30 + assert result["peak_threads"] == 9 + assert result["observations"]["launch"]["rss_mib"] == 30 + assert result["observations"]["settled"]["rss_mib"] == 13 + assert result["observations"]["settled"]["affinity"] == ["2", "4-7"] + assert [p["vm_hwm_mib"] for p in result["processes"]] == [15, 24] + assert result["requested"]["controls"]["OMP_NUM_THREADS"] == "3" + assert monitor.stop() == result # no additional final sample on repeated stop + assert any("not a simultaneous" in note for note in result["limitations"]) + + +def test_unavailable_proc_never_reports_zero_memory(tmp_path, monkeypatch): + monkeypatch.setattr(resources.sys, "platform", "darwin") + monkeypatch.setattr(resources.servers, "group_members", lambda _: pytest.fail("non-Linux must not read proc")) + result = resources.ProcessMonitor(11, {}).start().settled().stop() + assert result["status"] == "unavailable" + assert result["peak_rss_mib"] is None and result["peak_threads"] is None + assert result["samples"] == 0 + + +def test_process_limit_and_unreadable_status_are_explicit(tmp_path, monkeypatch): + monkeypatch.setattr(resources, "PROC_ROOT", tmp_path) + monkeypatch.setattr(resources.sys, "platform", "linux") + monkeypatch.setattr(resources, "MAX_PROCESSES", 2) + monkeypatch.setattr(resources.servers, "group_members", lambda _: [11, 12, 13]) + write_process(tmp_path, 11) + result = resources.ProcessMonitor(11, {}, poll_s=60).start().stop() + assert result["peak_rss_mib"] == 1 + assert any("sampling limit" in note for note in result["limitations"]) + assert any("unreadable" in note for note in result["limitations"]) + + +@pytest.mark.parametrize("pgid,poll_s", [(0, .05), (True, .05), (10, 0), (10, float("nan"))]) +def test_monitor_rejects_invalid_parameters(pgid, poll_s): + with pytest.raises(ValueError): + resources.ProcessMonitor(pgid, {}, poll_s) diff --git a/tests/test_resume.py b/tests/test_resume.py new file mode 100644 index 0000000..28d5a20 --- /dev/null +++ b/tests/test_resume.py @@ -0,0 +1,750 @@ +"""Resume checks must stay read-only and must never turn missing evidence into proof.""" + +import json +from pathlib import Path + +import pytest + +from inferencekit import cli, evidence as ev, registry +from inferencekit.mcp_server import fit +from inferencekit.registry import OpError + +from conftest import fake_call + + +NOW = "2026-01-01T01:00:00Z" + + +def write(path, data): + path.write_text(json.dumps(data)) + + +@pytest.fixture +def run(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + original = tmp_path / "original.json" + write(original, { + "model": {"id": "org/model", "revision": "a" * 40, "precision": "fp8"}, + "modality": "llm", "hardware": {"gpu": "Test GPU", "count": 1, "access": {"kind": "local"}}, + "budget": {"wall_clock_h": 4, "approved": True, "stage_timeboxes": False, "report_reserve_h": 0}, + }) + root = tmp_path / "runs" / "resume-test" + registry.call("intake_validate", {"file": str(original), "run_dir": str(root)}) + state = json.loads((root / "state.json").read_text()) + state["budget"]["run_started"] = "2026-01-01T00:00:00Z" + state["budget"]["stage_timeboxes"] = False + state["budget"]["report_reserve_h"] = 0 + state["current_stage"] = "4_tuning" + state["stages"]["4_tuning"]["status"] = "in_progress" + state["stages"]["4_tuning"]["started"] = "2026-01-01T00:00:00Z" + state["stages"]["3_candidates"]["status"] = "done" + state["next_action"] = "screen the pending candidate; reuse completed baseline" + store = ev.Store(root / "evidence") + rec = store.save(ev.build(fake_call(), kind="benchmark", claim_class="observed", summary="baseline screen", + data={"verdict": "UNPROVEN"}, missing=["full quality proof"], + limitations=["screening only"])) + state["stages"]["3_candidates"]["evidence"] = [rec["evidence_id"]] + state["candidates"]["baseline"] = {"evidence": [rec["evidence_id"]]} + state["best_history"] = [{"checks": "screened", "evidence": [rec["evidence_id"]]}] + write(root / "state.json", state) + return root, original, state, rec + + +def resume(root, **args): + return registry.call("intake_resume", {"run_dir": str(root), "now": NOW, **args}) + + +def snapshot(root): + return {str(p): p.read_bytes() for p in root.rglob("*") if p.is_file()} + + +def test_resume_checks_deduplicated_evidence_and_preserves_state_and_incomplete_proof(run): + root, _, state, rec = run + before = snapshot(root.parent.parent) + result = resume(root) + assert snapshot(root.parent.parent) == before + assert result["status"] == "context_ready" and result["evidence_summary"]["invalid"] == 0 + assert result["current_stage"] == state["current_stage"] and result["next_action"] == state["next_action"] + assert result["stages"]["3_candidates"]["status"] == "done" + assert result["evidence_root"] == str(root / "evidence") + assert len(result["items"]) == 1 + row = result["items"][0] + assert row["evidence_id"] == rec["evidence_id"] and row["status"] == "available" + assert row["integrity"] == {"id_ok": True, "schema_ok": True, "schema_errors": [], "cited_id_ok": True} + assert row["coverage"] == {"complete": False, "missing": ["full quality proof"]} + assert row["limitations"] == ["screening only"] + assert row["cited_by"] == ["stages.3_candidates.evidence", "candidates.baseline.evidence", + "best_history.0.evidence"] + assert "not a benchmark or quality proof" in result["scope"] + + +def test_resume_matches_existing_budget_and_keeps_explicit_overrides(run): + root, _, _, _ = run + result = resume(root) + expected = registry.call("intake_budget", {"run_dir": str(root), "now": NOW}) + assert {k: v for k, v in result["budget"].items() if k != "basis"} == { + k: v for k, v in expected.items() if k != "run_dir"} + assert "mutable checkpoint" in result["budget"]["basis"] + assert "not verified authorization" in result["budget"]["basis"] + assert result["budget"]["stage"]["time_box_h"] is None + assert result["budget"]["cap"]["report_reserve_h"] == 0 + assert result["budget"]["verdict"] == "ok" + + +def test_resume_policy_brief_uses_validated_intake_without_inferring_session_authorization(run): + root, original, _, _ = run + validated = registry.call("intake_validate", {"file": str(original)}) + result = resume(root) + assert result["policy"] == { + "source": "validated intake only; session authorization is not evaluated", + "budget_approved": validated["intake"]["budget"]["approved"], + "fidelity": validated["intake"]["fidelity"], + "quality": validated["intake"]["quality"], + "architecture_changes": validated["derived"]["architecture_changes"], + } + + +def test_resume_keeps_over_budget_action(run): + root, _, _, _ = run + result = resume(root, now="2026-01-02T00:00:00Z") + assert result["status"] == "needs_attention" + assert result["budget"]["verdict"] == "over_budget" + assert result["budget"]["action"].startswith("stop GPU work now") + + +def test_missing_evidence_is_reported_even_when_global_store_has_it(run, monkeypatch, tmp_path): + root, _, _, rec = run + folder = root / "evidence" / rec["evidence_id"] + other = tmp_path / "other-store" + other.mkdir() + folder.rename(other / folder.name) + monkeypatch.setenv("INFERENCEKIT_RUNS_DIR", str(other)) + result = resume(root) + assert result["status"] == "needs_attention" + assert result["evidence_summary"]["missing"] == 1 + assert result["items"][0]["evidence_id"] == rec["evidence_id"] + assert result["items"][0]["status"] == "missing" + assert result["stages"]["3_candidates"]["status"] == "done" + assert result["budget"]["verdict"] == "ok" + + +@pytest.mark.parametrize("damage", ["hash", "schema", "id", "json", "array", "nonfinite"]) +def test_corrupt_evidence_never_becomes_available(run, damage): + root, _, _, rec = run + path = root / "evidence" / rec["evidence_id"] / "evidence.json" + if damage == "hash": + rec["summary"] = "tampered" + elif damage == "schema": + rec.pop("summary") + rec["evidence_id"] = ev.compute_id(rec) + elif damage == "id": + rec = ev.finalize(ev.build(fake_call(), kind="benchmark", claim_class="observed", summary="another")) + elif damage == "array": + rec = [] + elif damage == "nonfinite": + rec["data"]["x"] = float("nan") + write(path, rec) + if damage == "json": + path.write_text("{invalid") + result = resume(root) + assert result["status"] == "needs_attention" + assert result["items"][0]["status"] == "invalid" + assert result["items"][0]["reason"] + + +def test_flat_run_directory_evidence_copy_is_supported(run): + root, _, _, rec = run + path = root / "evidence" / rec["evidence_id"] / "evidence.json" + path.rename(root / "evidence" / f"{rec['evidence_id']}.json") + result = resume(root) + assert result["status"] == "context_ready" and result["items"][0]["status"] == "available" + + +@pytest.mark.parametrize("file", ["original", "state.json", "intake.normalized.json"]) +def test_changed_intake_pin_is_refused_without_mutation(run, file): + root, original, _, _ = run + path = original if file == "original" else root / file + data = json.loads(path.read_text()) + if file == "original": + data["budget"]["wall_clock_h"] = 100 + else: + data["intake_sha256"] = "changed" + write(path, data) + before = snapshot(root.parent.parent) + with pytest.raises(OpError, match="intake changed"): + resume(root) + assert snapshot(root.parent.parent) == before + + +def test_redacted_source_is_refused(run): + root, original, _, _ = run + data = json.loads(original.read_text()) + data["hardware"]["access"] = {"kind": "ssh", "host": ""} + write(original, data) + with pytest.raises(OpError, match="redacted"): + resume(root) + + +def test_source_override_reuses_pinned_bytes_from_another_working_directory(run, monkeypatch, tmp_path): + root, original, _, _ = run + elsewhere = tmp_path / "elsewhere" + elsewhere.mkdir() + monkeypatch.chdir(elsewhere) + with pytest.raises(OpError, match="pass --file"): + resume(root) + assert resume(root, file=str(original))["status"] == "context_ready" + + +@pytest.mark.parametrize("model_id", ["/home/first/models/test", "/home/second/models/test", "~/models/test"]) +def test_resume_preserves_pinned_intake_across_home_directories(run, monkeypatch, model_id): + from inferencekit.ops import intake + + root, original, _, _ = run + data = json.loads(original.read_text()) + data["model"]["id"] = model_id + source = original.with_name("moved-intake.json") + fresh = root.with_name("moved-run") + write(source, data) + monkeypatch.setattr(intake, "_home", lambda: "/home/first") + registry.call("intake_validate", {"file": str(source), "run_dir": str(fresh)}) + state = json.loads((fresh / "state.json").read_text()) + state["budget"]["run_started"] = "2026-01-01T00:00:00Z" + write(fresh / "state.json", state) + before = snapshot(fresh) + monkeypatch.setattr(intake, "_home", lambda: "/home/second") + assert resume(fresh, file=str(source))["status"] == "context_ready" + assert snapshot(fresh) == before + + +@pytest.mark.parametrize("damage", ["suffix", "conflicting_home"]) +def test_resume_home_redaction_cannot_hide_changed_policy(run, monkeypatch, damage): + from inferencekit.ops import intake + + root, original, _, _ = run + data = json.loads(original.read_text()) + data["model"].update(id="/home/first/models/test", tokenizer="/home/second/models/test") + source = original.with_name("path-intake.json") + fresh = root.with_name("path-run") + write(source, data) + monkeypatch.setattr(intake, "_home", lambda: "/home/first") + registry.call("intake_validate", {"file": str(source), "run_dir": str(fresh)}) + path = fresh / "intake.normalized.json" + normalized = json.loads(path.read_text()) + if damage == "suffix": + normalized["intake"]["model"]["id"] = "~/models/other" + else: + normalized["intake"]["model"]["tokenizer"] = "~/models/test" + write(path, normalized) + monkeypatch.setattr(intake, "_home", lambda: "/home/third") + before = snapshot(fresh) + with pytest.raises(OpError, match="pinned run policy changed"): + resume(fresh, file=str(source)) + assert snapshot(fresh) == before + + +@pytest.mark.parametrize("price", [float("nan"), float("inf"), -float("inf")]) +def test_resume_refuses_nonfinite_checkpoint_spending(run, price): + root, _, state, _ = run + state["budget"]["meter"] = [{"start": "2026-01-01T00:00:00Z", "end": NOW, "price_usd_h": price}] + write(root / "state.json", state) + before = snapshot(root) + with pytest.raises(OpError, match="malformed budget/stage fields"): + resume(root) + assert snapshot(root) == before + + +@pytest.mark.parametrize("mutation", ["stage", "stage_type", "version", "citation", "budget", "time"]) +def test_malformed_state_fields_are_refused(run, mutation): + root, _, state, _ = run + if mutation == "stage": + state["current_stage"] = "missing" + elif mutation == "stage_type": + state["current_stage"] = [] + elif mutation == "version": + state["state_version"] = 2 + elif mutation == "citation": + state["stages"]["3_candidates"]["evidence"] = ["not-an-id"] + elif mutation == "budget": + state["budget"]["meter"] = [{"start": "bad-time"}] + write(root / "state.json", state) + with pytest.raises(OpError): + resume(root, **({"now": "bad-time"} if mutation == "time" else {})) + + +def test_cli_and_registry_share_resume_arguments(run, capsys): + root, original, _, _ = run + assert cli.main(["intake", "resume", str(root), "--file", str(original), "--now", NOW]) == 0 + result = json.loads(capsys.readouterr().out) + assert result == resume(root, file=str(original)) + + +def test_resume_redacts_secrets_in_state_notes_and_next_action(run, monkeypatch): + root, _, state, _ = run + monkeypatch.setenv("HF_TOKEN", "hf_not_real_resume_secret") + state["next_action"] = "test hf_not_real_resume_secret" + state["stages"]["3_candidates"]["notes"] = ['api_key="not_real_inline_secret"'] + write(root / "state.json", state) + text = json.dumps(resume(root)) + assert "hf_not_real_resume_secret" not in text and "not_real_inline_secret" not in text + + +@pytest.mark.parametrize("key,value", [ + ("host_hours_cap", 100), ("usd_cap", 100), ("spend_cap", "none"), + ("deadline", "2099-01-01T00:00:00Z"), ("stage_timeboxes", True), + ("report_reserve_h", 1), ("price_usd_h", 0.01), +]) +def test_state_budget_policy_cannot_override_the_pinned_intake(run, key, value): + root, _, state, _ = run + state["budget"][key] = value + write(root / "state.json", state) + before = snapshot(root) + with pytest.raises(OpError, match=f"state.json.budget.{key}"): + resume(root, now="2026-01-01T05:00:00Z") + assert snapshot(root) == before + + +@pytest.mark.parametrize("section,mutate", [ + ("intake", lambda d: d["quality"].update(fidelity="lossy")), + ("intake", lambda d: d["workload"].update(chat_template_kwargs={"enable_thinking": True})), + ("intake", lambda d: d["budget"].update(approved=False)), + ("derived", lambda d: d["budget"].update(host_hours_cap=100)), + ("derived", lambda d: d["fidelity_rule"].update(value="lossy")), +]) +def test_normalized_policy_edits_are_detected_even_when_hash_strings_are_unchanged(run, section, mutate): + root, _, _, _ = run + path = root / "intake.normalized.json" + data = json.loads(path.read_text()) + mutate(data[section]) + write(path, data) + with pytest.raises(OpError, match=f"intake.normalized.json.{section}"): + resume(root) + + +@pytest.fixture +def legacy_media_run(tmp_path): + def create(modality, workload=None): + original = tmp_path / "original.json" + root = tmp_path / "run" + data = {"run_id": "legacy-media", "model": {"id": "org/model", "revision": "a" * 40, "precision": "fp32"}, + "modality": modality, "hardware": {"gpu": "Test GPU", "count": 1, "access": {"kind": "local"}}, + "budget": {"wall_clock_h": 4, "approved": True, "stage_timeboxes": False, "report_reserve_h": 0}} + if workload is not None: + data["workload"] = workload + write(original, data) + registry.call("intake_validate", {"file": str(original), "run_dir": str(root)}) + state = json.loads((root / "state.json").read_text()) + state["budget"]["run_started"] = "2026-01-01T00:00:00Z" + write(root / "state.json", state) + normalized = json.loads((root / "intake.normalized.json").read_text()) + # a612865 normalized every modality with these defaults, retaining explicit values. + normalized["intake"]["workload"].setdefault("max_tokens", 256) + normalized["intake"]["workload"].setdefault("prompt_set", "dev") + write(root / "intake.normalized.json", normalized) + return root, original, normalized + return create + + +@pytest.mark.parametrize("modality", ["tts", "image", "stt", "video"]) +def test_legacy_implied_media_defaults_resume_without_changing_saved_policy(legacy_media_run, modality): + root, original, normalized = legacy_media_run(modality) + before = snapshot(root.parent) + result = resume(root) + assert result["status"] == "context_ready" + assert result["intake"]["sha256"] == normalized["intake_sha256"] + assert "workload" not in json.loads(original.read_text()) + assert snapshot(root.parent) == before + + +@pytest.mark.parametrize("modality", ["tts", "image"]) +@pytest.mark.parametrize("key,value", [("max_tokens", 512), ("prompt_set", "changed")]) +def test_legacy_media_defaults_do_not_allow_other_saved_values(legacy_media_run, modality, key, value): + root, _, normalized = legacy_media_run(modality) + normalized["intake"]["workload"][key] = value + write(root / "intake.normalized.json", normalized) + before = snapshot(root) + with pytest.raises(OpError, match=f"intake.normalized.json.intake.workload.{key}"): + resume(root) + assert snapshot(root) == before + + +@pytest.mark.parametrize("modality", ["tts", "image"]) +@pytest.mark.parametrize("key,explicit,saved", [("max_tokens", 512, 256), ("prompt_set", "custom", "dev"), + ("max_tokens", 256, 512), ("prompt_set", "dev", "changed")]) +def test_legacy_media_compatibility_preserves_explicit_source_fields(legacy_media_run, modality, key, explicit, saved): + root, _, normalized = legacy_media_run(modality, {key: explicit}) + assert resume(root)["status"] == "context_ready" + normalized["intake"]["workload"][key] = saved + write(root / "intake.normalized.json", normalized) + with pytest.raises(OpError, match=f"intake.normalized.json.intake.workload.{key}"): + resume(root) + + +def test_legacy_media_default_compatibility_does_not_change_original_hash_check(legacy_media_run): + root, original, _ = legacy_media_run("tts") + data = json.loads(original.read_text()) + data["workload"] = {"max_tokens": 256, "prompt_set": "dev"} + write(original, data) + with pytest.raises(OpError, match="intake changed since this run started"): + resume(root) + + +@pytest.mark.parametrize("field,value", [ + ("model", {"id": "different-model", "revision": "a" * 40}), + ("workload_pin", {"chat_template_kwargs": {"enable_thinking": True}}), +]) +def test_state_model_or_workload_pin_cannot_drift(run, field, value): + root, _, state, _ = run + state[field] = value + write(root / "state.json", state) + with pytest.raises(OpError, match=f"state.json.{field}"): + resume(root) + + +def test_documented_progress_meter_and_stage_box_updates_are_allowed(run): + root, _, state, _ = run + state["budget"]["meter"] = [{"start": "2026-01-01T00:00:00Z", "end": "2026-01-01T00:30:00Z", + "price_usd_h": 0.6}] + state["stages"]["4_tuning"].update(time_box_h=3, started="2026-01-01T00:10:00Z", notes=["time reassigned"]) + state["model"]["revision_resolved"] = "a" * 40 + state["decisions"].append({"choice": "continue", "asked_user": False}) + write(root / "state.json", state) + result = resume(root) + assert result["status"] == "context_ready" + assert result["budget"]["spent"]["host_hours"] == 0.5 + assert result["budget"]["spent"]["usd"] == 0.3 + + +@pytest.mark.parametrize("resolved", ["b" * 40, "main", "", 123]) +def test_immutable_revision_pin_rejects_a_conflicting_resolution(run, resolved): + root, _, state, _ = run + state["model"]["revision_resolved"] = resolved + write(root / "state.json", state) + before = snapshot(root) + with pytest.raises(OpError) as exc: + resume(root) + rejection = json.loads(str(exc.value)) + assert rejection["status"] == "blocked" + assert "state.json.model.revision_resolved" in rejection["reason"] + assert "Do not resume" in rejection["action"] + assert snapshot(root) == before + + +@pytest.mark.parametrize("revision,resolved", [("a" * 40, "A" * 40), ("main", "b" * 40), + ("release-v1", "c" * 40), (None, "d" * 40)]) +def test_matching_sha_and_legitimate_branch_resolutions_are_allowed(run, revision, resolved): + root, original, _, _ = run + data = json.loads(original.read_text()) + data["model"]["revision"] = revision + fresh_original = original.with_name("resolved-intake.json") + write(fresh_original, data) + fresh_root = root.with_name("resolved-run") + registry.call("intake_validate", {"file": str(fresh_original), "run_dir": str(fresh_root)}) + state = json.loads((fresh_root / "state.json").read_text()) + state["model"]["revision_resolved"] = resolved + state["budget"]["run_started"] = "2026-01-01T00:00:00Z" + write(fresh_root / "state.json", state) + assert resume(fresh_root)["status"] == "context_ready" + + +def test_mutable_clock_and_meter_never_establish_remaining_authorization(run): + root, _, state, _ = run + state["budget"]["run_started"] = "2025-12-31T20:00:00Z" + state["budget"]["meter"] = [{"start": "2025-12-31T20:00:00Z", "end": NOW, "price_usd_h": 0.6}] + write(root / "state.json", state) + exhausted = resume(root) + assert exhausted["status"] == "needs_attention" + assert exhausted["budget"]["verdict"] == "over_budget" + assert exhausted["budget"]["left"]["host_hours"] == -1 + state["budget"]["run_started"] = NOW + state["budget"]["meter"] = [] + write(root / "state.json", state) + before = snapshot(root) + reset = resume(root) + assert reset["status"] == "context_ready" + assert reset["budget"]["left"]["host_hours"] == 4 + assert reset["budget"]["spent"]["usd"] == 0 + for result in (exhausted, reset): + assert result["trust"]["checkpoint_history_authenticated"] is False + assert result["trust"]["remaining_authorization_verified"] is False + assert "do not authorize work" in result["trust"]["authorization"] + assert "not verified authorization" in result["budget"]["basis"] + assert snapshot(root) == before + + +def test_large_resume_result_keeps_late_failure_and_pages_without_losing_evidence(run): + root, _, state, first = run + store = ev.Store(root / "evidence") + ids = [first["evidence_id"]] + for i in range(199): + rec = store.save(ev.build(fake_call(), kind="benchmark", claim_class="observed", + summary=f"record {i}: " + "detail " * 80)) + ids.append(rec["evidence_id"]) + path = store.run_dir(ids[-1]) / "evidence.json" + rec = json.loads(path.read_text()) + rec["summary"] = "edited after capture" + write(path, rec) + state["stages"]["3_candidates"]["evidence"] = ids + write(root / "state.json", state) + result = fit(resume(root, limit=200), max_chars=12000) + assert result["status"] == "needs_attention" + assert result["budget"]["verdict"] == "ok" + assert result["evidence_summary"] == {"checked": 200, "available": 199, "missing": 0, "invalid": 1, + "order": "missing/invalid first"} + assert result["items"][0]["evidence_id"] == ids[-1] + assert result["items"][0]["status"] == "invalid" + assert result["next_offset"] == len(result["items"]) + seen = [item["evidence_id"] for item in result["items"]] + while result["next_offset"] is not None: + result = fit(resume(root, offset=result["next_offset"], limit=200), max_chars=12000) + seen.extend(item["evidence_id"] for item in result["items"]) + assert len(seen) == len(set(seen)) == 200 and set(seen) == set(ids) + + +def test_resume_page_continuation_preserves_source_override_and_fixed_time(run): + root, original, state, rec = run + state["stages"]["3_candidates"]["evidence"].append("ev_" + "b" * 64) + write(root / "state.json", state) + first = resume(root, file=str(original), limit=1) + assert first["next_call"] == {"tool": "intake_resume", "arguments": { + "run_dir": str(root), "file": str(original), "now": NOW, "offset": 1, "limit": 1}} + second = registry.call(first["next_call"]["tool"], first["next_call"]["arguments"]) + assert second["items"][0]["evidence_id"] == rec["evidence_id"] and second["next_offset"] is None + assert second["budget"] == first["budget"] + + +@pytest.mark.parametrize("field", ["notes", "next_action", "best"]) +def test_oversized_checkpoint_details_keep_resume_failures_and_usable_continuations(run, field): + from inferencekit.ops.store import get_path + + root, original, state, rec = run + huge = ["first log " + "a" * 20000, "second log " + "b" * 20000] + if field == "notes": + state["stages"]["4_tuning"]["notes"] = huge + elif field == "best": + state["best"] = {"single_user": {"notes": huge}} + else: + state["next_action"] = "next action " + " ".join(huge) + state["stages"]["3_candidates"]["evidence"].append("ev_" + "b" * 64) + write(root / "state.json", state) + before = snapshot(root) + raw = resume(root, file=str(original)) + raw_bytes = json.dumps(raw) + shown = fit(raw) + assert json.dumps(raw) == raw_bytes and snapshot(root) == before + assert len(json.dumps(shown)) <= 32000 + assert shown["status"] == "needs_attention" + assert shown["trust"] == raw["trust"] and shown["budget"] == raw["budget"] + assert shown["policy"] == raw["policy"] and shown["intake"] == raw["intake"] + assert shown["evidence_summary"] == raw["evidence_summary"] + assert shown["items"][0]["status"] == "missing" + assert shown["state_path"] == str(root / "state.json") + if shown.get("truncated"): + assert "neither proof nor authorization" in shown["note"] + for pointer in shown["detail_sources"]: + assert get_path(json.loads(Path(pointer["file"]).read_text()), pointer["path"]) + else: # A single long string can already fit after selecting one evidence row. + assert len(json.dumps(raw)) <= 32000 + seen = [row["evidence_id"] for row in shown["items"]] + for _ in range(3): + if shown["next_offset"] is None: + break + previous_offset = shown["offset"] + call = shown["next_call"] + assert call["arguments"]["file"] == str(original) and call["arguments"]["now"] == NOW + shown = fit(registry.call(call["tool"], call["arguments"])) + assert shown["offset"] > previous_offset + seen.extend(row["evidence_id"] for row in shown["items"]) + assert shown["next_offset"] is None + assert len(seen) == len(set(seen)) == 2 and rec["evidence_id"] in seen + + +def test_small_resume_result_keeps_exact_payload_and_oversized_rejection_stays_blocked(run): + root, original, state, _ = run + raw = resume(root) + before = json.dumps(raw) + assert fit(raw) is raw and json.dumps(fit(raw)) == before + state["next_action"] = "saved action " * 5000 + state["stages"]["4_tuning"]["notes"] = ["a" * 50000, "b" * 50000] + state["best"] = {"single_user": "c" * 50000} + write(root / "state.json", state) + data = json.loads(original.read_text()) + data["budget"]["wall_clock_h"] = 40 + write(original, data) + with pytest.raises(OpError) as exc: + resume(root) + rejection = json.loads(str(exc.value)) + assert len(str(exc.value)) <= 24000 + assert rejection["status"] == "blocked" and rejection["action"].startswith("Do not resume.") + assert rejection["saved_state"]["next_action"]["truncated"] is True + assert "diagnostic only" in rejection["trust"]["authorization"] + + +def test_oversized_evidence_qualifiers_keep_coverage_and_direct_source_paths(run): + from inferencekit.ops.store import get_path + + root, _, state, _ = run + rec = ev.Store(root / "evidence").save(ev.build( + fake_call(), kind="benchmark", claim_class="observed", summary="screen " + "a" * 20000, + missing=["missing proof " + "b" * 20000], limitations=["limitation " + "c" * 20000])) + state["stages"]["3_candidates"]["evidence"] = [rec["evidence_id"]] + state["candidates"] = {} + state["best_history"] = [] + write(root / "state.json", state) + shown = fit(resume(root)) + assert len(json.dumps(shown)) <= 32000 + assert shown["trust"]["remaining_authorization_verified"] is False + row = shown["items"][0] + assert row["status"] == "available" and row["coverage"]["complete"] is False + assert row["summary"] == rec["summary"] + for field, key in ((row, "limitations"), (row["coverage"], "missing")): + cut = field[key] + assert cut["truncated"] is True + source = cut["source"] + assert get_path(json.loads(Path(source["file"]).expanduser().read_text()), source["path"]) + assert shown["next_offset"] is None + + +def test_unshrinkable_resume_metadata_stops_with_bounded_cli_recovery(run): + root, original, state, _ = run + # Stage names have no length constraint. This valid checkpoint also repeats the name in budget.stage. + name = "custom_stage_" + "x" * 40000 + state["stages"][name] = dict(state["stages"]["4_tuning"]) + state["current_stage"] = name + state["stages"][name]["evidence"] = ["ev_" + "b" * 64] + write(root / "state.json", state) + raw = resume(root, file=str(original)) + shown = fit(raw) + assert len(json.dumps(shown)) <= 32000 + assert shown["status"] == "needs_attention" and shown["resume_context_status"] == raw["status"] + assert shown["trust"] == raw["trust"] and shown["evidence_summary"] == raw["evidence_summary"] + assert shown["evidence_summary"]["missing"] == 1 + assert "budget" in shown["omitted_fields"] and "CLI" in shown["recovery"] + assert "next_call" not in shown and "next_offset" not in shown + assert shown["state_path"] == str(root / "state.json") + assert shown["cli_arguments"] == {"run_dir": str(root), "file": str(original), "now": NOW} + + +@pytest.mark.parametrize("args", [{"offset": -1}, {"limit": 0}]) +def test_resume_rejects_invalid_page_arguments(run, args): + with pytest.raises(OpError, match="offset.*limit"): + resume(run[0], **args) + + +def blocked(root, **args): + with pytest.raises(OpError) as exc: + resume(root, **args) + result = json.loads(str(exc.value)) + assert result["status"] == "blocked" and result["action"].startswith("Do not resume.") + assert "diagnostic only" in result["trust"]["authorization"] + assert "budget" not in result and "policy" not in result # no validated success fields + return result + + +def drift(original): + data = json.loads(original.read_text()) + data["budget"]["wall_clock_h"] = 80 + write(original, data) + + +def test_drift_rejection_retains_read_only_independent_evidence_and_unvalidated_state_diagnostics(run): + root, original, state, rec = run + drift(original) + before = snapshot(root.parent.parent) + result = blocked(root) + assert snapshot(root.parent.parent) == before + assert result["saved_state"]["current_stage"] == "4_tuning" + assert result["saved_state"]["next_action"] == state["next_action"] + assert result["saved_policy_snapshot_unvalidated"]["budget"]["wall_clock_h"] == 4 + assert result["supplied_intake_snapshot_not_pinned"]["budget"]["wall_clock_h"] == 80 + assert result["saved_budget_calculation_unvalidated"]["left"]["host_hours"] == 3 + assert result["evidence_summary"]["checked"] == result["evidence_summary"]["available"] == 1 + assert result["evidence_summary"]["incomplete_records"] == 1 + assert result["evidence_summary"]["coverage_missing"]["items"] == ["full quality proof"] + assert result["evidence_summary"]["limitations"]["items"] == ["screening only"] + assert result["evidence_issue_page"]["total"] == 0 + + +@pytest.mark.parametrize("mutation", ["state_cap", "normalized_quality", "state_model"]) +def test_policy_rejections_remain_blocked_even_when_diagnostic_budget_says_continue(run, mutation): + root, _, state, _ = run + if mutation == "state_cap": + state["budget"]["host_hours_cap"] = 100 + write(root / "state.json", state) + elif mutation == "state_model": + state["model"]["id"] = "another-model" + write(root / "state.json", state) + else: + path = root / "intake.normalized.json" + data = json.loads(path.read_text()) + data["intake"]["fidelity"] = "lossy" + write(path, data) + result = blocked(root) + assert "pinned run policy changed" in result["reason"] + assert result["saved_budget_calculation_unvalidated"]["action"] == "continue" + assert result["trust"]["saved_policy_and_budget"].startswith("UNPROVEN") + + +def test_drift_diagnostics_check_corrupt_evidence_independently(run): + root, original, _, rec = run + drift(original) + rec["summary"] = "edited after capture" + write(root / "evidence" / rec["evidence_id"] / "evidence.json", rec) + result = blocked(root) + assert result["evidence_summary"]["invalid"] == 1 + assert result["evidence_issue_page"]["items"][0]["evidence_id"] == rec["evidence_id"] + + +def test_blocked_issue_pages_are_bounded_and_preserve_all_failure_ids(run): + root, original, state, _ = run + drift(original) + ids = [f"ev_{i:064x}" for i in range(37)] + state["stages"]["3_candidates"]["evidence"] += ids + state["next_action"] = "long diagnostic action " * 1000 + write(root / "state.json", state) + result = blocked(root, file=str(original), limit=1000) + assert len(json.dumps(result)) < 24000 + assert result["saved_state"]["next_action"]["truncated"] + seen = [] + while True: + assert result["evidence_summary"]["missing"] == 37 + seen.extend(item["evidence_id"] for item in result["evidence_issue_page"]["items"]) + if "next_call" not in result: + break + args = result["next_call"]["arguments"] + assert args["now"] == NOW and args["limit"] == 10 + with pytest.raises(OpError) as exc: + registry.call(result["next_call"]["tool"], args) + result = json.loads(str(exc.value)) + assert result["status"] == "blocked" + assert len(seen) == 37 and set(seen) == set(ids) + + +def test_drift_with_malformed_citations_does_not_report_an_empty_successful_audit(run): + root, original, state, _ = run + drift(original) + state["stages"]["3_candidates"]["evidence"] = "invalid" + write(root / "state.json", state) + result = blocked(root) + assert result["evidence_summary"]["complete_check"] is False + assert "error" in result["evidence_summary"] + assert "available" not in result["evidence_summary"] + + +def test_cli_drift_diagnostics_still_exit_one_with_no_success_output(run, capsys): + root, original, _, _ = run + drift(original) + assert cli.main(["intake", "resume", str(root), "--file", str(original), "--now", NOW]) == 1 + out = capsys.readouterr() + assert out.out == "" + result = json.loads(out.err.removeprefix("inferencekit: error: ")) + assert result["status"] == "blocked" and result["evidence_summary"]["checked"] == 1 + + +def test_nonfinite_saved_policy_still_produces_a_structured_rejection(run): + root, _, _, _ = run + path = root / "intake.normalized.json" + data = json.loads(path.read_text()) + data["intake"]["budget"]["wall_clock_h"] = float("nan") + write(path, data) + result = blocked(root) + assert result["saved_policy_snapshot_unvalidated"]["budget"]["wall_clock_h"] == "see source (non-finite value)" diff --git a/tests/test_round_evidence.py b/tests/test_round_evidence.py new file mode 100644 index 0000000..98704c4 --- /dev/null +++ b/tests/test_round_evidence.py @@ -0,0 +1,38 @@ +"""Scheduler evidence must match its cells and retain partial-memory qualifiers.""" +import pytest + +from inferencekit.serve.round_evidence import finalize_arm_records +from conftest import make_benchmark + + +def fixture_run(store): + source = make_benchmark(store, [100, 100, 100], t0=5, step=10) + cells = [{"label": "baseline", "round": i + 1, "status": "ok", "t_start": i * 10, + "t_end": i * 10 + 9, "ready": {"status": "ready"}, "stop": {"group_gone": True}, + "server_env": {"controls": {"OMP_NUM_THREADS": "2"}, "ld_preload": [], "omitted": []}, + "resources": {"status": "observed", "peak_rss_mib": 50 + i}} for i in range(3)] + plan = {"rounds": 3, "design": {"design": "Williams", "balanced": True}} + return source, cells, plan + + +def test_enriched_evidence_keeps_raw_capture_and_partial_memory(store): + source, cells, plan = fixture_run(store) + cells[1]["resources"]["status"] = "partial" + final = finalize_arm_records({"baseline": source["evidence_id"]}, cells, plan) + record = store.load(final["baseline"]) + assert record["data"]["memory"]["complete"] is False + assert record["data"]["memory"]["verdict"] == "UNPROVEN" + assert record["data"]["memory"]["peak_rss_mib"] == 52 + assert record["data"]["lifecycle"]["verdict"] == "PASS" + assert store.load(source["evidence_id"])["data"].get("scheduler_final") is None + + +@pytest.mark.parametrize("bad", ["duplicate", "outside_window"]) +def test_lifecycle_requires_unique_cells_bound_to_capture_windows(store, bad): + source, cells, plan = fixture_run(store) + if bad == "duplicate": + cells[1]["round"] = 1 + else: + cells[1]["t_end"] = 11 # client round 1 was actually captured at 15..16 + final = finalize_arm_records({"baseline": source["evidence_id"]}, cells, plan) + assert store.load(final["baseline"])["data"]["lifecycle"]["verdict"] == "UNPROVEN"