Repository navigation
xprobe v0.6.0: agent fast path - #46
Merged
Merged
Conversation
A 100-sample measurement printed ~120 KB of inline event pairs, which crowds out an agent's context while the statistics and quality fields it needs are buried. Measurement results move to schema 3.0: - `evidence` keeps one exemplar pair per latency rank (min, p50, p90, p99, max), using the same nearest-rank indices as the statistics, plus the artifact path; `--evidence full` still inlines every pair. - Live captures always write the bounded event capture, to a private temporary directory unless `--events-out` is set, so summary output never loses evidence. `measure --input` re-correlates it. - `quality` gives a deterministic `good`/`caution` verdict with coded issues for timeouts, heuristic or ambiguous correlation, unmatched events, clock error, and sample count. Unchanged records, including Event JSONL, stay at schema 2.0, so older artifacts remain readable. Live tests that inspect every pair request `--evidence full`.
NVML reports compute processes from every container on the GPU. When a listed PID was not visible in the caller's PID namespace, the ancestry walk read a missing /proc entry and failed the whole command with TARGET_EXITED, so discovery broke whenever another container used the GPU. Such a PID cannot descend from the root, so classify it as unavailable, re-walk once when an ancestor exits mid-walk, and skip workers that exit before inspection. Report skipped PIDs in a NVML_PROCESS_UNAVAILABLE warning; root exit and other inspection failures stay errors. Also update the Skill invariant checked by the contract test for the schema 3.0 measurement wording.
Agents and pipelines had to pass `--json --non-interactive --no-color` on every call, and the docs repeated that triplet dozens of times. Commands now emit their versioned JSON whenever stdout is not a terminal and keep the human report on a terminal. A global `--human` forces the report in a pipe and conflicts with `--json`. Existing flags remain accepted; `dev uprobe --jsonl` streams are unchanged.
Add deterministic readers for the triage window: per-thread on-CPU, runqueue, and blocked time from procfs schedstat; CFS throttling and quota for cgroup v1 and v2; and NVML utilization, clock event reasons, temperature, power, and PCIe link sampled through nvidia-smi.
Add the schema 3.0 TriageResult contract and deterministic rules that turn thread time, cgroup throttling, CPU hotspots, syscall totals, and GPU health into ranked findings with an estimated window share and a ready MeasurementSpec or environment action.
CPU sample analysis read every executable mapping and demangled every text symbol before resolving a single frame. On a PyTorch CUDA process that maps gigabytes of libtorch, cuBLAS, and cuDNN, a 2 s sample took 159 s to analyze. Resolve only modules that contain retained sampled addresses, and demangle only the symbols that contain those offsets. Lookup still uses the full text-symbol table, so zero-size symbols keep their next-symbol bound. The same 2 s sample now completes in 4.7 s.
`measure --pid P --triage --duration-ms N` answers where time went before any selector is known. In one window it concurrently collects per-thread on-CPU, runqueue, and blocked time; cgroup throttling; CPU samples; syscall totals; NVML device health; and, for CUDA targets, a kernel aggregate. Deterministic rules rank findings by an estimated window share and attach an environment action or a ready MeasurementSpec whose record capacity is sized from the observed rate. Full inventories stay in private per-source artifacts. Sources that are not permitted, absent, or over capacity are reported as unavailable with a reason; identity, cleanup, and internal failures still fail the command. `not_covered` states what triage cannot answer. `just test-triage-live` runs both commands against a PyTorch CUDA workload: triage, then the first proposed spec.
Measurement graphs need more than one start/end pair of CUDA endpoints in a single capture. Raise the Agent filter table from 2 to 8 and bump the control protocol to v5: the former reserved field now carries the active filter count on ARM (zero for other commands). Hot paths scan only the active filters, so single-edge captures keep their per-record cost. Aggregate ARM still requires exactly one start/end pair. The capture output ABI is unchanged.
Exact live measurement now builds a LivePlan of validated endpoints and merged requirements, then runs one generic capture loop that delegates correlation and sample completion to a LiveEvaluator. Attachment, polling, timeout, CUPTI shutdown, cleanup failures, and artifact preservation stay in one place. Identical CUDA filters from shared endpoints are armed once. Single-edge behavior is unchanged.
Add schema 3.0 MeasurementGraphSpec, validation, and result contracts. Graph validation resolves each edge with the single-edge rules, merges requirements, and reports graph, endpoint, and edge-scoped issues for structure, bounds, unknown or unused endpoints, and the two shared Linux collector slots.
`measure --spec graph.json` runs a measurement graph: named endpoints shared by up to eight edges, collected once and correlated per edge with the single-edge rules. `validate --spec graph.json` checks it read-only with graph, endpoint, and edge-scoped issues. Any edge failure fails the command with the edge named and keeps the capture artifact for offline re-correlation; edges are never joined into per-request paths. The multisource live test measures request-to-launch, launch-to-kernel, and kernel duration from one capture.
The Skill spread one workflow across ~5,800 words in seven files, with selector syntax and quality checklists repeated, and tests pinned more than seventy exact sentences so nothing could be removed. SKILL.md now drives two commands for unknown live work (triage, then the ranked next spec or a measurement graph), keeps the offline artifact route, and adds an escalation ladder for when the first pass explains nothing: missing sources, split spans with graphs, tail exemplars, blocked threads, and regressions, ending in explicit hand-off. Four overlapping references merge into selectors, interpreting, and a new deep-dive guide; the Skill drops to ~3,400 words. Contract tests now enforce word budgets, resolve links, reject orphaned references, require every documented xprobe flag to exist in that subcommand's --help, and check routes by behavior instead of prose.
Copies from pageable host memory are staged and synchronous, so they are a common hidden transfer cost, but aggregate inventories could not tell them from pinned copies. The Agent now keys memcpy aggregate slots by CUPTI source and destination memory kinds and reports them in the previously reserved record fields, behind a new feature flag so older Agents decode as unknown. Aggregate results move to schema 3.0 with `src_memory` and `dst_memory` per group. The PyTorch live test confirms pinned host buffers are reported as pinned for both copy directions. Triage still covers kernels only and says so in `not_covered`.
README now starts from triage, a ready next spec, and graphs instead of a four-command broad-to-narrow walk. Documentation drops the `--json --non-interactive --no-color` suffix now that pipes default to JSON, and release packaging, the live gate, and public verification move from development.md into releasing.md. The contract test checks every documented xprobe command in README and docs against --help.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v0.6.0 makes xprobe reach a localized answer for an unknown workload in two commands without removing any capability.
measure --pid P --triage --duration-ms N: one window of per-thread on-CPU/runqueue/blocked time, cgroup throttling, CPU samples, syscall totals, NVML health, and CUDA kernels; ranked findings with a readyMeasurementSpecor environment action;not_coveredstates what it cannot answer.measure --spec graph.jsonmeasures up to eight edges from one capture;validate --specreports graph/endpoint/edge-scoped issues. CUPTI control protocol v5 carries up to eight filters; hot paths scan only active ones.good/cautionquality verdict, rank exemplars, and the always-written capture artifact;--evidence fullkeeps every pair inline. A 100-sample result drops from ~125 KB to ~7 KB.--humanforces the report.xprobeflag exists in--help.Fixes
discoverno longer fails on shared GPUs when NVML lists processes from other containers.Tooling
rust-toolchain.toml;environment-pytorch.ymlpins the PyTorch live fixture; CLI orchestration split into modules.Verification
just fmt-check,lint,test,test-agent-contract,test-install,test-skill-install,test-cupti,test-bpfPYTORCH_ENV=… just test-release-live(19 recipes: BPF, CPU, CUDA 12/13 incl. 12.0 minimum, injection, NVTX, multisource graphs, PyTorch, triage, and all benchmarks) on RTX 3060