cli-contract.md

CLI contract

xprobe exposes exactly four public commands: doctor, discover, validate, and measure.

Common behavior

When stdout is not a terminal, every command writes one versioned JSON document to stdout; on a terminal it prints a human-readable report. --json forces JSON and --human forces the report; they cannot be combined. --non-interactive and --no-color are accepted for compatibility: commands never prompt and never color. Diagnostics, including the online-injection warning, go to stderr.

Measurement results carry schema_version: "3.0"; every other success record, error record, and Event JSONL line carries "2.0". Unknown JSON fields and unsupported schema versions are rejected. xprobe maintains only the current schema during the pre-1.0 protocol transition.

Exit	Meaning
0	Command completed and emitted a result
1	Validation, collection, decode, export, cleanup, or internal failure
2	Invalid CLI syntax from Clap
3	Target missing, exited, or reused
4	Permission denied for inspection, eBPF, or ptrace

Runtime failures use schemas/error.schema.json and include a stable code, message, recoverability, details, and hints.

doctor

xprobe doctor

Reports Linux/kernel identity, BTF, BPF/perf permissions, lockdown, ptrace policy, NVIDIA driver, CUDA toolkit/driver, CUPTI, namespaces, and conservative capability booleans. ok: true means diagnosis completed, not that every capability is available. The CUPTI check lists each supported installed major and its resolved library path.

discover

xprobe discover --pid 4242 [--limit 200]

Treats the PID as a process-tree root and queries NVML for active CUDA compute processes without attaching. It returns only root/descendant context holders, including PID plus procfs start time, parent PID, executable, command line, and GPU UUIDs. limit must be positive. total_candidates and truncated describe bounded output. The caller chooses one candidate and passes its PID to validate and measure; xprobe does not guess among workers.

NVML lists compute processes from every container sharing the GPU. A listed PID that is not visible in the caller's PID namespace, or that exits before inspection, cannot be a candidate; it is reported in a NVML_PROCESS_UNAVAILABLE warning with its nvml_pids instead of failing the command. Root exit and other inspection failures remain errors.

This process-candidate response is discovery schema 2.0. xprobe does not emit or maintain the pre-0.3 event-discovery shape.

An unavailable or failed NVML query is an explicit command error, not an empty candidate list.

Selectors

Selectors are supplied by the caller and checked by validate before any attachment. Host selector forms are:

uprobe:<binary>:<symbol>:entry
uprobe:<binary>:<symbol>:return
uprobe:<binary>:symbol=<full-demangled-c++-signature>:entry
uprobe:<binary>:symbol=<full-demangled-c++-signature>:return
uprobe:<binary>:+0x<file-offset>:entry
uprobe:<binary>:+0x<file-offset>:return
syscall:<name>:entry
syscall:<name>:exit
tracepoint:<category>:<name>
python:gc_start
python:gc_end

The symbol= form allows :: and other punctuation in a full C++ signature. Resolution returns the attachable mangled ELF name in symbol and its readable signature in symbol_demangled; captured host events retain both. It works for mapped CPython, native extension, and framework libraries, but does not resolve Python frames or module.qualname names.

Named syscall selectors are Linux x86_64 endpoints resolved by validate. Their eBPF path filters PID and syscall number before reserving an event, records the six scalar syscall ABI registers on entry, and records the scalar return value on exit. It never dereferences pointer arguments. Named tracepoints record identity and timestamp only; they do not copy the tracepoint payload. Unsupported syscall names and unavailable tracepoints fail explicitly.

Python GC selectors are capability-based, not process-name based. Validation reads the target's mapped CPython executable and libpython ELF metadata and requires both python:gc__start and python:gc__done USDT markers. Captured events contain process/thread identity, timestamp, mapped binary, provider, and marker name without copying generation or object payloads.

CUDA forms include Runtime/Driver API entry/exit, kernel/memcpy/memset activity start/end, and NVTX range boundaries. Kernel selectors accept name~REGEX; memcpy selectors accept kind=<HtoD|DtoH|DtoD|HtoH|PtoP>. NVTX selectors are cuda:nvtx_range_start:name~REGEX and cuda:nvtx_range_end:name~REGEX; their name expression must reduce to one nonempty exact, prefix, suffix, or contains filter shorter than 128 bytes.

validate

xprobe validate --pid 4242 \
  --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match exact

Validation is read-only. It verifies target identity, resolves host and Python USDT selectors, parses CUDA filters, and checks collection and correlation requirements. validate --cpu-sample checks target threads, perf-event access, and Python symbolization status. validate --syscall-aggregate checks the bounded eBPF aggregate path. Inventory modes do not take event-pair selectors. Results conform to schemas/validate.schema.json.

validate --spec graph.json validates a measurement graph without --pid; the spec's target selects the process. It emits schemas/measurement-graph-validate.schema.json (schema 3.0) with each named endpoint's resolution, each edge's policy recommendation, requirements merged across edges, and issues scoped to the graph, an endpoint, or an edge. Unknown or unused endpoints, duplicate edge names, missing bounds, and more than two syscall, tracepoint, or USDT endpoints are issues, not command errors.

Supported policies are exact, first-after, nearest, stack-nested, and stream-order. Exact uses deterministic CUPTI correlation IDs, NVTX range kind/ID, one named syscall's per-thread entry/exit lifecycle, or one CPython GC start/end lifecycle on the same thread. Nested requires entry/return of the same host function. Stream order requires GPU activity endpoints. Temporal policies always warn that they are heuristic. policy_recommendation reports the strongest compatible policy, a stable machine-readable reason, and all compatible alternatives. The caller still chooses the policy; xprobe does not silently replace or retry it.

requirements.agent_activation is not_required, already_loaded, injection_required, or startup_required. Online injection handles ordinary CUDA callbacks and activity; injection_required sets target_mutation: true and emits TARGET_PROCESS_WILL_BE_MODIFIED. NVTX dispatch is fixed by its first API initialization, so an unmapped Agent produces startup_required and an invalid result. Restart the target with NVTX_INJECTION64_PATH set to the matching xprobe Agent before its first NVTX call. A mapped Agent is verified again at ARM through the capture feature flag. Callers must stop on valid: false. Mapped CUDA/CUPTI majors that conflict or fall outside 12 and 13 produce UNSUPPORTED_CUDA_VERSION. CUDA major support does not imply host-clock alignment: measure accepts GPU-only pairs in the CUPTI clock and rejects CPU/GPU subtraction when the capture omits the host-monotonic feature flag.

measure

Live target:

xprobe measure --pid 4242 \
  --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match exact --samples 100 --timeout-ms 30000 \
  --events-out /tmp/xprobe-events.jsonl

Linux syscall duration:

xprobe measure --pid 4242 \
  --from 'syscall:mmap:entry' --to 'syscall:mmap:exit' \
  --match exact --samples 100 --max-events 1000

CPU/Python hotspot inventory:

xprobe measure --pid 4242 --cpu-sample --duration-ms 1000 \
  --frequency-hz 99 --max-samples 10000 --max-groups 256 \
  --stack-depth 64 --max-threads 1024

Syscall lifecycle inventory:

xprobe measure --pid 4242 --syscall-aggregate --duration-ms 1000 \
  --max-groups 256 --max-inflight 1024

CPython garbage-collection duration:

xprobe measure --pid 4242 \
  --from python:gc_start --to python:gc_end \
  --match exact --samples 100 --max-events 1000

NVTX application range:

xprobe measure --pid 4242 \
  --from 'cuda:nvtx_range_start:name~^forward$' \
  --to 'cuda:nvtx_range_end:name~^forward$' \
  --match exact --samples 100 --max-events 1000

Completed captures:

xprobe measure --input host.json --input cupti.bin \
  --from 'uprobe:/srv/app/server:request:entry' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match first-after --samples 100

Versioned live spec:

xprobe measure --spec measurement.json

Same-window triage of an unknown live workload:

xprobe measure --pid 4242 --triage --duration-ms 2000

Broad GPU activity inventory:

xprobe measure --pid 4242 \
  --from 'cuda:kernel_start' --to 'cuda:kernel_end' \
  --match exact --aggregate --duration-ms 1000 --max-groups 4096

Exactly one source mode is used: --pid, one or more --input, or --spec. At least one positive --samples or --duration-ms bound is required in direct mode. --timeout-ms defaults to 30 seconds and --max-events to 100,000.

--cpu-sample is live-only and requires --duration-ms. It opens bounded userspace callchain sampling events for the target's current threads. Defaults are 99 Hz, 10,000 observed samples, 256 retained stack groups, depth 64, and 1,024 threads. The result uses schemas/cpu-sample-inventory-result.schema.json: exact raw stack counts are grouped deterministically, hotspots report inclusive/exclusive counts and proportions, and attachable native frames include uprobe selector hints. Native ELF and /tmp/perf-<pid>.map Python symbols are resolved independently. python_status and resolved/unresolved frame counts prevent native fallback from being mislabeled as Python semantic coverage. Lost samples, truncated stacks, thread coverage, and capacity pressure remain visible.

--triage is live-only, requires --duration-ms, and emits schemas/triage-result.schema.json (schema 3.0). Within one window it concurrently collects procfs per-thread on-CPU, runqueue, and blocked time; cgroup CPU throttling; a CPU sample inventory; a syscall aggregate; NVML device health (utilization, clock event reasons, temperature, power, PCIe link); and, when the target has loaded libcuda, a CUDA kernel aggregate, which may inject the CUPTI Agent under the same rules as measure. The result keeps the top five hotspots, syscall groups, and kernel groups inline and writes each full inventory to a private artifact referenced from sources. findings are deterministic and ranked by an estimated share of the window; a finding whose impact cannot be bounded has impact_share: null and ranks last. Each finding carries an environment action or a ready MeasurementSpec for measure --spec, with record capacity sized from the observed rate. A source that is not permitted, absent, or exceeds its bounded capacity is unavailable with a reason and a TRIAGE_SOURCE_UNAVAILABLE warning; target exit or reuse, cleanup, and internal failures fail the command. not_covered lists what triage cannot answer, including where blocked threads wait. Blocked time includes idle waiting, so read it per thread. NVML utilization is a device-wide sampled estimate.

--syscall-aggregate is live-only and requires --duration-ms. PID-filtered raw syscall entry/exit programs retain at most --max-inflight thread starts and --max-groups syscall groups in BPF maps, then emit one bounded summary. Known x86_64 numbers include names and exact entry/exit selector hints; unknown numbers remain numeric. schemas/syscall-aggregate-result.schema.json exposes entries, matched and unmatched exits, inflight-at-end, drops, occupancy, errors, and total/min/max/mean duration. It contains no pointer values or per-call event stream.

--aggregate is live-only, duration-bounded, and accepts one matching kernel, memcpy, or memset activity start/end pair. It uses --max-groups (default 4,096), does not accept --samples or --events-out, and emits schemas/aggregate-inventory-result.schema.json. The result is a coarse inventory, not exact event evidence: it reports count, total/min/max/mean duration, optional transferred bytes, table occupancy, and drop completeness. Memcpy groups also split by src_memory and dst_memory (pageable, pinned, device, managed, ...); pageable host memory forces staged, synchronous copies. Aggregate results use schema 3.0. Aggregate kernel regex must be reducible to an exact, prefix, suffix, or contains filter because this mode intentionally has no Rust-side event pass. Each kernel group reports name_complete. CUPTI names that fill the fixed 127-byte observed prefix are marked false; their selector hints use that prefix instead of claiming an exact full name, so the next capture can still filter in the Agent hot path.

The three inventory modes have distinct result contracts and contain no exact event timeline. They cannot be passed to measure --input or exported with --events-out. Their groups and hotspots are evidence for choosing selectors; read-only validation still precedes the resulting exact measurement.

Live host endpoints attach PID-scoped eBPF probes. Linux syscall endpoints use raw tracepoints so they do not depend on tracingfs event IDs; ordinary named tracepoints use their kernel category and name. CPython GC endpoints attach to the exact mapped ELF selected by validation. Every endpoint link is attached before collection is armed. A samples-bound Linux capture allows bounded startup slack for a target already inside an event boundary. A duration capture that fills --max-events returns EVENT_RATE_TOO_HIGH rather than reporting partial success. CUDA endpoints automatically activate the CUPTI agent. If it is absent, --agent or XPROBE_CUPTI_AGENT_PATH selects the shared object; otherwise xprobe searches ../lib/xprobe/cuda12 or ../lib/xprobe/cuda13 according to the target and then the matching development build path. An unobservable target major is accepted only when exactly one supported CUPTI major is installed. Injection requires Linux x86_64, a shared mount namespace, and ptrace permission. It emits a stderr warning and CUPTI_AGENT_INJECTED. Each live call arms a fresh --max-events-bounded capture with endpoint filters. Final stop disables CUPTI; automatic injection also removes its private socket while leaving the .so mapped.

NVTX thread ranges preserve nesting through the thread ID and zero-based push level. Process ranges preserve the NVTX range ID and may end on a different thread; evidence reports both boundary threads plus start_tid. Names are copied into the fixed record bound with name_complete. ASCII nvtxRangePushA/nvtxRangeStartA and ASCII messages in their Ex forms are supported; wide and registered-string messages are not collected. This is timeline boundary collection only. It does not enable CUPTI Range Profiling, metric replay, or kernel performance-counter passes.

Completed inputs may be CUPTI ABI binary, bounded host-capture JSON, or Event JSONL. Repeated inputs are merged after target-PID checks. Unknown records, mixed targets, malformed captures, and excessive event counts fail explicitly.

Results conform to schemas/measurement-result.schema.json (schema 3.0) and contain:

• status and latency statistics;
• a quality verdict (good or caution) with coded issues;
• matched, unmatched, ambiguous, and dropped counts;
• method, confidence, and score;
• clock alignment and optional estimated error;
• collection completeness, host/CUDA totals, and CUPTI capacity, observed, retained, dropped, and utilization values when an ABI envelope provides them;
• evidence: one exemplar start/end pair per latency rank (min, p50, p90, p99, max), the artifact path, format, and event count, and with --evidence full every matched pair;
• structured warnings.

Live captures always write the complete bounded capture with mode 0600: --events-out PATH chooses the file; otherwise it goes to a new mode 0700 directory under the temporary directory. measure --input with the same selectors re-correlates it deterministically. Completed --input measurements write an artifact only with --events-out. The default format is JSONL; --format chrome writes Chrome Trace Event Format. Once collection has produced a capture, correlation, clock, drop, or capacity failures still write it and identify its path, format, and event count in the JSON error. Artifact failure returns TRACE_EXPORT_FAILED and preserves the original measurement error code in details.

No matched pairs return NO_MATCHED_SAMPLES. Cross-clock subtraction without declared alignment returns CLOCK_ALIGNMENT_FAILED. Drops and unknown clock error are never silently converted to high-quality results. Error details and hints carry selectors, policies, counters, clock domains, and explicit next actions where applicable; callers must not infer these fields from messages.

Measurement graphs

A graph measures several edges from one bounded capture of one process:

{
  "schema_version": "3.0",
  "name": "request_path",
  "target": {"pid": 4242, "process_start_time": 123456},
  "endpoints": {
    "request": "uprobe:/srv/app/server:handle_request:entry",
    "launch_exit": "cuda:runtime_api:cudaLaunchKernel:exit",
    "kernel_start": "cuda:kernel_start:name~^flash",
    "kernel_end": "cuda:kernel_end:name~^flash"
  },
  "edges": [
    {"name": "request_to_launch", "from": "request", "to": "launch_exit",
     "match_policy": "first_after", "samples": 100},
    {"name": "launch_to_kernel", "from": "launch_exit", "to": "kernel_start",
     "match_policy": "exact", "samples": 100},
    {"name": "kernel_duration", "from": "kernel_start", "to": "kernel_end",
     "match_policy": "exact", "samples": 100}
  ],
  "duration_ms": null,
  "timeout_ms": 30000,
  "max_events": 20000
}

measure --spec graph.json accepts this schemas/measurement-graph-spec.schema.json document. A graph has 1–8 named endpoints and 1–8 edges; every edge sets samples or the graph sets duration_ms. Collectors attach once per unique endpoint: uprobes have no further limit, CUDA endpoints share up to eight Agent filters, and syscall, tracepoint, and USDT endpoints share two Linux collector slots. max_events bounds the whole capture. Collection ends when every edge reaches its samples or the duration elapses.

The capture is validated once, then each edge is correlated independently with the single-edge rules. A graph never joins edges into per-request paths and never implies causality between edges. Results use schemas/measurement-graph-result.schema.json (schema 3.0): per-edge statistics, quality verdict, correlation, clock quality, and evidence, one shared collection summary, and the shared event artifact. Any invalid edge or edge correlation failure fails the whole command with the edge name in the message and details.edge; the captured events are still written to the artifact so the caller can re-correlate edges with measure --input.

Compatibility

Earlier low-level command spellings remain hidden during the pre-1.0 transition so existing integration fixtures can decode captures. They are not public API and must not be used by Skills or new automation.