xprobe v0.6.0: agent fast path

AGENTS.md

@@ -54,14 +54,16 @@
 
 - Follow `skills/xprobe-measure-latency/SKILL.md` for measurement tasks.
 - Choose the shortest Skill route supported by the evidence. Existing artifacts
-  do not require live readiness checks; known selectors may proceed directly to
-  validation and bounded measurement. Use broad-to-narrow inventory only when
-  selectors are unknown. Classify live work as CPU-only or GPU/mixed before
-  choosing collectors; `discover` is only for CUDA context holders. Scope
-  breadth and capture duration are independent. Do not guess selectors.
-- Read evidence pairs, artifact analysis, stream identity, collection quality,
-  and profiler overhead before interpreting latency. Summed concurrent GPU
-  duration is not wall time.
+  are analyzed offline; known boundaries go straight to bounded measurement;
+  unknown live work starts with one triage window and continues with its ranked
+  next spec or a measurement graph. Escalate only from evidence, and do not
+  guess selectors. `discover` is only for CUDA context holders.
+- Read the quality verdict, source coverage, stream identity, and profiler
+  overhead before interpreting latency. Summed concurrent GPU duration is not
+  wall time.
+- Keep the Skill entry point short and each fact in one place. Field meanings
+  belong to schemas and flags to `--help`; Skill tests check behavior, links,
+  and documented flags rather than pinning prose.
 - Keep reusable analysis deterministic and bundled with the Skill. Keep model
   orchestration in the caller rather than adding CLI commands or a service.
 

CMakeLists.txt

@@ -1,6 +1,6 @@
 cmake_minimum_required(VERSION 3.20)
 
-project(xprobe VERSION 0.5.2 LANGUAGES C)
+project(xprobe VERSION 0.6.0 LANGUAGES C)
 
 include(CTest)
 

Cargo.toml

@@ -10,7 +10,7 @@ members = [
 resolver = "2"
 
 [workspace.package]
-version = "0.5.2"
+version = "0.6.0"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/itdevwu/xprobe"

README.md

@@ -23,7 +23,7 @@ only installation action required from the user:
 
 ```bash
 npx skills@1 add \
-  https://github.com/itdevwu/xprobe/tree/v0.5.2/skills/xprobe-measure-latency \
+  https://github.com/itdevwu/xprobe/tree/v0.6.0/skills/xprobe-measure-latency \
   --global
 ```
 
@@ -37,55 +37,38 @@ GitHub-hosted build attestation verification.
 
 ## Measure
 
-For an unknown CPU workload, start with one representative sampled inventory:
+Triage an unknown process over one representative window. Findings are ranked
+by their estimated share of the window, and each carries a ready next step:
 
 ```bash
-xprobe validate --pid 4242 --cpu-sample \
-  --json --non-interactive --no-color
-
-xprobe measure --pid 4242 \
-  --cpu-sample --duration-ms 1000 \
-  --json --non-interactive --no-color
+xprobe measure --pid 4242 --triage --duration-ms 2000 > triage.json
+jq '.findings[0].next.spec' triage.json > next.json
+xprobe measure --spec next.json
 ```
 
-Use its hotspot and selector evidence, or a GPU aggregate inventory, to state a
-narrow hypothesis. Then validate and measure that boundary directly:
+Measure a known boundary directly, or several related boundaries from one
+capture with a [measurement graph](docs/cli-contract.md#measurement-graphs):
 
 ```bash
-xprobe validate --pid 4242 \
-  --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
-  --to 'cuda:kernel_start:name~flash.*' \
-  --match exact --json --non-interactive --no-color
-
 xprobe measure --pid 4242 \
   --from 'cuda:runtime_api:cudaLaunchKernel:exit' \
   --to 'cuda:kernel_start:name~flash.*' \
-  --match exact --samples 100 --timeout-ms 30000 \
-  --events-out /tmp/xprobe-events.jsonl \
-  --json --non-interactive --no-color
+  --match exact --samples 100
 ```
 
-Kernel launch latency is only one event pair. xprobe can first inventory sampled
-CPU/Python hotspots, syscall lifecycles, or GPU activity, then measure host
-function spans, CPython GC, syscall latency, named Linux events, CUDA API calls,
-GPU operation durations, transfers, NVTX ranges, and CPU/GPU paths after
-selecting the correct process.
-
-`measure` also accepts completed `--input` captures and versioned live
-`--spec` files. Evidence can be exported as `jsonl` or `chrome`. JSON results
-contain every matched start/end event, latency statistics, unmatched and
-ambiguous counts, collection completeness, CUPTI buffer usage, clock quality,
-correlation confidence, and warnings. With `--events-out`, the bounded capture
-is also preserved when correlation or clock validation fails.
+Output is JSON whenever stdout is not a terminal. Results carry latency
+statistics, a `good`/`caution` quality verdict, rank exemplar pairs, and the
+path of the bounded event capture, which `measure --input` re-correlates
+offline. See the [CLI contract](docs/cli-contract.md) for every mode.
 
 ## Public CLI
 
 | Command | Purpose |
 | --- | --- |
 | `doctor` | Report local eBPF, ptrace, NVIDIA, CUDA, and CUPTI capabilities |
 | `discover` | List NVML-confirmed CUDA context holders under a process-tree root |
-| `validate` | Check an event pair or inventory mode and report requirements without attaching |
-| `measure` | Inventory bounded work or correlate bounded event evidence |
+| `validate` | Check an event pair, graph, or inventory mode without attaching |
+| `measure` | Triage, inventory, or correlate bounded live or recorded evidence |
 
 `measure --pid` automatically loads the matching CUDA 12 or CUDA 13 CUPTI Agent
 when a selected endpoint requires it. It reports the target mutation on stderr
@@ -99,9 +82,11 @@ attach cannot retrofit an initialized NVTX dispatch.
 | Surface | Current support |
 | --- | --- |
 | OS/architecture | Linux x86_64, glibc 2.34 or newer |
+| Triage | One window of thread time, cgroup throttling, CPU samples, syscalls, NVML health, and CUDA kernels |
+| Measurement graphs | Up to eight edges from one capture |
 | CPU inventory | PID-scoped sampled user stacks with native ELF and CPython perf-map symbols |
 | Host events | ELF functions, named syscall/tracepoint boundaries, and CPython GC USDT |
-| Coarse aggregation | Bounded syscall lifecycle and CUDA kernel/memcpy/memset groups |
+| Coarse aggregation | Bounded syscall lifecycle and CUDA kernel/memcpy/memset groups, with pageable/pinned copies |
 | CUDA callbacks | Runtime and Driver API entry/exit |
 | GPU activity | Kernel, memcpy, and memset start/end |
 | Application ranges | Bounded ASCII NVTX thread and process ranges |
@@ -121,6 +106,7 @@ offset CUPTI clock as host monotonic.
 - [CLI contract](docs/cli-contract.md)
 - [CUPTI agent and injection](docs/cupti-agent.md)
 - [Development and hardware tests](docs/development.md)
+- [Releasing](docs/releasing.md)
 - [Agent integration](docs/agent-integration.md)
 - [Public JSON schemas](schemas/)
 

benchmarks/cuda-multiprocess/run-container.pl

@@ -200,13 +200,14 @@ sub warning_present {
 }
 
 sub load_json {
-    my ($path, $description) = @_;
+    my ($path, $description, $schema_version) = @_;
+    $schema_version //= '2.0';
     my $value = eval { $JSON->decode(read_text($path)) };
     fail("malformed $description JSON at $path: $@") if $@;
     require_condition(ref($value) eq 'HASH',
         "malformed $description JSON at $path: expected object");
     require_condition(defined($value->{schema_version})
-        && $value->{schema_version} eq '2.0',
+        && $value->{schema_version} eq $schema_version,
         "unexpected schema in $description");
     return $value;
 }
@@ -659,7 +660,7 @@ sub run_batch_body {
     require_condition($inventory_exit == 0,
         'representative aggregate inventory command failed');
     verify_identity($representative, 'after aggregate inventory');
-    my $inventory = load_json($inventory_stdout, 'aggregate inventory');
+    my $inventory = load_json($inventory_stdout, 'aggregate inventory', '3.0');
     require_condition($inventory->{ok} && $inventory->{status} eq 'completed',
         'aggregate inventory is not complete');
     my $inventory_collection = $inventory->{collection};
@@ -802,7 +803,7 @@ sub run_batch_body {
             "worker $worker->{ordinal} exact measurement command failed");
         verify_identity($worker, 'after exact measurement');
         my $exact = load_json($worker->{files}{measure_stdout},
-            'exact measurement');
+            'exact measurement', '3.0');
         my $quality = validate_exact_result($exact, $worker);
         my $injection_warning =
             warning_present($exact, 'CUPTI_AGENT_INJECTED');

cupti/include/xprobe/cupti_agent.h

@@ -10,16 +10,17 @@ extern "C" {
 #define XPROBE_CUPTI_AGENT_ABI_VERSION 4U
 #define XPROBE_CUPTI_OUTPUT_MAGIC "XPCUPTI"
 #define XPROBE_CUPTI_CONTROL_MAGIC "XPCTRL\0"
-#define XPROBE_CUPTI_CONTROL_VERSION 4U
+#define XPROBE_CUPTI_CONTROL_VERSION 5U
 #define XPROBE_CUPTI_NAME_LENGTH 128U
-#define XPROBE_CUPTI_FILTER_COUNT 2U
+#define XPROBE_CUPTI_FILTER_COUNT 8U
 #define XPROBE_CUPTI_VALUE_UNKNOWN UINT32_MAX
 
 enum xprobe_cupti_feature {
     XPROBE_CUPTI_FEATURE_HOST_MONOTONIC_TIMESTAMPS = 1U << 0,
     XPROBE_CUPTI_FEATURE_TRANSFER_RECORDS = 1U << 1,
     XPROBE_CUPTI_FEATURE_AGGREGATE_RECORDS = 1U << 2,
-    XPROBE_CUPTI_FEATURE_NVTX_RECORDS = 1U << 3
+    XPROBE_CUPTI_FEATURE_NVTX_RECORDS = 1U << 3,
+    XPROBE_CUPTI_FEATURE_MEMORY_KINDS = 1U << 4
 };
 
 enum xprobe_cupti_capture_mode {
@@ -80,7 +81,8 @@ struct xprobe_cupti_control_request {
     uint64_t record_capacity;
     uint64_t record_offset;
     uint32_t capture_mode;
-    uint32_t reserved;
+    /* ARM: active filters in filters[0..filter_count); other commands: 0. */
+    uint32_t filter_count;
     struct xprobe_cupti_filter filters[XPROBE_CUPTI_FILTER_COUNT];
 };
 
@@ -144,7 +146,11 @@ struct xprobe_cupti_aggregate_record {
     uint32_t kind;
     uint32_t device_id;
     uint32_t memcpy_kind;
-    uint32_t reserved[5];
+    /* CUpti_ActivityMemoryKind of a memcpy's source and destination; 0 for
+     * other activity. Valid when XPROBE_CUPTI_FEATURE_MEMORY_KINDS is set. */
+    uint32_t src_memory_kind;
+    uint32_t dst_memory_kind;
+    uint32_t reserved[3];
     char name[XPROBE_CUPTI_NAME_LENGTH];
 };