Io cpp bench default

[amirakb89]

Summary

Makes the native C++ benchmark (tests/cpp/io/bench_engine) the default
MORI-IO benchmark across the runner, CI, and docs, while keeping the Python
benchmark (tests/python/io/benchmark.py) available for parity. The C++ path
has no Python interpreter in the measurement loop and matches nixlbench timing,
so numbers are cleaner and directly comparable to NIXL.

What changed

Engine (code)

• bench_engine emits perf records. bench_engine now appends one JSON line
  per sweep point to $MORI_PERF_OUT, in the exact schema
  tests/python/perf_report.py writes, so the C++ and Python benches feed the
  one report pipeline (tools/perf/build_report.py). It captures per-iteration
  timing to report the Python bench's per-iteration metrics (avg/max bandwidth,
  avg/min latency) alongside the nixl whole-loop figure. No-op unless
  MORI_PERF_OUT is set; initiator-only; flock-appended.
• Runner engine switch. tools/run_internode_io_benchmark.sh takes
  --engine cpp|python (default cpp). The cpp path locates the built binary,
  sets LD_LIBRARY_PATH, and translates the torchrun rendezvous args
  (--master-addr/--host) to the C++ socket-bootstrap args
  (--master-ip/--self-ip). The python torchrun path is unchanged.

CI

• ci.yml: builds bench_engine on both nodes before the internode IO steps.
  The IO steps already call the runner with no --engine, so they switch to C++
  automatically.
• nightly.yml: same build step; drops the now-redundant standalone C++ bench
  step -- the write-sweep step covers it via the runner and emits perf records
  through the existing MORI_PERF_OUT wiring. The read smoke step keeps
  MORI_PERF_OUT cleared.

Docs

• docs/MORI-IO-BENCHMARK.md now leads with the C++ benchmark and keeps Python
  as the parity alternative; adds a "Which benchmark to use" section. The Fabric
  (UALink super-node) section is preserved and noted as also supported by C++.
• README.md and docs/MORI-IO-GUIDE.md point at the C++ default.

Validation

Verified on a two-node MI308X / ionic (AINIC) cluster:

• C++ and Python benches produce matching bandwidth across a 1--8 MiB GPU write
  sweep; C++ ~10-20% faster; byte-exact validation on both.
• The emitted C++ perf JSONL renders correctly through build_report.py.
• Both runner paths (--engine cpp default and --engine python) run two-node.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 499a401d2a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Create the perf output directory before opening the file

On a clean nightly workspace, MORI_PERF_OUT is $GITHUB_WORKSPACE/_perf/perf.jsonl, but no step creates _perf before the MORI-IO write sweep. Unlike Python's record_perf, this implementation calls fopen without creating the parent, so it logs an error, continues successfully, and silently omits every IO row from the uploaded performance data. Create the parent directory before opening the file.

Useful? React with 👍 / 👎.

[amirakb89]

"Fixed in c442590. WritePerfRecords now creates the parent directory before fopen (mkdir-p, matching Python's
os.makedirs(..., exist_ok=True)), so on a clean nightly workspace the _perf dir is created and IO rows are no longer
dropped. Verified by pointing MORI_PERF_OUT at a non-existent nested path."

[chatgpt-codex-connector]

Emit perf records for single-process XGMI runs

When invoked with --backend xgmi --xgmi-single-process and MORI_PERF_OUT set, RunBenchmark returns through RunXgmiSingleProcess before reaching this sole writer call, so the C++ benchmark produces no perf records even though the equivalent Python mode does. Preserve the RunSweep results in the single-process path and pass them to WritePerfRecords there as well.

Useful? React with 👍 / 👎.

[amirakb89]

"Fixed in c442590. RunXgmiSingleProcess now captures the RunSweep results and calls WritePerfRecords, so
--xgmi-single-process emits perf records like the equivalent Python mode."

[maning00]

Thanks for moving the native benchmark into the common perf pipeline. I found three correctness issues in the new perf-recording path that can make the nightly data corrupt or nondeterministic; I think these should be addressed before merge:

1. Keep the flock held until buffered data has actually been flushed (tests/cpp/io/bench_engine.cpp:921-925).
   The current order is flock(LOCK_EX) -> fputs() -> flock(LOCK_UN) -> fclose(). Because FILE* is buffered, fputs() does not guarantee that the full payload has reached the file before the unlock. The nightly sweep emits about 9 KB per initiator; a same-sized glibc probe split this into an 8192-byte write while locked and a tail write from fclose() after the unlock. With eight initiators writing the same JSONL, another payload can therefore be inserted in the middle of a JSON line, and build_report.py will silently skip the malformed line.
   Please either call and check fflush(fh) while the lock is still held, or more simply leave the lock held and let fclose() flush, close, and release it. A forked multi-writer test that parses every resulting JSON line would cover this path.

2. Do not publish perf rows before correctness validation succeeds (tests/cpp/io/bench_engine.cpp:1245-1257, and the same ordering in RunXgmiSingleProcess at 1310-1324).
   Both paths currently call WritePerfRecords() before the checksum comparison. If the transfer completes but the bytes are corrupt, the benchmark returns non-zero only after the rows have already been written. The nightly workflow uploads perf JSONL with if: always() and always builds the report, so data from a known-failing run can become the newest displayed result.
   Please write records only after validation succeeds. For a multi-GPU run, aggregate the validation status from every pair first, so no process publishes a partial run when another pair fails.

3. Use one deterministic writer (or write one aggregate), rather than one writer per initiator (tests/cpp/io/ bench_engine.cpp:1222-1248).
   isInitiator is true in all eight initiator processes, so all eight append rows with identical params; the records contain no GPU/global-rank identity. build_report.py deduplicates these using the same op/backend/msg/batch series key and keeps the first record in the file. The reported GPU pair is therefore selected by process scheduling—rank 0 is additionally delayed by printing the aggregate—rather than by an explicit policy.
   If the report is intended to remain pairwise, the minimal fix is to let only globalRank == 0 write. If the intended metric is
   total eight-pair bandwidth, use the existing allgather results to construct and write exactly one aggregate record.