Skip to content

Run the XPOIS, xFit and XScan device pipeline through Dragon - #43

Merged
tpn merged 2 commits into
mainfrom
codex/013-dragon-device-pipeline
Sep 23, 2026
Merged

tpn merged 2 commits into
mainfrom
codex/013-dragon-device-pipeline

Conversation

@tpn

@tpn tpn commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Run complete XPOIS, xFit, and XScan image-pair jobs in persistent Dragon workers, reusing one device context per worker. Compact launch and result descriptors bind each result to its inputs, configuration, and scientific evidence while preserving Torch-before-CuPy startup.

Live Dragon and GPU execution remain unverified for this revision.

@tpn
tpn requested a review from melo-gonzo September 22, 2026 13:31
@tpn tpn added the ai-review Request a focused CodeRabbit review label Sep 22, 2026
@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: NVIDIA/cuPhoton/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a7a4f2a9-45df-4e90-8616-469ffbc2b85f

📥 Commits

Reviewing files that changed from the base of the PR and between 7439af0 and 6348319.

📒 Files selected for processing (3)
  • docs/components/xscan.md
  • src/cuphoton/xscan/dragon_pipeline.py
  • tests/xscan/test_dragon_pipeline.py

Included review availability: Your plan provides up to 12 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds a persistent Dragon-based XScan device pipeline. It validates inputs and results, reuses device contexts across shard items, audits published summaries, and documents direct and Dragon execution.

Changes

XScan device pipeline

Layer / File(s) Summary
Pipeline contracts and validation
src/cuphoton/xscan/dragon_pipeline.py, tests/xscan/test_dragon_pipeline.py
Defines versioned schemas and validates identities, hashes, transfer receipts, results, and scientific evidence. Tests cover malformed results and evidence that conflicts with configuration.
Preflight and persistent worker execution
src/cuphoton/xscan/dragon_pipeline.py, tests/xscan/test_dragon_pipeline.py, docs/components/xscan.md
Checks configuration and input files, validates worker options, and builds work items. Each shard initializes one device context and reuses it across items. Tests cover preflight rejection, GPU identity probing, and context reuse or initialization failure. Documentation describes pipeline requirements, input descriptors, and direct execution.
Batch orchestration and terminal auditing
src/cuphoton/xscan/dragon_pipeline.py, tests/xscan/test_dragon_pipeline.py, docs/components/xscan.md
Adds run_dragon_device_pipeline with Dragon manifests, timeouts, and record validators. Tests cover coordinator audits of file-backed summaries, runtime records, and device identity. Documentation describes Dragon execution and result handling.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to 63483

This change adds a validated, persistent Dragon pipeline for XPOIS, xFit and XScan jobs, along with tests and documentation. No concrete defect remains open against the current revision. The author reports that live multi-GPU Dragon runs are still unverified, so owners should confirm a real run, but nothing identified blocks the merge.

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch from f861bed to e6dfba1 Compare September 22, 2026 13:50
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from 2d9dc4f to 99f5ddc Compare September 22, 2026 13:50
@tpn tpn added ai-review Request a focused CodeRabbit review and removed ai-review Request a focused CodeRabbit review labels Sep 22, 2026
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from 99f5ddc to d440c19 Compare September 22, 2026 14:34
@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch from e6dfba1 to 9e71f82 Compare September 22, 2026 14:34
@tpn

tpn commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from d440c19 to f009ace Compare September 22, 2026 17:38
@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch from 9e71f82 to a5b6002 Compare September 22, 2026 17:38
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from f009ace to ee6c686 Compare September 22, 2026 18:59
@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch 2 times, most recently from 745fe81 to 5f5f45f Compare September 22, 2026 21:04
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from ee6c686 to 2c30ede Compare September 22, 2026 21:04

@melo-gonzo melo-gonzo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: the PR tree is byte-identical to the open GitLab branch for every file it touches, and my earlier findings remain as posted. Merge after the internal MR is merged so the histories match.

@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch from 5f5f45f to 305a333 Compare September 22, 2026 23:02
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from 2c30ede to 800e3d5 Compare September 22, 2026 23:02
@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch from 305a333 to 6eb72a5 Compare September 23, 2026 00:06
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch 2 times, most recently from a49391a to fbb50d9 Compare September 23, 2026 01:46
@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch 2 times, most recently from 36f3e8e to d9bb505 Compare September 23, 2026 02:21
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch 3 times, most recently from 2579bce to 3646c0a Compare September 23, 2026 03:32
@tpn

tpn commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch from bc5fe97 to e0589c6 Compare September 23, 2026 18:09
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from 97e0135 to fb53953 Compare September 23, 2026 18:09
tpn added 2 commits September 23, 2026 16:14
Reuse one placed device context across complete image-pair jobs. Bind
compact launch and result descriptors to input, configuration, and
scientific evidence identities, and retain Torch-before-CuPy startup.

Document direct execution and Dragon worker reuse with the required
checkpoint and feature-schema contracts.

Signed-off-by: Trent Nelson <trentn@nvidia.com>
Treat JSON integers outside floating range as invalid runtime metadata.
This lets the coordinator persist a failed terminal summary when a
worker's context-load duration cannot be represented as a finite number.

Signed-off-by: Trent Nelson <trentn@nvidia.com>
@tpn
tpn force-pushed the codex/013-prereqs-device-coordinator branch from e0589c6 to 7439af0 Compare September 23, 2026 23:18
@tpn
tpn force-pushed the codex/013-dragon-device-pipeline branch from fb53953 to 6348319 Compare September 23, 2026 23:18
@tpn
tpn changed the base branch from codex/013-prereqs-device-coordinator to main September 23, 2026 23:19
@tpn

tpn commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@tpn
tpn merged commit aa4bf18 into main Sep 23, 2026
12 of 30 checks passed
@tpn
tpn deleted the codex/013-dragon-device-pipeline branch September 24, 2026 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-review Request a focused CodeRabbit review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants