Skip to content

fix(deploy): verify Dragonfly Registry access after rollouts - #160

Merged
rcyw merged 4 commits into
inclusionAI:developfrom
hky1999:hky1999/registry-connectivity
Nov 11, 2026
Merged

rcyw merged 4 commits into
inclusionAI:developfrom
hky1999:hky1999/registry-connectivity

Conversation

@hky1999

@hky1999 hky1999 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Registry relocation can block Dragonfly image preparation when ingress relies only on CIDRs. This adds a companion Helm release for precise seed ingress and automatic authenticated Nydus verification, while leaving Registry Deployment/PVC ownership with its existing deployment.

The guard detects Registry/seed identity changes and checks every configured seed (including three- and five-seed profiles). Success is recorded only after all seeds pass and the identities remain stable. Policy creation is opt-in after reviewing existing client ingress. Curl failures, bounded redirects, cross-origin credential stripping, HTTPS proxy trust, credential-safe logging prerequisites and response/decompression limits are enforced. Helm leaves the Job-written state field uninitialized.

Validation:

  • 27 focused checker/chart/real-curl tests pass, plus Helm lint and deployment syntax checks.
  • Test coverage retains truncated HTTP 200, HTTPS CONNECT, redirect credential isolation, three-/five-seed compatibility, rollout consistency and ingress/state ownership. Duplicate cases and fake exec lifecycle fixtures were removed; environment-dependent RSS measurement is kept as separate qualification rather than a routine CI gate. Test code was reduced from 699 to 555 lines without changing runtime behavior.
  • Live qualification covered direct reads from all five production seeds, isolated Dragonfly HTTPS bootstrap reads and dummy-token logging, and server-side reapplication preserving Job-written state. Proxy cache-hit verification does not prove a fresh origin fetch.

Operational limits: asynchronous post-rollout verification, not a readiness gate or automatic rollback. Proxy probes require verified request-log redaction/suppression. The production checker was suspended when unsafe INFO logging was found; this PR does not roll Registry/Dragonfly or resume that job.

@iamWHTWD iamWHTWD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the chart and checker at 317ba889. The local test suite passed. The inline comments cover first-install Registry access, probe reliability, HTTPS and credential handling, and Helm upgrade behavior.

Comment thread deploy/registry-connectivity/check.py Resolved
Comment thread deploy/registry-connectivity/check.py Resolved
Comment thread deploy/registry-connectivity/check.py Resolved
Comment thread deploy/registry-connectivity/check.py Resolved

rcyw commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Reviewed head 317ba8894a43bf9645173d4e681de02e009b75e0 against develop@db99c935f85b5935b9ce79e46860ab6ff5bc8948.

The companion-release design is useful: keeping Registry Deployment/PVC ownership separate, ANDing namespace and Pod selectors, restricting exec to configured Pod names, and advancing success state only after stable verification are sensible boundaries. The existing inline review tracks other ingress, transport, credential and operational concerns. I reproduced two additional compatibility issues to address alongside those threads before merging.

R1 — [P2] Support the configured seed count rather than requiring exactly five

Both the chart and fingerprint() reject any seed count other than five. AKernel's default Aliyun seed pool size is three, and that size is passed through to seedClient.replicas. The new deployment entry point recommends this guard for private Registry/Dragonfly deployments, but it cannot be used with that normal profile.

Reproduction:

  • Set dragonfly.seedNames to three distinct names: Helm rendering fails with exactly five distinct Dragonfly seedNames are required.
  • Call the real checker with the same three configured names and all three Pods Ready: it still fails with all five configured Dragonfly seeds must be present and Ready.
  • The existing five-seed tests pass.

Please accept a nonempty, duplicate-free configured list, require every listed seed to be present/Ready and to pass verification, and generate the same name-scoped exec RBAC from that list. Five can remain an example/default profile; installing a connectivity guard should not require expanding an existing three-seed deployment. Add positive three-/five-seed cases and negative missing, unready and duplicate-name coverage.

R2 — [P2] Handle valid Registry blob redirects

The curl invocation does not follow redirects, and the bootstrap read treats a non-200 response as a failed check. Blob redirects are part of the Distribution API, and the OCI specification allows redirects while restricting Authorization forwarding across host boundaries. A healthy redirecting/object-storage-backed Registry can therefore produce permanently failed guard Jobs.

I exercised the actual SeedHTTP.get() and verify_seed() against loopback HTTP fixtures, replacing only Kubernetes exec transport with a local subprocess running the same head | curl | base64 command:

  • The direct-200 control passed, including manifest/bootstrap validation and a proxy-form HTTP request.
  • Changing the blob response to a same-origin 307 with a readable Location produced authenticated bootstrap read failed.
  • A bounded curl --location control downloaded exactly the expected layer bytes and SHA-256.

Please support bounded redirects with an explicit credential-forwarding policy and retain final digest/size validation. Also parse the final response body correctly: simply adding --location is insufficient with --include, because the current decoder skips CONNECT/100 blocks but not intermediate redirect headers. Add direct/proxy redirect, loop/limit, and cross-origin credential-isolation regressions; do not enable unrestricted credential forwarding.

Validation and scope

The existing 18 checker/chart tests, make deploy-script-check, and Helm lint with complete fixture values all passed locally (Python 3.12.3, PyYAML 6.0.3, Helm 3.22.0). The additional reproduction had one passing direct-200 control and three failing assertions corresponding to the two findings above. CI run 42772539960 has 11 successful jobs and publication skipped, but its current tests do not cover these compatibility branches.

These were isolated local fixtures, not a live CNI/Kubernetes-exec or production rollout qualification. The reported five-seed production results do not cover a three-seed topology or a redirecting Registry. Please rerun the affected tests and CI on the resulting head after the fixes.

Non-blocking follow-ups: add build/minimal-start coverage for the checker image and pinned dependencies, and distinguish proxy cache-hit verification from evidence of a fresh origin fetch.

@hky1999
hky1999 marked this pull request as ready for review November 11, 2026 09:23

hky1999 commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

@rcyw R1/R2 are addressed in daf07fd: any nonempty unique seed list is supported with matching exec RBAC, and blob redirects are bounded with cross-origin credentials stripped and final digest/size checks retained. Three-/five-seed and direct/proxy redirect regressions pass. Latest head f5f6c22 has 27 passing local tests after deduplication; CI is running. Cache-hit scope is clarified in the README; image build/minimal-start automation remains a non-blocking follow-up.

rcyw commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Re-reviewed head f5f6c22c96b4b15c8fc557002cdeb2eca60f9074. Both R1 and R2 from my earlier review are resolved. I found no remaining blocking issue or direct SDK/runtime regression within the reviewed scope, and the current code is suitable for approval.

Compatibility fixes verified

  • Configured seed counts: the chart and checker now accept a nonempty, duplicate-free configured list, require every listed seed to be present/Ready and to pass, and generate exec RBAC from that same list. Three- and five-seed profiles are covered.
  • Registry blob redirects: reads follow a bounded chain as separate requests, preserve same-origin authentication, permanently strip Authorization/Proxy-Authorization/Cookie after crossing an origin, and retain final digest/size checks. Loop/limit, non-HTTP, credentialed-URL and HTTPS-downgrade failures are covered.
  • I reran all four assertions from the original independent reproduction: three-seed chart rendering, three-seed checker eligibility, the direct-200 control, and a real HTTP 307 bootstrap read now pass. The only fixture adaptation retained the new logging/CA prerequisites when updating the seed list; no assertions were relaxed.

I also checked the source changes and regressions corresponding to the other review threads: NetworkPolicy is opt-in with an existing-client review requirement; empty/invalid port lists fail; curl exit status is checked independently of HTTP status; HTTPS proxy trust is explicit; proxy probes require verified credential-safe logging; Helm no longer initializes the Job-owned state field; and response/decompression limits plus two-seed concurrency bound the workload.

Validation

  • Local python -m unittest discover -s deploy/registry-connectivity/tests -v: 27 tests passed, including actual curl truncation, TLS CONNECT, and direct/proxy redirect cases.
  • make deploy-script-check, Helm lint with complete fixture values, and the four original reproduction assertions all passed.
  • Local tools: Python 3.12.3, PyYAML 6.0.3 and Helm 3.22.0; verification used isolated checkouts and loopback fixtures.
  • CI 42994713646 passed all 11 applicable jobs; publication was skipped. I checked the contract-job log: it tested merge 07cb8ffb968c3da77629c78a1275d3af140bea73, combining this head with develop@b58153276357a18d3cc0db28cdb10cb3a8072208, and actually ran the 27 new tests. Head and CI are unchanged on the final recheck.

The production checker is reported suspended after unsafe proxy INFO logging was discovered. Production proxy checks still require verified logging/redaction and CA prerequisites; code approval does not qualify or resume that deployment. This review did not perform live CNI enforcement, a Helm 4 upgrade, a production rollout, or peak-RSS qualification. The documentation distinguishes those boundaries and cache-hit verification from a fresh origin fetch.

Checker-image build/minimal-start automation remains a non-blocking follow-up.

@rcyw rcyw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving verified head f5f6c22c96b4b15c8fc557002cdeb2eca60f9074. R1/R2 from my earlier review are resolved; the 27 focused tests, four independent reproduction assertions, Helm lint and deployment syntax checks passed. CI 42994713646 is green on this head's merge with develop@b581532.

See the detailed re-review and validation results, including the production logging/CA prerequisites and qualification limits. Checker-image build/minimal-start CI remains non-blocking. No merge performed.

@rcyw
rcyw merged commit 3d3dbdc into inclusionAI:develop Nov 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants