fix(server): refresh the device cache on every watch tick

[pescn]

What type of PR is this?

Bug fix. This repository has no kind/* labels, so the /kind command cannot be used here. The enhancement label currently on this PR was applied automatically and is not accurate; please relabel it to bug.

What this PR does / why we need it:

watchAndRegister refreshed the cached device view only while GetUnHealthIDs() reported at least one unhealthy device:

unhealthy := ps.mgr.GetUnHealthIDs()
if len(unhealthy) > 0 {
    if err := ps.mgr.UpdateDevice(); err != nil { ... }
    ps.healthCh <- unhealthy[0]
}

When a device is unhealthy while the plugin starts — a node that boots faster than the Ascend driver initialises — and recovers shortly afterwards, that condition is already false on the next iteration. The cache keeps the startup snapshot forever, and because ListAndWatch only sends again when it receives on healthCh, kubelet also keeps the startup device list.

The node then advertises no allocatable devices and Pods fail admission with Allocate failed due to no healthy devices present, while npu-smi reports every device healthy. registerHAMi() keeps publishing the same stale snapshot to hami.io/node-register-<commonWord>, so the scheduler sees the stale health too. Only restarting the Pod recovers the node. The same condition keeps a partially enumerated device list from ever growing.

This PR:

• refreshes the device cache on every iteration of watchAndRegister;
• publishes a ListAndWatch update whenever the device set changes, in either direction, using a fingerprint over the published device UUIDs and their health so the steady state stays free of redundant updates;
• bounds the healthCh send, so that a missing ListAndWatch consumer cannot stall the loop and thereby also stop the HAMi node registration that follows it (healthCh is unbuffered);
• adds regression tests for recovery, late device discovery, steady state, and the missing-consumer case.

Which issue(s) this PR fixes:

Fixes #129

Special notes for your reviewer:

• The change is limited to the device-cache refresh loop. Device allocation, scheduling, resource accounting, vNPU handling, and both slicing modes are untouched; go.mod / go.sum are unchanged.

• ListAndWatch itself is not modified, so this does not overlap with fix(server): stop ListAndWatch on stream closure #126 (stream-closure handling); the two are complementary — that PR fixes the consumer side of the stream lifecycle, this one fixes the producer side never producing.

• healthUpdateSendTimeout is a package-level variable rather than a constant only so the regression test can shorten it.

• Each of the four new tests fails on main and passes with this change:

  test	failure on main
  TestWatchAndRegister_RepublishesRecoveredDevices	expected kubelet to receive an updated device list, got 1 response(s)
  TestWatchAndRegister_PublishesLateDiscoveredDevices	expected 16 devices in the last response, got 8
  TestWatchAndRegister_SteadyStateDoesNotResend	expected the device cache to be refreshed at least once
  TestWatchAndRegister_ContinuesWithoutListAndWatchConsumer	registration stops, annotation stays Health:false

Verification performed on macOS/arm64 (go1.26.6):

• make test — all packages pass
• go vet ./internal/... — clean
• golangci-lint v2.1.0 run ./internal/... — 0 issues
• go test -race -count=5 ./internal/server/ -run TestWatchAndRegister — clean
• device-plugin binary build — succeeds

Field context: the failure was observed on a two-node RKE2 v1.35.5 cluster with 4 × Ascend 910B4-1 per node running v1.4.0, and the observable signature is a disagreement between hami.io/node-register-<commonWord> and the device series exported by the plugin's own dcmi collector on :9395 (the latter reads the hardware live and stays correct during the failure).

Does this PR introduce a user-facing change?:

No.

Summary by CodeRabbit

• Bug Fixes
  • Device availability updates now reflect health recovery and newly discovered devices.
  • Unchanged device lists are no longer resent unnecessarily.
  • Device registration continues reliably even when no consumer is connected.
  • Updates honor shutdown signals and use bounded delivery timeouts.
• Tests
  • Added coverage for device recovery, late discovery, steady-state behavior, and registration without an active consumer.

[hami-robot]

@pescn: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

Details

In response to this:

What type of PR is this?

/kind bug

What this PR does / why we need it:

watchAndRegister refreshed the cached device view only while GetUnHealthIDs() reported at least one unhealthy device:

unhealthy := ps.mgr.GetUnHealthIDs()
if len(unhealthy) > 0 {
   if err := ps.mgr.UpdateDevice(); err != nil { ... }
   ps.healthCh <- unhealthy[0]
}

When a device is unhealthy while the plugin starts — a node that boots faster than the Ascend driver initialises — and recovers shortly afterwards, that condition is already false on the next iteration. The cache keeps the startup snapshot forever, and because ListAndWatch only sends again when it receives on healthCh, kubelet also keeps the startup device list.

The node then advertises no allocatable devices and Pods fail admission with Allocate failed due to no healthy devices present, while npu-smi reports every device healthy. registerHAMi() keeps publishing the same stale snapshot to hami.io/node-register-<commonWord>, so the scheduler sees the stale health too. Only restarting the Pod recovers the node. The same condition keeps a partially enumerated device list from ever growing.

This PR:

• refreshes the device cache on every iteration of watchAndRegister;
• publishes a ListAndWatch update whenever the device set changes, in either direction, using a fingerprint over the published device UUIDs and their health so the steady state stays free of redundant updates;
• bounds the healthCh send, so that a missing ListAndWatch consumer cannot stall the loop and thereby also stop the HAMi node registration that follows it (healthCh is unbuffered);
• adds regression tests for recovery, late device discovery, steady state, and the missing-consumer case.

Which issue(s) this PR fixes:

Fixes #129

Special notes for your reviewer:

• The change is limited to the device-cache refresh loop. Device allocation, scheduling, resource accounting, vNPU handling, and both slicing modes are untouched; go.mod / go.sum are unchanged.
• ListAndWatch itself is not modified, so this does not overlap with fix(server): stop ListAndWatch on stream closure #126 (stream-closure handling); the two are complementary — that PR fixes the consumer side of the stream lifecycle, this one fixes the producer side never producing.
• healthUpdateSendTimeout is a package-level variable rather than a constant only so the regression test can shorten it.
• Each of the four new tests fails on main and passes with this change:

test	failure on main
TestWatchAndRegister_RepublishesRecoveredDevices	expected kubelet to receive an updated device list, got 1 response(s)
TestWatchAndRegister_PublishesLateDiscoveredDevices	expected 16 devices in the last response, got 8
TestWatchAndRegister_SteadyStateDoesNotResend	expected the device cache to be refreshed at least once
TestWatchAndRegister_ContinuesWithoutListAndWatchConsumer	registration stops, annotation stays Health:false

Verification performed on macOS/arm64 (go1.26.6):

• make test — all packages pass
• go vet ./internal/... — clean
• golangci-lint v2.1.0 run ./internal/... — 0 issues
• go test -race -count=5 ./internal/server/ -run TestWatchAndRegister — clean
• device-plugin binary build — succeeds

Field context: the failure was observed on a two-node RKE2 v1.35.5 cluster with 4 × Ascend 910B4-1 per node running v1.4.0, and the observable signature is a disagreement between hami.io/node-register-<commonWord> and the device series exported by the plugin's own dcmi collector on :9395 (the latter reads the hardware live and stays correct during the failure).

Does this PR introduce a user-facing change?:

No.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: da41ac9b-8e74-43c7-8c9a-76e5b105f42d

📥 Commits

Reviewing files that changed from the base of the PR and between 1cf92f2 and 8e4a50f.

📒 Files selected for processing (3)

• internal/server/register.go
• internal/server/server.go
• internal/server/watch_and_register_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The server now refreshes device state on every timer tick, fingerprints device UUIDs and health states, republishes changed device lists, bounds update delivery, and preserves registration progress when no ListAndWatch consumer is connected. Tests cover recovery, late discovery, steady state, and shutdown behavior.

Changes

Device state republishing

Layer / File(s)	Summary
Watcher refresh and publication flow internal/server/register.go, internal/server/server.go	watchAndRegister refreshes devices on every tick and republishes only when device UUIDs or health states change. Start records the initial fingerprint, and update sends use a configurable timeout.
Watcher and registration regression coverage internal/server/watch_and_register_test.go	Tests cover recovered devices, devices discovered after startup, unchanged steady state, and registration without a ListAndWatch consumer.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to 8e4a5

This localized change refreshes device state and republishes updates when needed; no actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant watchAndRegister
  participant AscendManager
  participant ListAndWatch
  participant NodeRegistration
  watchAndRegister->>AscendManager: UpdateDevice on each timer tick
  AscendManager-->>watchAndRegister: Current device state
  watchAndRegister->>ListAndWatch: Publish when fingerprint changes
  watchAndRegister->>NodeRegistration: Register current device state

Loading

Possibly related PRs

• Project-HAMi/ascend-device-plugin#126: Both changes modify device publication and stream update behavior in the server.

Suggested labels: enhancement

Suggested reviewers: archlitchi, dsfans2014

Poem

A rabbit watches devices turn,
Healthy lights return and burn.
New ones join the published line,
Steady states send once, then shine.
Registration hops along—
Even when the stream is gone.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address issue #129 by republishing recovered and late-discovered devices without restarting the plugin.
Out of Scope Changes check	✅ Passed	The implementation and regression tests remain within the stated device-cache refresh and republishing objectives.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: refreshing the server device cache on every watch tick.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[archlitchi]

/lgtm

[hami-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: archlitchi, pescn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [archlitchi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment