Make ARM Machine goals authoritative and acknowledge RP node updates

[Thalia Wang (wenxuan0923)]

Summary

• Treat an existing ARM Machine as the source of truth during bootstrap and reconciliation.
• Remove client-side goal fallbacks and use the Machine ETag exclusively as SettingsVersion.
• Persist complete current and previous applied goals, rejecting state without an AppliedGoal.
• Add required AKS-managed node labels without including them in ARM custom labels.
• Acknowledge RP-applied label/taint-only updates when already reflected on the Kubernetes Node, avoiding unnecessary repaves. Other goal changes continue to require the existing node-deletion signal.

Goal-state ownership

Local configuration seeds the initial goal with Kubernetes version, max pods, labels, taints, and kubelet image-GC thresholds. After Get or Create returns a valid Machine, its complete goal replaces the config-derived goal.

The final nspawn goal remains hybrid: Kubernetes version, labels, and taints come from the Machine goal, while networking, credentials, runtime versions, images, mounts, and other host settings remain config-derived.

When best-effort Machine registration fails, bootstrap continues with the local goal and an empty SettingsVersion.

[hbc (bcho)]

this might be problematic if i use a config with the same machine name across multiple different hosts? This could result in different hosts trying to join as the same node to the cluster

[Thalia Wang (wenxuan0923)]

Yea this is a real risk but it seems to be pre-existing tho...? The old code also accepted the Machine when versions matched, or overwrote it when they differed...

[hbc (bcho)]

that is true... maybe ask in a different way: why we want to override with remote state instead of following the local config? If we want to override it, maybe just ignore the version from config at all? Also I am curious about the impact of preflight and offline bootstrapping as both steps are relying on goal state here

[Thalia Wang (wenxuan0923)]

We override it because the ARM Machine resource should be the source of truth once it exists. The local config is only used for the initial bootstrap and is never updated. If the customer later updates labels or taints through PUT Machine, or upgrades k8s version through PUT AgentPool calls, we want to use those remote values. Otherwise, a restart or repave could use the stale local config and undo the updates.

We still need the version from config to create the Machine initially, or as a fallback if optional registration fails. Once we get a valid Machine, we use its version instead.

[hbc (bcho)]

We still need the version from config to create the Machine initially, or as a fallback if optional registration fails. Once we get a valid Machine, we use its version instead

But isn't this the current behavior before the change?

[hbc (bcho)]

how about this, we add a check in preflight check to 1) fetch the machine state 2) warn or fail if the k8s version doesn't match between the config and remote config. I don't want us to force upgrade in this code path because the change from this PR ignores the preflight check result, which could make the offline bootstrap fails.

[Thalia Wang (wenxuan0923)]

Added a new preflight check for comparing the remote/local config here: fc907f4

[hbc (bcho)]

Tracking a reconciliation invariant identified while working on #263:

NodeReboot is a soft reboot that calls RestartNode, which re-runs nodestart.StartNode and re-renders the active nspawn machine configuration. Any value sourced from the authoritative Machine goal during bootstrap/repave must therefore also be restored from the persisted AppliedGoal during NodeReboot; otherwise it can revert to the local config value.

maxPods and image GC thresholds are currently immutable on the AKS side, so #263 intentionally leaves their existing goal ownership/persistence unchanged. However, #256 is the right place to establish this invariant generally for current and future Machine-owned fields. The AppliedGoal persistence and use from RestartNode in this PR appear to provide the correct foundation.

Suggested regression coverage:

1. Apply a Machine goal containing mutable Machine-owned values.
2. Persist it as AppliedGoal.
3. Simulate daemon restart/state reload.
4. Resolve a NodeReboot.
5. Verify all Machine-owned values still come from AppliedGoal, while config-owned values continue to come from local config.

If max pods or image GC thresholds become mutable later, they should be added to daemon.ResolveMachineGoalState and covered by the same test rather than being applied only in the repave path.