bug(node): bootstrap-enrolled nodes and routers cannot recover autonomously after signing-key retirement

[toasterbook88]

Summary

In v0.1.0-alpha.9 (bcf9bc2), PR #339 introduced recoverStaleIdentity to allow OIDC-authenticated nodes to survive signing-key rotation by exchanging a stored refresh grant for a fresh ID token and re-enrolling under the new key.

However, headless nodes and routers enrolled via bootstrap tokens possess no OIDC refresh grant (LoadRefreshToken returns storage.ErrNotFound). When the control plane rotates signing keys past their grace period (--key-rotation-interval + --key-grace-period), the old signing key is retired and purged from s.store.GetAllValidKeys(ctx).

Once the key that signed the bootstrap identity's Biscuit is absent from GetAllValidKeys():

1. POST /refresh verification (identity.VerifyExpiredAndExtractPeerID) fails with biscuit: invalid signature before the control plane can consult the stored node record. The control plane returns 401 Unauthorized.
2. On daemon restart, node.Start detects that the stored identity fails role verification under current control plane keys. Because no refresh grant exists, recoverStaleIdentity fails, and the daemon crashloops with exit code 1 (loaded identity fails role requirement).
3. Routers calling POST /routers/lease with an identity signed by a retired key are similarly rejected with 401 Unauthorized.

As a result, bootstrap-enrolled nodes and routers become unable to recover autonomously after signing-key retirement. They are not permanently unrecoverable in an absolute sense, but recovery currently requires manual operator intervention: minting a new bootstrap token, clearing client state, and manually removing the stale cached record from the control plane's enrollment_requests table.

Accelerated Key-Retirement Test Matrix

To evaluate this behavior, we ran an accelerated rotation integration test against v0.1.0-alpha.9 (rotation interval: 5s, grace period: 2s, biscuit TTL: 24h):

Case	Scenario	Expected Behavior	Observed Result	Status
Case 1	Bootstrap node awake through rotation	Refreshes under new key while old key in grace	Node refreshed; adopted Key 2 Biscuit	PASS
Case 2	Bootstrap node suspended before rotation, resumes inside grace	Refreshes inside active grace window	Node refreshed while 2 keys valid	PASS
Case 3	Bootstrap node suspended across old-key retirement	Autonomous /refresh rejected (401)	HTTP 401 Invalid biscuit: signature verification failed: biscuit: invalid signature	PASS (expected failure reproduced)
Case 4	Daemon restart after old-key retirement	Startup fails; router rejects biscuit	loaded identity fails role requirement "sam:role:node"; router verification returns invalid signature	PASS (expected failure reproduced)
Case 5	Router lease renewal across old-key retirement	Lease renewal rejected (401)	HTTP 401 Router ... failed biscuit verification: no valid key found for verification	PASS (expected failure reproduced)
Case 6	OIDC node equivalent across old-key retirement	Autonomous recovery via refresh grant (#339)	Re-enrolled via /register with refreshed JWT; new Biscuit verified under Key 2; peer.ID preserved	PASS

Key Observations

• In Cases 1 and 2, as long as the Biscuit's signing key remains in GetAllValidKeys(), /refresh succeeds.
• In Case 3, the failure occurs at the cryptographic verification boundary in HandleRefresh:

  pID, err := identity.VerifyExpiredAndExtractPeerID(trustedKeys, currentBiscuitBytes, s.config.BiscuitTimeout)

  Because trustedKeys only includes non-retired keys from GetAllValidKeys(), signature verification fails before the handler ever reaches s.store.GetNode(ctx, pID.String()).
• In Case 6, OIDC nodes seamlessly survive key retirement because #339's recoverStaleIdentity fallback successfully contacts the OIDC token endpoint and re-registers. Headless bootstrap nodes have no equivalent autonomous recovery path.

Design Considerations for Remediation

Autonomous recovery for bootstrap nodes should be possible without requiring human operators to mint new tokens and reset state, while preserving core security properties:

• Proof-of-Possession: Any recovery or re-authentication must prove possession of the node's private key (node.key matching the public key stored in the control plane).
• Freshness & Replay Protection: Must adhere to replay/freshness protections (such as the timestamp and single-use checks recently reinforced in enforce freshness boundary on token refresh #350).
• Revocation & Admission: Banned nodes (s.store.BanNode / CheckAdmission) must not be permitted to renew or recover.
• Role & Policy Enforcement: The issuing role and policy bindings must be re-validated against current mesh policy.

Possible architectural options include:

1. Permitting /refresh to fall back to a challenge signed by the node's registered private key when the Biscuit's signing key has aged out, provided the node's public key is known and not banned.
2. An autonomous re-enrollment handshake where a node with a retired Biscuit can present a fresh challenge signed by its private key to obtain an updated Biscuit under the active signing key.