[scotwells]

Requested by Scot Schuchert-Wells · project thread

Summary

When an egress shard's node goes down, compute nodes keep sending tenant egress to it for the whole outage, so tenants lose internet access and a NAT64 standby is never promoted. The shard's SID route is withdrawn, but a compute node whose only other route is an upstream default over an SRv6 uplink still resolves the SID through that default and keeps counting the shard as alive. The reachability check now asks the kernel which FIB entry the SID matches and treats a default-route match as unreachable, so a withdrawn SID takes its shard out of every group on every node and the pool metrics and alerts see it. Kernels that cannot answer a FIB-match lookup keep the previous behaviour.

Related to datum-cloud/infra#7781

Test plan

• Unit tests and lint pass for the egress route package
• The new kernel test (default route over an uplink rejected, specific locator route accepted) passes in the privileged unit job
• In a lab, rebooting the primary shard's node moves its tenants to the remaining shard within seconds, and the pool reports it unreachable
• When the node returns, its shard rejoins and a NAT64 standby is held again

---

Generated by Claude Code

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[ecv]

Approve.

A SID that only matches a default route is now treated as unreachable, which matches how the fabric advertises shards and makes failover actually trigger. The check is narrow and fails open when the kernel cannot answer.

• rejectDefaultRoute uses RouteGetWithOptions(FIBMatch: true) in the same netns and table as the existing RouteGet, so it adds no new table or VRF assumption. A nil Dst or a /0 mask counts as default. A /1-style split default is not caught; acceptable.
• A FIBMatch error or empty answer returns nil, so an old kernel keeps the previous behaviour instead of failing every SID. Good.
• The local shard is unaffected: its SID resolves out a non-uplink link and was already rejected before this check, and held already exempts c.local.
• nat64Standby: a primary that only default-routes now has hop.err != nil, so anyPrimaryAlive goes false and the standby is released. That is the intended failover, and it flaps only as fast as the route does, since groups rebuild on route change plus the 30s backstop.
• The test covers default-only vs specific route over an uplink. It does not cover the case where every remote shard falls through to default (group_empty).

Follow-ups:

• The premise "the fabric advertises every SID as a specific route" is a deployment assumption. A lab or site that reaches peer shards only through a static default over an uplink will now mark every remote shard down and leave the class with zero active shards, dropping tenant NAT. Confirm the staging labs advertise per-locator routes before this ships to them, and consider a log line or metric when a candidate is rejected for default-only.
• Add a shardgroup test with all candidates default-only that pins the group_empty outcome and the standby release.

[scotwells]

Thanks for the review. Here's where the follow-ups stand:

• Per-locator routes: I checked every deployment that tracks main. Each egress node advertises its shard locator as its own /64, so live shards resolve through a specific route and only a withdrawn one falls to the default. A deployment that reaches peer shards only through a static default would lose them, which is the reason this change flips the rule.
• group_empty test and a rejection log line: these go in a follow-up PR, so this one can merge with its current approval. That PR will add a shard-group test where every candidate is default-only, pinning the group_empty outcome and the standby release, plus a log line when a candidate is rejected for matching only the default route.

---

Generated by Claude Code