Skip to content

ip: preserve connected routes during protocol replay - #703

Draft
hcaldicott wants to merge 1 commit into
DPDK:mainfrom
hcaldicott:fix/connected-routes
Draft

hcaldicott wants to merge 1 commit into
DPDK:mainfrom
hcaldicott:fix/connected-routes

Conversation

@hcaldicott

@hcaldicott hcaldicott commented Aug 17, 2026

Copy link
Copy Markdown

During control-plane replay, a routing daemon can briefly advertise a
connected prefix before it has learned the interface address, then
withdraw it again once it reconciles. route4_add() replaces the
address-owned connected route with the protocol route, so the later
withdrawal removes the only route to the prefix while the address stays
configured — the connected network is unreachable until the address is
removed and re-added.

This change treats address-owned connected routes as authoritative:
protocol adds for such prefixes are accepted as a silent no-op, and API
deletes of internal-origin routes are refused the same way. Deletions now
resolve their nexthop with an exact-prefix lookup first, so the nexthop
type of a covering route is never used to delete a more specific prefix;
the previous longest-prefix lookup remains as the fallback when no exact
entry exists. Same handling for both address families.

Found while integrating FRR restart handling for EVPN multihoming (#698),
but the race only needs a routing daemon replaying routes over a
configured address.

Testing

  • New smoke test connected_route_ownership_test.sh replays the
    add/withdraw cycle over configured IPv4 and IPv6 addresses and asserts
    the connected routes survive with their link origin.
  • Without the fix (test-only applied to main): the test fails at the
    first assertion — after add/withdraw, 172.16.0.0/24 is gone from the
    RIB entirely.
  • With the fix: the new test passes, the full unit suite passes, and
    ip_loadbalance_test.sh confirms ordinary route add/del and ECMP
    behaviour is unchanged (AlmaLinux 9 container, arm64).

Related: #698

@rjarry rjarry left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Harrison,

Your check is a bit loose. See how to make it more robust below.

Also could you remove any mention that you were assisted by AI? It does not bring much value and we prefer to keep the git history leaner.

Thanks!

Comment thread modules/ip/control/route.c
Comment thread modules/ip/control/route.c
Comment thread modules/ip6/control/route.c
Comment thread modules/ip6/control/route.c
@hcaldicott
hcaldicott force-pushed the fix/connected-routes branch from 3b4929b to b973e2e Compare August 20, 2026 00:18
@hcaldicott

Copy link
Copy Markdown
Author

Hey Harrison,

Your check is a bit loose. See how to make it more robust below.

Also could you remove any mention that you were assisted by AI? It does not bring much value and we prefer to keep the git history leaner.

Thanks!

Good pickups - I will address these and re-run my lab tests!

@hcaldicott
hcaldicott force-pushed the fix/connected-routes branch from b973e2e to 25cac2c Compare August 20, 2026 00:52
@hcaldicott
hcaldicott marked this pull request as draft August 20, 2026 00:59
@hcaldicott

Copy link
Copy Markdown
Author

I have updated the PR. I am now rebuilding our physical lab on this to re-test the corrected behaviour. Converting back to a draft until my testing is completed.

@rjarry

rjarry commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Hey @hcaldicott, did you make progress with your testing?

@hcaldicott

Copy link
Copy Markdown
Author

Hey @hcaldicott, did you make progress with your testing?

Sorry for the slow turnaround - I've been quite busy and had kind of forgotten to finish this off - thanks for the bump to remind me. I have now re-tested the reworked guards on the physical cluster. Converting back from draft.

@hcaldicott

Copy link
Copy Markdown
Author

Testing is done — the reworked guards are now validated on our physical lab (4x Lenovo SE350, X722 fabric, iBGP EVPN full mesh, FRR 10.6.1 with the grout dplane plugin).

Setup: the PR head cherry-picked onto our production branch, packaged through our staging pipeline (unit suite runs in the package build), and installed on one fabric node carrying live EVPN services. The node's address-owned routes are two fabric /31s (origin link), the /32 router-id (origin internal) and the per-interface IPv6 link-local /64s.

Results:

Ownership guards, exercised directly against the real prefixes: deleting any of the address-owned routes is refused with EBUSY; a protocol-style re-add of a connected /31 via the fabric peer is accepted as a no-op and the route keeps its link origin.
5 consecutive FRR restarts (full zebra replay through dplane_grout): all connected routes present after every cycle, BGP re-Established 3/3 in ~14s, all EVPN prefixes re-received, RIB size stable. No dplane errors logged.
One full grout restart: the address replay recreates the connected routes and the immediately following FRR replay does not displace them — this is exactly the race that originally motivated the patch.

@hcaldicott
hcaldicott marked this pull request as ready for review September 3, 2026 08:30
Comment on lines +414 to +415
if (is_addr_owned_route(req->vrf_id, GR_IFACE_ID_UNDEF, &req->dest.ip, req->dest.prefixlen))
return api_out(0, 0, NULL);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Link-local ownership lookup misses 🐞 Bug ≡ Correctness

Both IPv6 ownership checks use GR_IFACE_ID_UNDEF, so link-local destinations are scoped to
interface zero and cannot match connected routes stored under their owning interface. An FRR replay
using a nexthop ID can therefore replace the connected route, while the subsequent withdrawal also
looks under the wrong scope.
Agent Prompt
## Issue description
IPv6 connected-route ownership checks use `GR_IFACE_ID_UNDEF`. For link-local prefixes this produces a different RIB key from the interface-scoped key used when the connected route was installed, allowing protocol replay to replace it.

## Issue Context
`addr6_linklocal_scope()` embeds the interface ID into link-local RIB keys. Resolve the add request's nexthop before checking ownership and use its interface ID; ensure deletion detects address-owned link-local routes despite lacking an interface field in its request.

## Fix Focus Areas
- modules/ip6/control/route.c[403-485]
- modules/ip6/control/ip6.h[23-35]
- modules/ip6/control/address.c[201-230]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcaldicott do you think this is a legit concern? I don't think FRR would try to replace a link local route. That does not make much sense to me. And we don't have any interface ID to use here anyway.

@rjarry rjarry left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for testing again!

During control-plane replay, a routing daemon can briefly advertise a
connected prefix before it has learned the interface address, then
withdraw it again once it reconciles. route4_add() replaced the
address-owned connected route with the protocol route, so the later
withdrawal removed the only route to the prefix while the address
stayed configured, cutting off the connected network until the
address was re-added.

Add is_addr_owned_route() which checks that the route's RIB origin is
internal and that its nexthop carries the local address flags. Accept
protocol adds for such prefixes as a silent no-op and refuse deleting
them with EBUSY. Deletions resolve their nexthop with an exact-prefix
lookup first so the nexthop type of a covering route is never used to
delete a more specific prefix; the previous longest-prefix lookup
remains as the fallback when no exact entry exists.

Add a smoke test replaying the add/withdraw cycle over configured IPv4
and IPv6 addresses, checking the withdrawal is refused and the
connected routes survive.

Signed-off-by: Harrison Caldicott <harrison@itsfubar.com.au>
Reviewed-by: Robin Jarry <rjarry@redhat.com>
@grout-bot
grout-bot force-pushed the fix/connected-routes branch from 25cac2c to 0b6e7cf Compare September 3, 2026 08:33
@hcaldicott
hcaldicott marked this pull request as draft September 3, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants