Race Conditions on LAC ( PPPoE LCP Termination - L2TP ICRQ ) cause Bng Blaster L2TP Session Counter Issue

[slieberth]

Description

While running LNS failover tests, we noticed a corner case that can lead to incorrect L2TP session statistics in the BNG Blaster.

The original root cause appears to be a race condition on the LAC/BNG side, where the teardown of a previous PPPoE session overlaps with the establishment of a new L2TP session.

In the specific scenario, the packet sequence looks like this:

Frame 10626   PPP LCP Termination Request (PPP session triggered the previous L2TP Tunnel)
Frame 10643   L2TP ICRQ
Frame 10680   L2TP ICRP
Frame 10683   L2TP CDN
Frame 11394   L2TP ICRQ retransmission

The LAC enters the disconnect path before the L2TP session establishment completes. As a consequence, the ICCN is never transmitted. After a timeout, the LAC retransmits the ICRQ and the session is established successfully on the second attempt.

Functionally, this is not an issue for the BNG Blaster, since the retransmitted ICRQ eventually results in a successful session establishment.

However, during this sequence we observe that the reported number of active L2TP sessions exceeds the number of active PPPoE subscribers.

For example:

PPPoE sessions: 100
l2tp-sessions : 104

Looking at bbl_l2tp.c, the L2TP session counter is incremented immediately when an ICRQ is received:

bbl_l2tp_icrq_rx(...)
{
    g_ctx->l2tp_sessions++;

    l2tp_session->state = BBL_L2TP_SESSION_WAIT_CONN;
}

In the failure scenario, the expected ICCN never arrives because the LAC has already entered the disconnect path. A retransmitted ICRQ therefore creates another L2TP session object before the previous one has been cleaned up.

I'm not sure whether this is considered expected behaviour or whether the session statistics should instead reflect only fully established L2TP sessions.

Version

0.9.35

JSON configuration

{
  "interfaces": {
    "capture-include-streams": false,
    "qdisc-bypass": false,
    "access": [
      {
        "interface": "enp1s0f2",
        "outer-vlan-min": 128,
        "outer-vlan-max": 129,
        "inner-vlan-min": 7,
        "inner-vlan-max": 7,
        "qinq": false,
        "stream-group-id": 1,
        "type": "pppoe",
        "authentication-protocol": "PAP"
      }
    ],
    "tx-interval": 0.1,
    "rx-interval": 0.1,
    "tx-threads": 2,
    "rx-threads": 2,
    "network": {
      "interface": "enp1s0f0",
      "address": "10.189.214.2/24",
      "gateway": "10.189.214.1"
    }
  },
  "pppoe": {
    "reconnect": true,
    "discovery-timeout": 3,
    "discovery-retry": 20,
    "vlan-priority": 6
  },
  "ppp": {
    "mru": 1492,
    "authentication": {
      "username": "lacX",
      "password": "X2022",
      "timeout": 1,
      "retry": 10
    },
    "lcp": {
      "conf-request-timeout": 5,
      "conf-request-retry": 30,
      "keepalive-interval": 30,
      "keepalive-retry": 3
    },
    "ipcp": {
      "enable": true
    },
    "ip6cp": {
      "enable": false
    }
  },
  "traffic": {
    "stream-burst-ms": 5
  },
  "sessions": {
    "count": 100
  },
  "access-line": {
    "agent-remote-id": "DEU.CUST.{session-global}",
    "agent-circuit-id": "49_123_456_ABCD xpon 0/0/{session-global}"
  },
  "l2tp-server": [
    {
      "name": "BBL1-LNS1",
      "address": "10.189.214.2",
      "secret": "bngblaster"
    },
    {
      "name": "BBL1-LNS2",
      "address": "10.189.214.3",
      "secret": "bngblaster"
    }
  ],
  "streams": [
    {
      "name": "downstream",
      "direction": "downstream",
      "length": 492,
      "pps": 13070,
      "type": "ipv4",
      "stream-group-id": 1,
      "vlan-priority": 0,
      "priority": 0
    },
    {
      "name": "upstream",
      "direction": "upstream",
      "length": 492,
      "pps": 3036,
      "type": "ipv4",
      "stream-group-id": 1,
      "vlan-priority": 0,
      "priority": 0
    }
  ]
}

How to Reproduce

The issue is timing-dependent and has not yet been reproducible on demand.

It was observed during repeated LNS failover tests with approximately 100 PPPoE subscribers distributed across multiple L2TP tunnels.

The failover was triggered by blocking one of the active L2TP tunnels, forcing the affected PPPoE sessions to reconnect via the remaining LNS.

Occasionally, the teardown of a previous PPP session overlaps with the establishment of a new L2TP session. In this situation, the LAC retransmits an ICRQ before the previously created L2TP session object has been cleaned up, causing the l2tp-sessions counter to exceed the actual number of active subscribers.

I am happy to provide details from the recorded capture file, both on the PPPoE and on L2TP side

[slieberth]

Hi Christian,

if you're happy to accept a pull request from me, I'd be happy to provide a small enhancement for the session counters:

• Count l2tp-sessions only after the ICCN has been received (i.e. only fully established sessions).
• Introduce an additional counter, l2tp-session-attempts, which is incremented when an ICRQ is received. This would reflect ongoing session establishment attempts.

pls, let me know ...

Stefan

[GIC-de]

Actually for me looks correct since the counter is incremented once session object is created and decremented once removed. After tunnel terminates (e.g. timeout), session count should become correct again.

[slieberth]

Hi Christian,

I did not wait for the L2TP echo timeout, as the automated test case currently aborts once the incorrect session counters are detected.

However, in my recorded test case, the l2tp-session is actually terminated by the LAC, which sends a CDN for this particular L2TP session. In theory, this should allow the LNS to remove the session object and decrement the session counter again.

Frame 10626 PPP LCP Termination Request (previous PPP session)
Frame 10643 L2TP ICRQ
Frame 10680 L2TP ICRP
Frame 10683 L2TP CDN
Frame 11394 L2TP ICRQ (new L2TP session from the LAC)

If I look at the code, the received CDN should trigger the removal of the session, including decrementing the session counter:

static void
bbl_l2tp_cdn_rx(...)
{
    ...
    l2tp_session->state = BBL_L2TP_SESSION_TERMINATED;
    ...
    bbl_l2tp_session_delete(l2tp_session);
}

So I would have expected the session counter to return to the correct value.