edgeHub: no shared cooldown for a known-dead upstream — each queued operation rebuilds the cloud proxy (874 rebuilds in 1s observed), retaining ~6 KB each

[guy-radford-sunswap]

Summary

On a device whose cellular uplink is detached by the network roughly once a minute, edgeHub rebuilds its
per-client cloud proxy ~874 times in a single second during each outage window, retaining roughly
6 KB per rebuild. Over 33 h this took edgeHub's cgroup anon from ~0 to 230 MiB, against otherwise
identical peer devices sitting at 56–84 MiB after 4.6 days.

The proximate cause looks like the absence of any shared cooldown for a known-dead upstream: while
the link is down, each queued operation independently drives a fresh cloud-proxy creation, and there is
nothing that says "the upstream failed 5 ms ago, don't rebuild yet".

On the EOL question, upfront: the device runs edgeHub 1.4.43 (versionInfo.json:
build 105359236, commit 9f72e9db5e558be4f68345e757e659bf489ef492, image digest
sha256:308604711ff15a9324f0fb281bb1d1eaf9e94b57ea27e85555e385eb3db8706d, built 2024-10-09) with daemon
aziot-edge 1.5.13. I know 1.4 is out of support and we are moving to 1.5.x regardless. I am filing
because the implicated code appears unchanged on main — see the source walk below — so this looks
like a live issue rather than an artefact of an EOL build. Happy to be told otherwise.

Environment

• edgeHub mcr.microsoft.com/azureiotedge-hub:1.4 → 1.4.43; edgeAgent :1.4; daemon aziot-edge 1.5.13
• Raspberry Pi CM4, 909 MiB RAM, ARM64, Linux containers, kernel 6.1.21-v8+
• edgeHub capped at 450 MiB (HostConfig.Memory = MemorySwap = 471859200)
• Upstream: LTE via a cellular modem. The network detaches the PDN session every ~59 s (confirmed
  independently by the carrier's own session records and by ModemManager: [cm] emm-detached). Radio is
  healthy throughout (RSRP −90 dBm). This is a carrier-side fault we are chasing separately — but it makes
  an excellent, if unwelcome, test rig for upstream-flap behaviour.

Current behaviour

Steady state between outages is ~1 cloud-proxy rebuild per 2 minutes. Then, in one second:

95  2026-07-30T07:54:50
874 2026-07-30T07:54:51   <-- 874 × "Client <id> connected to cloud, processing existing subscriptions."

The burst begins with:

[INF] - "Closing connection for device: <id>, , "
[INF] - Disposing MessagingServiceClient for device Id <id> because of exception -
[INF] - Setting device proxy inactive for device Id <id>
[INF] - Removing device connection for device <id> with removeCloudConnection flag 'True'.
[INF] - Retrying cloud proxy operation SendMessageAsync for <id>.
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Amqp resource is disconnected.
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSendingLink.SendAmqpMessageAsync(AmqpMessage amqpMessage, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.SendEventAsync(Message message, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck)
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action)
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage)

Within that same burst minute there are also 766 × Skipping <id> for subscription processing, as it is currently being processed, i.e. rebuilds arrive faster than subscription processing completes.

Quantified over a 7.6 h window: 39 burst-minutes containing 34,177 of 34,391 total rebuilds (99.4 %),
~5 bursts/hour, ~900 rebuilds each. ~60 network detaches/hour, so only ~8 % of detaches escalate into a
burst — presumably those landing while an operation is in flight.

⚠️ Burst size grows monotonically over one container lifetime: 481 rebuilds at 00:35 → 962 at 07:46,
roughly doubling in 7.6 h with no restart. My best guess is the store-and-forward backlog growing, so each
outage has progressively more queued operations to fail — but I have not confirmed that.

Memory consequence

Two windows, same container, no restart in between:

window	bursts	rebuilds	cgroup anon
07:41:14 → 07:47:46 (392 s)	2	1,916	217.9 → 229.7 MiB (+11.8 MiB)
08:04:12 → 08:16:21 (729 s, 13 samples)	0	~6	flat, 230.5–233.6 MiB

⇒ ~6.3 KB retained per rebuild. The second window is the control: the 59 s detach cadence continued
throughout it (1 detach/min in all 13 samples) while the heap did not move — so growth tracks rebuilds,
not detaches and not elapsed time. Net long-run retention after GC is lower, ~1.4 KB/rebuild (~7 MiB/h).

Source walk — why I think there is no shared cooldown

From edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/cloud/RetryingCloudProxy.cs on main:

const int RetryCount = 2;
...
async Task<T> ExecuteOperation<T>(Func<ICloudProxy, Task<T>> func, string operation)
{
    int i = 0;
    while (true)
    {
        ICloudProxy cloudProxy = await this.GetCloudProxy();
        try { return await func(cloudProxy); }
        catch (Exception e)
        {
            if (cloudProxy.IsActive) { Events.ThrowExceptionProxyActive(...); throw; }
            if (++i == RetryCount) { Events.ThrowExceptionRetryCountReached(...); throw; }
            Metrics.AddRetryOperation(this.id, operation);
            Events.Retrying(this.id, e, operation);   // the log line above
        }
    }
}

async Task<ICloudProxy> GetCloudProxy()
{
    if (!this.innerCloudProxy.IsActive)
    {
        using (await this.cloudProxyLock.LockAsync())
        {
            if (!this.innerCloudProxy.IsActive)
            {
                Events.GettingNewCloudProxy(this.id);
                Try<ICloudProxy> cloudProxyTry = await this.cloudProxyGetter();
                ...
                this.innerCloudProxy = cloudProxyTry.Value;

Two things stand out:

1. RetryCount = 2, so a single operation retries at most once — one operation cannot account for 874
   rebuilds. The observed rate therefore implies ~874 distinct operations failing in that second, each
   contributing its own rebuild. This is a stampede across operations, not one runaway loop.
2. GetCloudProxy()'s double-checked lock prevents concurrent duplicate creation, but not a
   sequential stampede. If the freshly created proxy is immediately inactive — which is exactly the
   case while the PDN is still down — then each waiter in turn re-enters the lock, sees
   !IsActive, and creates another proxy. N pending operations produce N proxy creations back to back.

There is no Task.Delay, backoff, or circuit-breaker anywhere in this file (grep -ci 'delay|backoff|sleep|circuit' on main → 0). Nothing records "the upstream just failed" in a way that
would let the next operation skip or defer its rebuild attempt.

Expected behaviour

While the upstream is known to be down, cloud-proxy re-creation should be rate-limited or
circuit-broken per client, so that N pending operations during an outage produce O(1) rebuild attempts
with backoff rather than O(N) immediate ones. A short shared cooldown (even 1–2 s) keyed on the last
failed creation would collapse ~900 rebuilds into a handful, and would make the retained-memory cost
proportional to outage count rather than to queued-operation count.

What I have NOT verified

• Whether a mitigation exists elsewhere in the connection-establishment path
  (cloudProxyGetter / ConnectivityAwareClient / ClientProvider) that should already be damping this.
  If so, it is not damping it in practice here, and I would appreciate a pointer.
• The exact RetryingCloudProxy.cs contents at our build commit 9f72e9db — that commit is not resolvable
  in the public history, so the source above is main. The observed log lines map cleanly onto it.
• Whether 1.5.x behaves differently at runtime. I have not been able to test 1.5.x on this device yet.
• The ~6 KB/rebuild retention is measured from cgroup anon, not from a heap dump, so I cannot name the
  retained type. I can capture a dotnet-dump if that would help — the device sits at ~230 MiB against a
  450 MiB cap, so there is headroom.

Related

• EdgeHub Disconnect/Reconnect loop 830k d2c messages in 24h from one device #5201 — same symptom class (unstable link → disconnect/reconnect loop, huge d2c volume), closed as
  resolved "somewhere between EdgeAgent/EdgeHub 1.4.6 and 1.4.10". We are on 1.4.43, so this is not that
  regression — but the burst shape is similar enough that it may be a partial fix or a second path to the
  same behaviour.
• IoT Edge hub loses connections/direct methods after error while re authenticating devices #7487, EdgeHub restarts automatically, but all other modules do not restart, leading to miscommunication between the custom module and the EdgeHub module.(Interruption of data receiving after 30 days after automatic server certificate renewal) #7497 — also mention Amqp resource is disconnected around re-authentication.

Reproduction

Any uplink that drops the transport every ~60 s while messages are queued should show it. A tc/netem or
iptables rule that blackholes the AMQP endpoint for ~1 s once a minute, with a module publishing steadily,
should reproduce the rebuild stampede without needing a misbehaving cellular network.

[yophilav]

Hi @guy-radford-sunswap ,
Thanks for reporting the issue! Let me try to reproduce it and then we can internally triage. :)