We hold per-user RedLock for every request. It works fine under a normal load.
But the problem starts when sometimes our server response time gets slower.
The request inbound rate remains the same for our scenario.
However the response time got longer, concurrent request count went higher.
Hence ExtendLockLifetime is called more.
https://github.com/samcook/RedLock.net/blob/master/RedLockNet.SERedis/RedLock.cs#L314
ExtendLockLifetime is called by Timer, and Timer callbacks are dispatched to ThreadPool.
ThreadPool doesn't like blocking tasks, and it is crucial to have performant ThreadPool for performant async C#.
In our case, all threads in the ThreadPool are blocked by the sudden surge of ExtendLockLifetime invocations, thus it causes ASP.NET server to almost stall.
.NET tries to spawn new threads to resolve starvation, but it cannot catch up with the rate of blocked threads with the default threading config. (and they do not recommend manually adjusting it)
https://learn.microsoft.com/en-us/dotnet/core/runtime-config/threading
Even if we change the config to spin up new threads quickly, it is not performant enough with that amount of threads due to the massive number of threads and context switches.
We hold per-user RedLock for every request. It works fine under a normal load.
But the problem starts when sometimes our server response time gets slower.
The request inbound rate remains the same for our scenario.
However the response time got longer, concurrent request count went higher.
Hence
ExtendLockLifetimeis called more.https://github.com/samcook/RedLock.net/blob/master/RedLockNet.SERedis/RedLock.cs#L314
ExtendLockLifetimeis called byTimer, and Timer callbacks are dispatched to ThreadPool.ThreadPool doesn't like blocking tasks, and it is crucial to have performant ThreadPool for performant async C#.
In our case, all threads in the ThreadPool are blocked by the sudden surge of
ExtendLockLifetimeinvocations, thus it causes ASP.NET server to almost stall..NET tries to spawn new threads to resolve starvation, but it cannot catch up with the rate of blocked threads with the default threading config. (and they do not recommend manually adjusting it)
https://learn.microsoft.com/en-us/dotnet/core/runtime-config/threading
Even if we change the config to spin up new threads quickly, it is not performant enough with that amount of threads due to the massive number of threads and context switches.