"Blocking found group but reserve failed" errors on op-worker container plus high CPU usage

[denisgomesfranco]

Hello. First of all, thanks for such a nice application! I've been testing OpenPanel with a few online stores that I host and it's really good.

But today I found a rather unusual problem with my setup. Suddenly, CPU usage on the server I'm running OpenPanel jumped from 6% to 62%. Since I'm working on some integrations I thought maybe I did something wrong, but with help from AI I nailed down what happened.

Somehow the op-worker container (Redis) got stuck with "Blocking found group but reserve failed: group=06da2f33f2dda361f48d404fb2066f43 (reserve took 0ms)" being printed to the log quite rapidly. I don't know much about Redis but AI helped me through it. Somehow this key group got stuck or corrupt or something (I still don't know how, though). This key group was seemingly being used by one of the sites I added to OpenPanel, so in the end I had to wipe up this key group and do some other cleanup.

It first told me to check what's up.
LLEN groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43:active
ZCARD groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43
ZRANGE groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43 0 -1 WITHSCORES
GET groupmq:group_events:lock:06da2f33f2dda361f48d404fb2066f43
ZSCORE groupmq:group_events:ready 06da2f33f2dda361f48d404fb2066f43

Then ZRANGE returned two jobs, I had to run these on them:
HGETALL groupmq:group_events:job:<JOB_ID>
EXISTS groupmq:group_events:processing:<JOB_ID>

Then I had to unblock the key group:
MULTI
ZREM groupmq:group_events:ready 06da2f33f2dda361f48d404fb2066f43
DEL groupmq:group_events:lock:06da2f33f2dda361f48d404fb2066f43
DEL groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43:active
EXEC

Then I had to remove those jobs:

• for each <JOB_ID>:
  ZREM groupmq:group_events:processing <JOB_ID>
  DEL groupmq:group_events:processing:<JOB_ID>
  DEL groupmq:group_events:job:<JOB_ID>

DEL groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43

After that, the stuck key was cleared and everything went back to normal. Before this I tried redeploying OpenPanel, restarted the container, rebooted the server, nothing helped. Clearing that stuck key is what helped. I also had to clear the log files as they grew to over 2 GB.

One thing I found odd though, is (and correct me if I'm wrong) but Redis is an in memory cache, maybe it should have cleared this by itself after a restart?

In any case, this happened to me and I'm posting it here in hopes it could help someone.