Fix reconnect races that can leave producers permanently disconnected

[AmiradelBeyg]

Motivation

After a broker restart, producers/consumers with sends in flight can hit two related races in the reconnect handling, observed at scale (tens of thousands of producers on one client):

1. Duplicate reconnect scheduling. Every ChannelDisconnected event enqueues a new CloseChannel + EstablishNewChannel action. A producer can receive more than one disconnected event for the same outage — one from the connection teardown (ChannelManager.Dispose) and one from SubProducer.MessageDispatcher when an in-flight send fails. The stale second action then runs after the first reconnect has succeeded: it closes the freshly re-established channel (sending CommandCloseProducer for a healthy producer) and forces another reconnect cycle.

2. Lost reconnect decision. Process.Handle is invoked synchronously on the caller's thread, so those events race. The switch writes ChannelState, but CalculateState() reads the shared field afterwards — the two steps are not atomic. A stale Connected write can land between a Disconnected write and its CalculateState() call, making both threads read Connected: the disconnect is handled but no reconnect is ever scheduled. Since the dispatcher stops itself after reporting the failure and the channel is already deregistered, no further event arrives — the producer stays disconnected forever while State reports Connected, which is unrecoverable from the application side (no terminal state transition ever fires).

These races become much easier to observe since #292 (included in 5.3.2): before that fix, large-scale broker-restart scenarios tended to exhaust the ThreadPool before the reconnect logic could run to completion, masking the behavior described here.

Modifications

Two commits, one per race:

• Deduplicate reconnect scheduling — Process.ScheduleReconnect runs at most one reconnect action at a time and re-evaluates the channel state after the action completes, so a disconnect arriving mid-reconnect schedules a follow-up instead of a duplicate, and a replacement channel is never closed by a stale action. Applied to producer, consumer, and reader processes.

• Synchronize state transitions — a per-process lock makes the state update and the resulting decision atomic, so every event's decision is made against that event's own state and every handled disconnect schedules a reconnect. The lock is on the lifecycle-event path only (a handful of events per channel transition) and never executes on the message send path.

New ProcessReconnectTests cover initial connect, stale-action-closes-replacement-channel, disconnect-during-reconnect, and a concurrency test firing 500 concurrent connected/disconnected pairs, for all three process kinds. Handle_WhenChannelDisconnectedTwice_DoesNotCloseReplacementChannel fails against master and passes with this change.

Verifying this change

• Make sure that the change passes the CI checks.

All existing unit tests pass.