feat Add grace period before force-reconnect while awaiting edge conf…

[arminfelder]

Pull Request description

handle reconnects gracefully, by adding a a grace period of 3x the reconnect timeout, to give it time to process a huge queues after connecting.

General checklist

• You have reviewed the guidelines document.
• Labels that classify your pull request have been added.
• The milestone is specified and corresponds to fix version.
• Description references specific issue.
• Description contains human-readable scope of changes.
• Description contains brief notes about what needs to be added to the documentation.
• No merge conflicts, commented blocks of code, code formatting issues.
• Changes are backward compatible or upgrade script is provided.
• Similar PR is opened for PE version to simplify merge. Crosslinks between PRs added. Required for internal contributors only.

Front-End feature checklist

• Screenshots with affected component(s) are added. The best option is to provide 2 screens: before and after changes;
• If you change the widget or other API, ensure it is backward-compatible or upgrade script is present.
• Ensure new API is documented here

Back-End feature checklist

• Added corresponding unit and/or integration test(s). Provide written explanation in the PR description if you have failed to add tests.
• If new dependency was added: the dependency tree is checked for conflicts.
• If new service was added: the service is marked with corresponding @TbCoreComponent, @TbRuleEngineComponent, @TbTransportComponent, etc.
• If new REST API was added: the RestClient.java was updated, issue for Python REST client is created.
• If new yml property was added: make sure a description is added (above or near the property).

[MazurenkoNick]

Thanks for the patch and the clear write-up, @arminfelder.

The failure mode is real and I confirmed the chain: the reconnect loop ticks every
cloud.reconnect_timeout (3000 ms) and blindly tears down the connection, while server-side
EdgeGrpcSession#processConnect calls sessionOpenListener.accept(...) synchronously before writing the
response — reaching KafkaEdgeGrpcSession#destroy(), which blocks up to 5s in
awaitConsumerTermination(). 5s of handshake against a 3s tick never converges.

We're fixing this in #230, so I'm closing this one. The reason for going that way rather than merging
here: reconnect_timeout * 3 gives a flat 9s window on every attempt, so a server that needs longer still
thrashes, just on a 12s period. #230 replaces the fixed-rate loop with exponential backoff plus jitter,
so the window grows until it converges regardless of how slow the server is, and a fleet reconnecting
after an outage doesn't retry in lockstep.

Appreciate you tracking this down; it made confirming the chain much faster.