Repository navigation
Conversation
…tart_async() semantics Addresses python-zk#594 Addresses python-zk#727
…ure kazoo.testing package - Replace legacy ZookeeperCluster process driver with testcontainers 4.x using base+overlay compose layouts (docker-compose.base.yml). - Add auth matrix overlays for digest, sasl-digest, tls, and sasl-gssapi with certgen and alpine KDC sidecars. - Scaffold docker-compose.capture.yml feature overlay and tshark sidecar for pcapng network packet capture, splitting ensemble into netns-holder + service pairs. - Emit TLS keylog files and context certificates from tls-secrets-agent for Wireshark/pcapng TLS decryption. - Restructure kazoo.testing package into kazoo.testing.common (core logic, helpers) and kazoo.testing.fixtures (pytest plugins), retiring harness.py. - Move compose overlays, JAAS configs, and Dockerfile resources into kazoo.testing package using importlib.resources. - Migrate integration tests off KazooTestCase to zkclient/zkensemble pytest fixtures and retire legacy root compose files. - Add registered pytest markers (zk_version, zk_auth, zk_features) with collection-time evaluation and skip reasons. - Introduce ZK axis enums (backports.strenum / enum.StrEnum) and bump default ZK version to 3.9.5. - Add contract & unit tests reaching 93%+ coverage on kazoo.testing pure helpers and subprocess functions. - Optimize GitHub Actions workflows with test matrices, docker layer caching, and Windows WSL2 runner support.
Do not use the session timeout as connection timeout. This value is too large and results in a bad non-responsive server holding up the client long enough for the session to timeout. Use the KazooRetry object to use an increasing backoff timeout and cycle over all servers quickly, working around bad servers with minimal impact.
ceache
force-pushed
the
feature/connection-retry
branch
from
September 19, 2026 02:28
d24a2bd to
26311a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes python-zk#685
Why is this needed?
Previously,
ConnectionHandler._connect()set the TCP socket connect timeout toclient._session_timeout / 1000.0(which defaults to 10 seconds, but can be up to 40 seconds).If any member of the ZooKeeper ensemble was unresponsive or dropped packets without sending a TCP RST (e.g. host crash, blackholed IP, firewall drop, network partition), the client would block for the entire session timeout on that single host. In a multi-node ensemble, cycling through unreachable hosts could take 10+ seconds, causing active ZooKeeper sessions on the remaining healthy ensemble members to prematurely expire before reconnection could succeed.
Proposed Changes
In
kazoo.protocol.connection.ConnectionHandler._connect_attempt(), passtimeout=retry.cur_delaytoself._connect()instead ofclient._session_timeout / 1000.0.cur_delayProperty inKazooRetry:Added typed
@property def cur_delay(self) -> floatreturningself._cur_delay.Initial connection attempts cycle through all ensemble members quickly using the initial retry delay (default
0.1s), escalating with exponential backoff on subsequent passes (0.2s,0.4s,0.8s... up tomax_delay).Once the TCP socket is connected,
self._invoke(client._session_timeout / 1000.0 / len(client.hosts), connect)remains unchanged, ensuring the ZooKeeper server has adequate quorum negotiation time to process and commit session registration.retry.reset()on successful connection resets_cur_delayto initial delay (0.1s), ensuring subsequent reconnections begin with fast failover cycles.Added
kazoo/tests/unit/test_cnx_retry.pyto verify thatcreate_connectionis called with the initial timeout across all hosts on the first pass, and escalates tomax_delayacross subsequent retry passes.Rebased cleanly on top of
feature/docker-compose-tests(8cb597a), adding modern typing (timeout: float) and SNI parameters (hostname,check_hostname).Does this PR introduce any breaking change?
No breaking API changes. The connection retry logic is internal and more responsive to unreachable ensemble members.