[lesnik512]

Adds just test-dr, an opt-in suite that breaks brokers of a 3-node Redpanda cluster while messages are in flight and checks that at-least-once delivery holds.

What it does

tests_dr/compose.yml starts three Redpanda nodes (fca-dr-0..2, external listeners on localhost 19092, 29092 and 39092). Each scenario creates a topic with replication factor 3 and min.insync.replicas=2, produces with acks=all, and injects a fault through the docker CLI once the consumer is processing. It then restores the node(s), produces more, and waits for convergence.

Scenario	Fault	Restored after
kill_non_coordinator	docker kill one broker that is not the group coordinator	20 s
kill_coordinator	docker kill the group coordinator	20 s
pause_coordinator	docker pause the group coordinator	30 s
kill_all_brokers	docker kill all three	30 s
shutdown_during_full_outage	kill all three, then stop the app, heal, start a new app	n/a

Every scenario asserts zero missing messages and zero commit lag at the end. The four fault scenarios also assert that is_kafka_handler_healthy never reads False (sampled every 250 ms) and that commit_stall_sec is back to 0 once commits converge. kill_all_brokers additionally asserts that commit_stall_sec goes above 0 during the outage, which covers #113 end to end. Each scenario writes a JSON report to tests_dr/results/ (gitignored) with timings, stall numbers and a count of WARNING+ log lines.

The app is wired in the README lifespan order: FastStream(broker), then initialize_concurrent_processing, then broker.start().

The shutdown scenario catches CancelledError from broker.stop(). aiokafka 0.14.0's Fetcher.close() sometimes raises it when no broker is reachable; this library has nothing to do with it. The workaround is tracked in #116.

Why it stays out of just test and CI

tests_dr/ sits outside testpaths, so just test and test-ci do not collect it (still 258 tests). It is also in norecursedirs, because the floors job runs pytest . and an explicit path skips testpaths. pytest tests_dr still collects it. It has no __init__.py, so --cov=. does not count it against the 100% coverage floor; that is why INP001 is ignored for it.

CI does not run it. The suite takes almost four minutes, needs three Redpanda containers and docker CLI access to kill and pause them, and depends on real timeouts and leader elections. On shared runners that is a likely source of flaky reds and slows every PR. It is meant to be run by hand after changing commit, rebalance or shutdown code; AGENTS.md says so.

just test-dr runs on the host (it has to reach the docker CLI), passes -o addopts= -p no:randomly so the scenarios run serially and in order, and always tears the cluster down through a shell trap, including when a test fails.

Checklist

• just test-dr: 5 passed in 205 s locally (macOS, OrbStack), rebased on Log a non-retriable commit error once at ERROR and keep retrying it #115
• just test: 258 passed, nothing from tests_dr collected
• just lint-ci clean (ruff, ty, eof-fixer)

Local run numbers:

Scenario	Acked	Missing	Duplicates	Unhealthy samples	Max stall during fault	Final stall	Fully committed (s after fault)
kill_non_coordinator	4000	0	0	0	0.0 s	0	24.7
kill_coordinator	4000	0	0	0	0.0 s	0	24.6
pause_coordinator	3509	0	0	0	24.8 s	0	36.6
kill_all_brokers	3000	0	0	0	24.9 s	0	36.5
shutdown_during_full_outage	2000	0	1586	n/a	n/a	n/a	43.2

In pause_coordinator, 491 produce calls timed out while the coordinator was paused. The suite counts only acked messages, so they do not show up as missing. The 1586 duplicates in the shutdown scenario are expected: offsets processed during the outage could not be committed, so the second app redelivers them.