docs(chaos): chaos testing output and docs

docs/chaos-testing.md

@@ -0,0 +1,57 @@
+# Chaos Testing: Does Failover Actually Work?
+
+A load balancer's job is simple: **when a backend server dies, users should see little or no disruption.**
+
+This test verifies that GoBalancer maintains service availability by intentionally killing backend servers while real traffic continues to flow through the proxy.
+
+## The setup
+
+Everything runs in Docker (`docker-compose.yml`):
+
+- **3 backend servers** behind GoBalancer on port 8080 (one is slow on purpose).
+- A **load generator** (`test/chaos/loadgen`) sending 500 requests/second for 5 minutes.
+- A **kill loop** (`test/chaos/run.sh`) that kills one backend for 15s, restarts it, and repeats — 10 times.
+
+The load generator sends at a fixed rate no matter how slow replies are, so a stall shows up as a real error instead of being hidden.
+
+## What it checks
+
+A run passes only if all three are green:
+
+1. **Chaos registered** — the backends were really killed (~10 evictions). Stops a fake "pass" where the kills silently failed.
+2. **Fast failover** — no error window longer than 1 second.
+3. **No leak** — the balancer doesn't pile up dead connections and their background workers after each kill.
+
+## How to run it
+
+```bash
+bash test/chaos/run.sh         # ~5 min: load traffic + kill/restart backends
+go run test/chaos/verdict.go   # grades the run: PASS / FAIL
+```
+
+## The result
+
+```
+chaos registered: 10 backend evictions (expect ~10) -> PASS
+fast failover: 0 error bursts, worst = 0s (limit 1s) -> PASS
+no leak:  goroutines 14 -> 27 (slop 25) -> PASS
+outcome breakdown:
+   ok         149842
+latency: p50=0s p99=21ms max=32ms
+PASS
+```
+
+**Every one of ~150,000 requests succeeded**, across ten backend kills, with no error window and no leak. The 21 ms tail is just the one slow backend.
+
+
+## Fixes for Problems Observed in Previous Runs
+
+- **A client hanging up is no longer blamed on the backend** — that used to wrongly evict healthy backends, which is what caused the big failure windows.
+- **Requests now retry on a healthy backend** when the first one is dead, so the user never sees the failure.
+- **Dead connections are dropped the moment a backend is evicted**, instead of lingering — which keeps the worker count flat.
+
+## Output files (in `test/chaos/`)
+
+- `results.csv` — one line per request: `time,latency_ms,class` (`class` = `ok`, `http_<code>`, `timeout`, `error`).
+- `goroutines.log` — worker count, once per second.
+- `summary.txt` — `baseline=… final=… evictions=…`.

test/chaos/goroutines.log

@@ -0,0 +1,298 @@
+goroutine baseline: 14
+1787500695,14
+1787500696,24
+1787500697,24
+1787500698,24
+1787500699,24
+1787500700,24
+1787500701,24
+1787500702,25
+1787500703,25
+1787500704,25
+1787500705,24
+1787500706,27
+1787500707,27
+1787500708,27
+1787500709,28
+1787500710,30
+1787500711,30
+1787500712,28
+1787500713,29
+1787500714,28
+1787500715,28
+1787500716,29
+1787500717,29
+1787500718,28
+1787500719,28
+1787500720,29
+1787500721,28
+1787500722,28
+1787500723,31
+1787500724,29
+1787500725,31
+1787500726,29
+1787500727,29
+1787500728,29
+1787500729,30
+1787500730,29
+1787500731,30
+1787500732,31
+1787500733,31
+1787500734,30
+1787500735,30
+1787500736,30
+1787500737,31
+1787500738,30
+1787500739,33
+1787500740,29
+1787500741,29
+1787500742,30
+1787500743,28
+1787500744,28
+1787500745,28
+1787500746,30
+1787500747,28
+1787500748,28
+1787500749,28
+1787500750,28
+1787500752,29
+1787500753,28
+1787500754,28
+1787500755,28
+1787500756,29
+1787500757,31
+1787500758,28
+1787500759,31
+1787500760,31
+1787500761,30
+1787500762,31
+1787500763,31
+1787500764,31
+1787500765,30
+1787500766,31
+1787500767,30
+1787500768,32
+1787500769,31
+1787500770,30
+1787500771,23
+1787500772,23
+1787500773,23
+1787500774,23
+1787500775,27
+1787500776,27
+1787500777,27
+1787500778,27
+1787500779,27
+1787500780,25
+1787500781,25
+1787500782,25
+1787500783,25
+1787500784,25
+1787500785,27
+1787500786,26
+1787500787,25
+1787500788,28
+1787500789,30
+1787500790,31
+1787500791,31
+1787500792,30
+1787500793,31
+1787500794,32
+1787500795,31
+1787500796,29
+1787500797,30
+1787500798,29
+1787500799,28
+1787500800,29
+1787500801,29
+1787500802,29
+1787500803,29
+1787500804,28
+1787500805,30
+1787500806,28
+1787500807,29
+1787500808,28
+1787500809,27
+1787500810,28
+1787500811,30
+1787500812,29
+1787500813,28
+1787500815,29
+1787500816,28
+1787500817,29
+1787500818,29
+1787500819,28
+1787500820,27
+1787500821,30
+1787500822,31
+1787500823,30
+1787500824,30
+1787500825,30
+1787500826,34
+1787500827,31
+1787500828,32
+1787500829,33
+1787500830,33
+1787500831,34
+1787500832,30
+1787500833,30
+1787500834,29
+1787500835,29
+1787500836,29
+1787500837,31
+1787500838,29
+1787500839,29
+1787500840,32
+1787500841,30
+1787500842,29
+1787500843,30
+1787500844,30
+1787500845,31
+1787500846,28
+1787500847,30
+1787500848,30
+1787500849,29
+1787500850,31
+1787500851,32
+1787500852,31
+1787500853,32
+1787500854,30
+1787500855,30
+1787500856,30
+1787500857,29
+1787500858,31
+1787500859,30
+1787500860,30
+1787500861,29
+1787500862,24
+1787500863,27
+1787500864,24
+1787500865,25
+1787500866,23
+1787500867,25
+1787500868,23
+1787500869,24
+1787500870,24
+1787500871,24
+1787500872,24
+1787500873,30
+1787500874,27
+1787500875,27
+1787500876,27
+1787500877,26
+1787500878,26
+1787500879,26
+1787500880,29
+1787500881,29
+1787500882,30
+1787500883,29
+1787500884,29
+1787500885,29
+1787500886,30
+1787500887,31
+1787500889,27
+1787500890,26
+1787500891,27
+1787500892,26
+1787500893,30
+1787500894,30
+1787500895,28
+1787500896,30
+1787500897,29
+1787500898,29
+1787500899,28
+1787500900,28
+1787500901,28
+1787500902,28
+1787500903,30
+1787500904,28
+1787500905,28
+1787500906,28
+1787500907,28
+1787500908,28
+1787500909,28
+1787500910,30
+1787500911,30
+1787500912,32
+1787500913,32
+1787500914,33
+1787500915,32
+1787500916,34
+1787500917,31
+1787500918,32
+1787500919,31
+1787500920,31
+1787500921,32
+1787500922,30
+1787500923,27
+1787500924,26
+1787500925,28
+1787500926,28
+1787500927,27
+1787500928,27
+1787500929,26
+1787500930,30
+1787500931,28
+1787500932,28
+1787500933,27
+1787500934,27
+1787500935,28
+1787500936,29
+1787500937,29
+1787500938,32
+1787500939,33
+1787500940,33
+1787500941,35
+1787500942,34
+1787500943,34
+1787500944,35
+1787500945,34
+1787500946,33
+1787500947,35
+1787500948,34
+1787500949,34
+1787500950,34
+1787500951,34
+1787500952,35
+1787500953,25
+1787500954,25
+1787500955,28
+1787500956,25
+1787500957,25
+1787500958,29
+1787500959,27
+1787500960,27
+1787500961,28
+1787500963,28
+1787500964,28
+1787500965,27
+1787500966,27
+1787500967,28
+1787500968,28
+1787500969,27
+1787500970,27
+1787500971,27
+1787500972,30
+1787500973,30
+1787500974,29
+1787500975,28
+1787500976,30
+1787500977,27
+1787500978,28
+1787500979,30
+1787500980,31
+1787500981,31
+1787500982,30
+1787500983,30
+1787500984,30
+1787500985,31
+1787500986,29
+1787500987,28
+1787500988,28
+1787500989,28
+1787500990,29
+1787500991,28
+1787500992,28
+1787500993,29
+1787500994,29
+goroutine final: 27