high RPM

[zhejunliux]

Excessive retry loop in _http_completion causes high RPM during transient errors

Location

dacomp-da/evaluation_suite/core/llm_client.py, line 157 (function _http_completion)

Problem

The HTTP completion function uses a retry loop of 3000 iterations with 0.2s sleep, which causes two issues:

1. Excessive RPM during transient errors: When the API gateway returns a 502/503 or other non-JSON response, the code retries at ~5 requests/second (300 RPM). This can overwhelm the gateway, trigger additional rate limiting, and exacerbate the outage.

2. Ineffective for persistent failures: If the model consistently returns invalid content (e.g., truncated JSON), retrying 3000 times won't fix it — it just burns 10 minutes of wall time and thousands of wasted API calls.

Current behavior

for attempt in range(1, 3001):
    # ... request ...
    except ValueError:
        # Non-JSON response (e.g., 502 HTML page)
        time.sleep(0.2)  # 300 RPM retry rate
        continue