si/concurrency

[stingram]

Summary

This PR introduces multi-chip concurrency mapping and quota management.

By automatically backing off the API client against quota limits, we can fan out auto_search distribution sweeps without bottlenecking on local TPU ports or dropping candidates to transient Vertex/AI Studio 429s.

Changes

1. Multi-Chip Server Allocation (setup.sh, server_config.py, tpu_server.py):
   • Assigns and strides sub-ports uniquely per TPU / CPU worker across concurrent scaling.
   • Bounds multi-chip routing logic specifically handling edge cases for cpu_port collisions when --chips > 1.
2. LLM Concurrency (auto_agent/config.py, hitl_agent/config.py):
   • Added a Tenacity-backed generate_content / generate_content_async monkeypatch for the google.genai SDK. This wraps all core LLM API calls with exponential backoff and jitter timeouts, allowing expansion across num_parallel_runs > 4 without getting blocked by quota violations.
3. Distribution Metric Overhaul (analyze_timing.py, run_search.py):
   • Updated time aggregations to properly calculate standard and edge timing bounds spanning across parallel evaluations instead of linear serial bounds.
   • Addressed "global time" overlaps rounding accurately to computation-hours.
4. Environment Quality of Life (prepare_maxkernel.sh):
   • Added the ruff package installation pipeline to support code formatting

[gemini-code-assist]

Code Review

This pull request introduces support for multi-chip TPU configurations by dynamically generating evaluation configurations and launching multiple TPU servers based on a new --chips argument. It also implements GenAI API rate-limit handling via monkey patching with tenacity retries, and enhances search timing analysis to report real-world wall time and concurrency acceleration. Feedback highlights several improvement opportunities: adding a missing 90-second timeout to the HITL agent's async monkey patch for consistency, restoring a removed return type annotation in server_config.py, defensively handling potential ValueError exceptions when parsing the PORT environment variable in tpu_server.py, correcting a unit mismatch in the timing summary where seconds are incorrectly labeled as computation-hours, and validating that the --chips input is a positive integer in the setup script.