feat(allocation_policies): pardon rejections when the cluster is idle

[pbhandari]

Linear: EAP-692

Summary

When the cluster is idle, pardon allocation-policy rejections (can_run=False) so the query still runs. Policies stay attached; we still record what they would have done.

Idle = cluster_load < T_load and concurrent_queries < T_conc. Missing/stale loadinfo (-1) is not idle.

Off by default:

• storage_routing.enable_get_cluster_loadinfo (existing kill switch, still false)
• storage_routing.idle_cluster_load_threshold default 0
• storage_routing.idle_concurrent_queries_threshold default 0

A pardoned rejector gets can_run=True, max_threads=MAX_THRESHOLD, max_bytes_to_read=0. Caps from policies that already allowed are unchanged.

Applies to classic db_query (query’s storage set) and RPC routing (still EAP default get_cluster_loadinfo()).

Notes

• is_pardonable is currently always True (not an automator knob yet).
• Classic first-reject break is unchanged: if the first rejector is pardoned, later policies still run.
• Observability: allocation_policy.db_request_pardoned + explanation["idle_pardon"] with loadinfo.

Testing

• tests/query/allocation_policies/test_allocation_policy_base.py
• tests/web/test_db_query.py
• tests/web/rpc/v1/test_storage_routing.py
• tests/web/rpc/v1/routing_strategies/test_cluster_loadinfo.py

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ae1a2d8. Configure here.}

[cursor]

Idle pardon options missing from schema

High Severity

is_idle reads storage_routing.idle_cluster_load_threshold and storage_routing.idle_concurrent_queries_threshold, but those keys are not declared in the snuba sentry-options schema. get_option then always falls back to 0, so cluster_load < 0 and concurrent_queries < 0 never hold for real loadinfo, and the idle pardon path cannot be enabled.

Additional Locations (1)

• snuba/web/rpc/storage_routing/load_retriever.py#L42-L52

 

^{Reviewed by Cursor Bugbot for commit ae1a2d8. Configure here.}