Queue WLM requests denied by a throttle limit instead of rejecting

[dzane17]

Summary

Adds request queueing on top of the shipped WLM throttling. When a search exceeds its workload group's throttle limit, instead of an immediate HTTP 429 the coordinator parks it in a bounded per-group queue and admits it once a permit frees. Scope: throttle-limit denials only (resource-limit rejection is unchanged).

Base branch: 3.7-wlm-throttling.

How it works

• Config — a new queue object on the workload group, sibling of throttling, with a single setting size (default 0 = queueing disabled; over-limit requests are rejected immediately as before). Wire-gated at V_3_7_0.
• Coordinator-side parking — a denied request's ActionListener is held (no thread consumed, only the listener + open connection). Admitted later by one of three drains:
  • node tier — a completing request's permit close() drains the freed bucket (immediate, local);
  • cluster tier (owner-push) — the bucket's ring owner pushes a reserved-lease grant to a waiting coordinator over a new internal transport action; round-robin across coordinators for fairness;
  • backstop sweep — reaps cancelled entries and re-attempts node-tier admission for a bucket the completion drain missed.
• No queue timeout — a parked request has no wall-clock deadline. Legitimate queue wait is unbounded (it grows with backlog ÷ throughput), so any fixed cap would eventually cancel healthy, still-connected requests. A client bounds its own wait with cancel_after_time_interval (per-request or the search.cancel_after_time_interval cluster setting) or by disconnecting — either cancels the task, which evicts the parked entry promptly. The sweep only reaps already-cancelled tasks as defense-in-depth.
• Stats — GET _wlm/stats gains total_queued, total_queue_rejections, queued_current, queue_peak, and queue-wait aggregates (queue_wait_count, total_queue_wait_millis, max_queue_wait_millis).

Testing

• Unit — queue container (FIFO per bucket, shared depth/peak, dynamic size, cancelled-only eviction), queue service (park/drain/cancel, recursion-safe async admit), settings validation, owner-push drain + fairness, stats wire round-trip.
• Integration — WlmQueueingIT: park-then-admit and queue-full → 429 end-to-end on a real cluster.
• Manual (3-node live cluster) — verified across ~22 edge scenarios plus a load test: queue-full backpressure, cancellation (client disconnect / cancel_after_time_interval / _tasks/_cancel) evicting parked requests promptly, cross-node owner-push draining with no stranding, round-robin fairness, mass-cancel recursion safety, queue.timeout rejected as an unknown setting, and throughput/latency behavior under bursts up to 3000 concurrent. ./gradlew :server:precommit and :plugins:workload-management:precommit pass.

Notes

• No client-facing REST changes beyond the queue.size field on the workload group API.
• queue.timeout is intentionally not a setting — it is rejected as an unknown key so a stale config surfaces clearly.