tuning default read load

[Haleygo]

From the resource usage in our sandbox, I think the concurrency could be lower to achieve better query load balancing for most test cases.

[makasim]

Memory is still quite spiky. Also, reducing concurrency means less parallelism on vmselect side; hence, we might miss some concurrency-related issues.

[Haleygo]

@makasim

Memory is still quite spiky.

That seems inevitable in the sandbox from what I can see, since the query rate is spiky—there is no guarantee that queries can spread evenly—and each query imposes a different load.
The point of this pull request is to reduce those spikes by lowering concurrent queries, but setting concurrency too low could lead to missed evaluations for the nodeexporter group.

Also, reducing concurrency means less parallelism on vmselect side; hence, we might miss some concurrency-related issues.

I’m not sure which concurrency-related issue you’re referring to, but I don’t think that is the point of the prometheus-benchmark repo. I also don’t think our sandbox would have such an issue, since we provide vmselect with sufficient resources and the load is stable.

This change reduces spikes that could lead to OOMs or concurrentLimitExceeded errors in smaller setups. It improves benchmark performance; you can see that both average memory usage and query latency have dropped, which aligns with my understanding of the benchmark’s purpose.

[makasim]

I’m not sure which concurrency-related issue you’re referring to

Data races mostly, nothing in particular; just a higher chance for the code under benchmark to be exercised concurrently

[makasim]

What about 2,2 ?