feat: Add lag/idle plot simulator for cost-based autoscaler by kfaraz · Pull Request #19687 · apache/druid

kfaraz · 2026-07-15T08:50:35Z

Description

Tuning the lag-based auto-scaler can sometimes be a little involved.

This patch aims to allow making the process simpler by simulating the optimal task count chosen by the auto-scaler under different conditions.

Changes currently in this PR

UI code is generated by Claude and may contain mistakes
Add a simulate API currently supported for Kafka supervisors only
This API creates a CostBasedAutoScaler in "simulate" mode and generates the optimal task count for various input values of lag (based on criticalLag)
Add a UI panel in the supervisor dialog which shows up only for "kafka" supervisors
Add a single plot between task count vs lag

Web-console screenshot

Other required changes

Use a POST API instead of GET so that the current autoscalerConfig may be sent as payload
Wire up the UI side so that changes made in the simulate window are reflected in the autoscalerConfig and vice versa
Fix up the taskIdleRatio field to accept decimal numbers, currently the field resets to integers
Maybe more plots that help the user make a more informed decision
Perhaps a plot with the task idleness as well

This PR has:

…scaler

kfaraz · 2026-07-15T08:53:59Z

  }

+  @GET
+  @Path("/{id}/autoscaler")


We will rename this API based on what we decide the final result set to look like:

Suggested change

@Path("/{id}/autoscaler")

@Path("/{id}/autoscaler/simulate")

kfaraz · 2026-07-15T08:54:33Z

+      @QueryParam("taskCountMin") int taskCountMin,
+      @QueryParam("taskCountMax") int taskCountMax,
+      @QueryParam("maxProcessingRatePerTask") int maxProcessingRatePerTask,
+      @QueryParam("optimalTaskIdleRatio") double optimalTaskIdleRatio,
+      @QueryParam("criticalLag") int criticalLag,


Instead of multiple query params, we should just be able to send a POST payload object as a CostBasedAutoScalerConfig.

cc: @Fly-Style

Done in 9e550b2

+      @QueryParam("maxProcessingRatePerTask") int maxProcessingRatePerTask,
+      @QueryParam("criticalLag") int criticalLag,
+      @QueryParam("currentTaskCount") Integer currentTaskCount,
+      @Context HttpServletRequest request


FrankChen021

Severity	Findings
P0	0
P1	1
P2	2
P3	0
Total	3

Reviewed 11 of 11 changed files.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021 · 2026-07-24T12:19:39Z

+        requestedTaskCount,
+        ((SeekableStreamSupervisor<?, ?, ?>) supervisor).getIoConfig().getTaskCount()
+    );
+    InvalidInput.conditionalException(


[P1] Bound the simulation task counts at the API boundary

The request body can set taskCountMin to zero, which the config constructor accepts; with a positive live task count this reaches computeValidTaskCounts and divides by zero. It can also set an arbitrarily large taskCountMax, which is reused as partitionCount and makes each of the 40 samples scan linearly through that range, allowing an authorized request to monopolize the Overlord CPU. Require a positive minimum and impose a practical maximum before running the simulation.

FrankChen021 · 2026-07-24T12:19:39Z

+    );
+
+    // Assumption: enough partitions to reach taskCountMax.
+    final int partitionCount = config.getTaskCountMax();


[P2] Simulate the supervisor's actual topology

The live cost scaler uses supervisor.getPartitionCount() and the supervisor IO config's task duration, but this endpoint substitutes taskCountMax and a hard-coded hour. For example, a two-partition supervisor configured with a maximum of ten is shown recommendations that cannot run, and a custom task duration changes the lag-recovery cost curve. Read both values from the selected supervisor so the chart predicts what its scaler would actually choose.

It is under user control; let them play.

These values are not currently user-controlled: the panel/API do not expose partition count or task duration; they derive partition count from taskCountMax and fix duration at 3600 seconds. Thus a two-partition supervisor can still show impossible recommendations up to 10 tasks, and non-hour task durations produce a different curve. Please either read both from the selected supervisor, or expose them as explicit simulator inputs and label the assumptions.

Reviewed 11 of 11 changed files.

FrankChen021 · 2026-07-24T12:19:39Z

+    final int lagStepSize = criticalLag / 20;
+    final CostBasedAutoScaler autoscaleSimulator = CostBasedAutoScaler.createSimulator(config, supervisorId);
+    for (int i = 0; i < 40; ++i) {
+      final double observedAggregateLag = lagStepSize * i * 1.0;


[P2] Avoid overflowing the generated lag values

lagStepSize * i is evaluated as int before conversion to double. Large but valid criticalLag query values therefore wrap to negative lag in later samples, producing nonsensical task-count recommendations. Promote either operand before multiplying, or keep the calculation in long/double throughout.

FrankChen021

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

The task-count bounds and lag-overflow findings are addressed. A separate inline reply covers the remaining topology-assumption discussion.

Reviewed 11 of 11 changed files.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021 · 2026-07-25T14:25:10Z

+          avgProcessingRatePerTask,
+          maxProcessingRatePerTask * 1.0
+      );
+      final int optimalTaskCount = autoscaleSimulator.computeOptimalTaskCount(costMetrics);


[P2] Suppress production logs for simulation samples

computeOptimalTaskCount emits an INFO metrics line on every call and often a second candidate-table INFO line. This loop calls it 40 times per API request, while the panel issues requests after debounced edits, so ordinary simulator use floods Overlord logs with at least 40 entries per edit. Gate these logs for simulator instances as metrics already are.

Fixed in 9d7e9c1

Thanks—the panel's default payload now suppresses the original computation/candidate logs and metrics. One INFO path remains outside !isSimulation at CostBasedAutoScaler.java:380: a positive minCostDropPercentForScaling can still log for simulation samples. Please gate that log too.

Reviewed 11 of 11 changed files.

kfaraz · 2026-07-30T12:19:52Z

@Fly-Style , there is a merge conflict. Also, the patch currently seems broken. The request being sent by the web-console is a malformed JSON and throws a 400 error. Would you be able to take a look?

Fly-Style · 2026-07-31T13:45:28Z

@kfaraz 5c53c26

FrankChen021

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 11 of 11 changed files.

This is an automated review by Codex GPT-5.6-Sol

Fly-Style · 2026-08-03T10:22:27Z

@vogievetsky please review :)
Your review is decisive

kfaraz · 2026-08-03T11:13:53Z

Please don't merge this just yet, there are a few more tweaks required in the API.

FrankChen021

Severity	Findings
P0	0
P1	0
P2	2
P3	0
Total	2

Reviewed 11 of 11 changed files. Both prior critical-lag findings are resolved; this update still has read-only authorization and simulation log-volume problems.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021 · 2026-08-03T12:26:39Z

+  @Path("/{id}/autoscaler")
+  @Consumes(MediaType.APPLICATION_JSON)
+  @Produces(MediaType.APPLICATION_JSON)
+  @ResourceFilters(SupervisorResourceFilter.class)


[P2] Authorize the read-only simulation as READ

SupervisorResourceFilter derives authorization from the HTTP method, so this POST requires DATASOURCE WRITE permission. The endpoint only computes and returns a plot, while the new tab is exposed to users who can view Kafka supervisors; read-only operators therefore see the tab but every simulation request is rejected. Use a READ-specific authorization path or filter for this read-only POST.

FrankChen021 · 2026-08-03T12:26:39Z

+          avgProcessingRatePerTask,
+          maxProcessingRatePerTask * 1.0
+      );
+      final int optimalTaskCount = autoscaleSimulator.computeOptimalTaskCountInternal(costMetrics, true);


[P2] Suppress threshold logs during simulation

Passing true suppresses the main computation log and metrics, but computeOptimalTaskCountInternal still emits INFO messages for every high- or critical-lag point. With the 40-row sweep and the submitted threshold, a single request emits roughly 25 INFO lines, and the UI issues another request after each debounced input change. Guard the high/critical threshold logs with simulation mode as well.

kfaraz added 3 commits July 10, 2026 10:24

wip: temp changes

f16ee18

Revert extra changes

febfc60

Merge branch 'master' of github.com:apache/druid into visualize_auto_…

6cb49f8

…scaler

kfaraz requested a review from Fly-Style July 15, 2026 08:50

kfaraz commented Jul 15, 2026

View reviewed changes

Revert extra changes

c70f681

github-actions Bot added Area - Web Console Area - Ingestion labels Jul 15, 2026

Enhance UI and backend codebase

d763878

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread ...-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorResource.java Fixed

Fly-Style added 2 commits July 16, 2026 16:09

Use slider instead

1ab2cd3

Checkstyle

2134fce

Fly-Style requested a review from vogievetsky July 20, 2026 07:46

Fly-Style added 2 commits July 24, 2026 12:02

AI review and fixes

fab1406

Merge branch 'master' into visualize_auto_scaler

5842e43

Fly-Style marked this pull request as ready for review July 24, 2026 09:04

github-advanced-security AI found potential problems Jul 24, 2026

View reviewed changes

Comment thread ...-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorResource.java Fixed

Send full valid config instead

9e550b2

github-advanced-security AI found potential problems Jul 24, 2026

View reviewed changes

FrankChen021 reviewed Jul 24, 2026

View reviewed changes

AI-review-2

be21548

Fly-Style requested a review from FrankChen021 July 24, 2026 14:06

FrankChen021 reviewed Jul 25, 2026

View reviewed changes

Fly-Style added 2 commits July 27, 2026 13:33

Reduce logs and metrics amount for simulation purposes

9d7e9c1

Checkstyle

cb41f39

Fly-Style added 2 commits July 31, 2026 16:25

Merge branch 'master' into visualize_auto_scaler

c149873

Add autoscaler type to correctly send config via https

5c53c26

github-advanced-security AI found potential problems Jul 31, 2026

View reviewed changes

Comment thread ...java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScaler.java Dismissed

FrankChen021 reviewed Aug 1, 2026

View reviewed changes

Comment thread ...java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScaler.java Outdated

Comment thread web-console/src/dialogs/supervisor-table-action-dialog/auto-scaler-panel/auto-scaler-panel.tsx

Add critical lag

8cf345a

Fly-Style requested a review from FrankChen021 August 3, 2026 10:21

Fly-Style approved these changes Aug 3, 2026

View reviewed changes

FrankChen021 reviewed Aug 3, 2026

View reviewed changes

Uh oh!

Conversation

kfaraz commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes currently in this PR

Web-console screenshot

Other required changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz commented Jul 30, 2026

Uh oh!

Fly-Style commented Jul 31, 2026

Uh oh!

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Fly-Style commented Aug 3, 2026

Uh oh!

kfaraz commented Aug 3, 2026

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kfaraz commented Jul 15, 2026 •

edited

Loading