Skip to content

perf: do not match each token in aggregation - #474

Merged
dkharms merged 3 commits into
mainfrom
0-skip-matching
Aug 5, 2026
Merged

perf: do not match each token in aggregation#474
dkharms merged 3 commits into
mainfrom
0-skip-matching

Conversation

@dkharms

@dkharms dkharms commented Jul 24, 2026

Copy link
Copy Markdown
Member

Description

When getting tokens for aggregation, we scan each token and checking it against * -- so basically, useless computations and IO (we have to load+unpack token blocks). I suggest to return range [firstTID; lastTID].

Obviously, this optimization will have more impact on high-cardinality fields that span over several token blocks (like remote_addr or something).

Query Type mean (ms) stddev (ms) p(50) (ms) p(95) (ms) p(99) (ms) iterations
base comp diff base comp diff base comp diff base comp diff base comp diff base comp diff
k8s_pod:payment-backend-eu-0 | group by (client_ip) | count
cold 304.33 287.94 -5.39% 7.48 4.93 -34.12% 303.76 287.90 -5.22% 317.10 305.11 -3.78% 317.98 312.73 -1.65% 52.00 53.00 +1.92%
k8s_pod:payment-backend-eu-0 | group by (client_ip) | count
hot 168.43 163.55 -2.90% 4.15 4.27 +3.03% 168.14 163.17 -2.96% 176.48 173.26 -1.82% 181.75 180.69 -0.58% 711.00 731.00 +2.81%
* | group by (k8s_pod) | count
cold 132.04 129.33 -2.05% 5.33 4.21 -21.03% 131.31 128.26 -2.32% 143.63 137.18 -4.49% 2637.98 142.37 -94.60% 54.00 57.00 +5.56%
* | group by (k8s_pod) | count
hot 28.05 27.68 -1.31% 1.87 1.50 -19.56% 27.65 27.42 -0.84% 33.66 32.99 -2.02% 38.01 36.31 -4.49% 4208.00 4258.00 +1.19%

  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

@dkharms dkharms added the performance Features or improvements that positively affect seq-db performance label Jul 24, 2026
@codecov-commenter

codecov-commenter commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.20%. Comparing base (7cc2790) to head (c431f4e).

Files with missing lines Patch % Lines
frac/processor/aggregator.go 90.00% 1 Missing and 1 partial ⚠️
frac/processor/eval_tree.go 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #474      +/-   ##
==========================================
+ Coverage   71.16%   71.20%   +0.04%     
==========================================
  Files         232      232              
  Lines       18466    18492      +26     
==========================================
+ Hits        13141    13168      +27     
+ Misses       4343     4340       -3     
- Partials      982      984       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Indexer-4 4ad8f1 900dcf
677543019.00 B/op 771867370.00 B/op 1.14 🔴

@eguguchkin
eguguchkin requested review from cheb0 and forshev July 27, 2026 10:56
@ozontech ozontech deleted a comment from seqbenchbot Jul 27, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 27, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 27, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 27, 2026
@ozontech ozontech deleted a comment from github-actions Bot Jul 27, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 28, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@ozontech ozontech deleted a comment from seqbenchbot Aug 5, 2026
@dkharms

dkharms commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot start search-aggregation 0-skip-matching count client_ip --query 'k8s_pod:payment-backend-eu-0' --duration 5m

@seqbenchbot

seqbenchbot commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

Started benchmark 7259172c.

Show details
  • scenario search-aggregation;
  • against 0-skip-matching;
  • running for 5m0s;

I'll post the statistics here once it finishes.
To end it early, run @seqbenchbot stop 7259172c.

Here is a list of helpful links:

  • Take a look at Grafana dashboard;
  • Live-tailing logs are also available;

Have a great time!

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
AggWide/size=1000000-4 7cc279 39d4db
529.00 B/op 866.00 B/op 1.64 🔴

@seqbenchbot

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

Benchmark 7259172c (scenario search-aggregation) has finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary
Query Type mean (ms) stddev (ms) p(50) (ms) p(95) (ms) p(99) (ms) iterations total
group by (client_ip) | count
hot 7195.14 1086.77 6964.03 9198.98 10160.19 205 47176

Compare it against another run with @seqbenchbot compare 7259172c <other-id>.

@dkharms

dkharms commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot start search-aggregation main count client_ip --query 'k8s_pod:payment-backend-eu-0' --duration 5m

@seqbenchbot

seqbenchbot commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

Started benchmark 30cb03cb.

Show details
  • scenario search-aggregation;
  • against main;
  • running for 5m0s;

I'll post the statistics here once it finishes.
To end it early, run @seqbenchbot stop 30cb03cb.

Here is a list of helpful links:

  • Take a look at Grafana dashboard;
  • Live-tailing logs are also available;

Have a great time!

@dkharms
dkharms merged commit bea6b56 into main Aug 5, 2026
8 checks passed
@dkharms
dkharms deleted the 0-skip-matching branch August 5, 2026 18:13
@seqbenchbot

Copy link
Copy Markdown
Collaborator

Oh-oh, @dkharms ('>o.o)'>!

Something went wrong and I couldn't process your request.
Please take a closer look at error message:

benchmark `30cb03cb` finished without a readable report: report is empty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Features or improvements that positively affect seq-db performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants