perf: optimize `re` filter by dkharms · Pull Request #484 · ozontech/seq-db

dkharms · 2026-08-03T18:46:08Z

Description

Previously the re filter compiled the user's regular expression and ran a full regexp.Match against every token in the index.

This change parses the regex AST (via regexp/syntax) at query-parse time and extracts the constant literal fragments it must contain: a prefix, a suffix, and any literal substrings in the middle.

On top of literal pre-filtering, when the token list is ordered we use the extracted prefix to narrow the search range with a binary search, so we only scan the contiguous slice of tokens that can possibly match the prefix instead of the whole index.

In overall I see a huge improvement in search latencies over different queries (3x-150x).

Take a look at results:

Show results

Query	Type	`mean (ms)`			`stddev (ms)`			`p(50) (ms)`			`p(95) (ms)`			`p(99) (ms)`			`iterations`
Query	Type	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff
`transaction_id:re("tx-cafe.*")`	cold	`365.25`	`113.50`	`-68.93%`	`19.29`	`8.19`	`-57.58%`	`369.00`	`111.00`	`-69.92%`	`377.50`	`120.00`	`-68.21%`	`377.50`	`120.00`	`-68.21%`	`4.00`	`4.00`	`0.00%`
`transaction_id:re("tx-cafe.*")`	hot	`175.85`	`1.20`	`-99.32%`	`48.88`	`0.52`	`-98.93%`	`156.00`	`1.00`	`-99.36%`	`255.00`	`2.00`	`-99.22%`	`298.50`	`2.50`	`-99.16%`	`20.00`	`20.00`	`0.00%`
`transaction_id:re(".needle.")`	cold	`1307.75`	`246.50`	`-81.15%`	`121.26`	`31.08`	`-74.37%`	`1240.00`	`244.00`	`-80.32%`	`1394.50`	`266.50`	`-80.89%`	`1394.50`	`266.50`	`-80.89%`	`4.00`	`4.00`	`0.00%`
`transaction_id:re(".needle.")`	hot	`911.75`	`2.00`	`-99.78%`	`25.92`	`0.00`	`-100.00%`	`904.00`	`2.00`	`-99.78%`	`956.00`	`2.00`	`-99.79%`	`957.50`	`2.00`	`-99.79%`	`20.00`	`20.00`	`0.00%`
`transaction_id:re(".l.")`	cold	`1233.50`	`218.50`	`-82.29%`	`16.78`	`11.96`	`-28.75%`	`1227.00`	`216.00`	`-82.40%`	`1247.00`	`226.50`	`-81.84%`	`1247.00`	`226.50`	`-81.84%`	`4.00`	`4.00`	`0.00%`
`transaction_id:re(".l.")`	hot	`919.90`	`2.05`	`-99.78%`	`34.27`	`0.39`	`-98.85%`	`913.00`	`2.00`	`-99.78%`	`995.00`	`3.00`	`-99.70%`	`998.50`	`3.00`	`-99.70%`	`20.00`	`20.00`	`0.00%`
`transaction_id:re(".*beef")`	cold	`1227.50`	`421.25`	`-65.68%`	`17.99`	`101.20`	`+462.50%`	`1225.00`	`370.00`	`-69.80%`	`1241.00`	`473.50`	`-61.85%`	`1241.00`	`473.50`	`-61.85%`	`4.00`	`4.00`	`0.00%`
`transaction_id:re(".*beef")`	hot	`995.75`	`34.45`	`-96.54%`	`116.47`	`6.30`	`-94.59%`	`964.00`	`31.00`	`-96.78%`	`1229.00`	`44.00`	`-96.42%`	`1311.00`	`47.00`	`-96.41%`	`20.00`	`20.00`	`0.00%`

Part of #355 (There is some unfinished business still, we will get back to it later)

I have read and followed all requirements in CONTRIBUTING.md;
I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

codecov-commenter · 2026-08-03T18:50:52Z

Codecov Report

❌ Patch coverage is 88.88889% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.31%. Comparing base (bea6b56) to head (ecad208).

Files with missing lines	Patch %	Lines
parser/token_re.go	82.05%	10 Missing and 4 partials ⚠️
pattern/pattern.go	96.82%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #484      +/-   ##
==========================================
+ Coverage   71.19%   71.31%   +0.12%     
==========================================
  Files         232      232              
  Lines       18492    18622     +130     
==========================================
+ Hits        13165    13281     +116     
- Misses       4344     4354      +10     
- Partials      983      987       +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dkharms · 2026-08-04T16:09:44Z

@seqbenchbot start search-regular main 'transaction_id:re("tx-cafe.*")' --cold --duration=5m

seqbenchbot · 2026-08-04T16:09:47Z

Nice, @dkharms <(-^,^-)=b!

Started benchmark 7f858c31.

Show details

scenario search-regular;
against main;
running for 5m0s;

I'll post the statistics here once it finishes.
To end it early, run @seqbenchbot stop 7f858c31.

Here is a list of helpful links:

Take a look at Grafana dashboard;
Live-tailing logs are also available;

Have a great time!

seqbenchbot · 2026-08-04T16:18:04Z

Nice, @dkharms <(-^,^-)=b!

Benchmark 7f858c31 (scenario search-regular) has finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary

Query	Type	`mean (ms)`	`stddev (ms)`	`p(50) (ms)`	`p(95) (ms)`	`p(99) (ms)`	`iterations`	`total`
`transaction_id:re("tx-cafe.*")`	cold	`426.27`	`38.30`	`423.46`	`498.91`	`527.17`	`95`	`104`

Compare it against another run with @seqbenchbot compare 7f858c31 <other-id>.

dkharms · 2026-08-04T16:18:34Z

@seqbenchbot start search-regular 355-re-optimization 'transaction_id:re("tx-cafe.*")' --cold --duration=5m

seqbenchbot · 2026-08-04T16:18:37Z

Nice, @dkharms <(-^,^-)=b!

Started benchmark 6f695674.

Show details

scenario search-regular;
against 355-re-optimization;
running for 5m0s;

I'll post the statistics here once it finishes.
To end it early, run @seqbenchbot stop 6f695674.

Here is a list of helpful links:

Take a look at Grafana dashboard;
Live-tailing logs are also available;

Have a great time!

seqbenchbot · 2026-08-04T16:27:19Z

Nice, @dkharms <(-^,^-)=b!

Benchmark 6f695674 (scenario search-regular) has finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary

Query	Type	`mean (ms)`	`stddev (ms)`	`p(50) (ms)`	`p(95) (ms)`	`p(99) (ms)`	`iterations`	`total`
`transaction_id:re("tx-cafe.*")`	cold	`155.24`	`21.22`	`154.23`	`203.95`	`232.27`	`106`	`104`

Compare it against another run with @seqbenchbot compare 6f695674 <other-id>.

dkharms · 2026-08-05T10:37:17Z

+	// NOTE(dkharms): We do not allow overriding of case-sensitivity
+	// in case if search is case-insensitive.
+	//
+	// This way `re` filter works consistently with how `keyword`
+	// and `text` search behave.
+	overridable := config.CaseSensitive
+	if !overridable && hasCaseSensitivityOverride(re) {
+		return nil, fmt.Errorf(
+			"store is configured for case-insensitive search: " +
+				"you cannot override this option",
+		)
 	}

-	lex.Next()
+	// NOTE(dkharms): Again, if store works in case-insensitive mode
+	// we simulate behaviour of `keyword` and `text` indexes.
+	// Should we really do this?
+	if !overridable {
+		expr = strings.ToLower(expr)
+	}


This decision is debatable.

dkharms added the performance Features or improvements that positively affect seq-db performance label Aug 3, 2026

ozontech deleted a comment from seqbenchbot Aug 4, 2026

dkharms marked this pull request as ready for review August 5, 2026 10:36

dkharms commented Aug 5, 2026

View reviewed changes

dkharms added 2 commits August 5, 2026 21:14

perf: optimize re filter

0b75dd8

fix: fixed pre-match check and suffix check

ecad208

dkharms force-pushed the 355-re-optimization branch from 7b43684 to ecad208 Compare August 5, 2026 18:14

Navigation Menu

Uh oh!

perf: optimize `re` filter - #484

perf: optimize `re` filter#484
dkharms wants to merge 2 commits into
mainfrom
355-re-optimization

dkharms commented Aug 3, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 3, 2026 •

edited

Loading

Uh oh!

dkharms commented Aug 4, 2026

Uh oh!

seqbenchbot commented Aug 4, 2026 •

edited

Loading

Uh oh!

seqbenchbot commented Aug 4, 2026

Uh oh!

dkharms commented Aug 4, 2026

Uh oh!

seqbenchbot commented Aug 4, 2026 •

edited

Loading

Uh oh!

seqbenchbot commented Aug 4, 2026

Uh oh!

dkharms Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dkharms commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

codecov-commenter commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dkharms commented Aug 4, 2026

Uh oh!

seqbenchbot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seqbenchbot commented Aug 4, 2026

Uh oh!

dkharms commented Aug 4, 2026

Uh oh!

seqbenchbot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seqbenchbot commented Aug 4, 2026

Uh oh!

dkharms Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dkharms commented Aug 3, 2026 •

edited

Loading

codecov-commenter commented Aug 3, 2026 •

edited

Loading

seqbenchbot commented Aug 4, 2026 •

edited

Loading

seqbenchbot commented Aug 4, 2026 •

edited

Loading