feat(ourlogs): add regex search support to the EAP query syntax - #123788
Draft
JoshuaKGoldberg wants to merge 12 commits into
Draft
feat(ourlogs): add regex search support to the EAP query syntax#123788JoshuaKGoldberg wants to merge 12 commits into
JoshuaKGoldberg wants to merge 12 commits into
Conversation
Adds a `Matches` operator to the search grammar, using the same reserved-unicode marker encoding as the existing Contains/StartsWith/EndsWith wildcard operators, and resolves it to a `ComparisonFilter` with `OP_REGEXP` for EAP datasets. Unlike the wildcard operators, the pattern is carried verbatim rather than being rewritten: `SearchValue.is_regex` suppresses wildcard translation and escape sequence handling, so `\*` stays a literal asterisk and `a*b` stays a quantifier. Patterns are validated at parse time, rejecting empty patterns and the backreference and lookaround syntax that RE2 does not support. Regex is rejected on virtual column contexts and on keys backed by a filter alias, since those converters resolve values against Sentry models and would match the pattern as a literal. There is no `OP_NOT_REGEXP`, so `!=` and `NOT IN` wrap the match in a `NotFilter`. `OP_REGEXP` is referenced by its wire value until sentry-protos publishes a release containing it (getsentry/sentry-protos#420). Snuba's translation to ClickHouse `match()` is likewise not yet released (getsentry/snuba#8437), so this path is not end-to-end testable yet. Fixes LOGS-958
github-actions
Bot
added
the
Scope: Backend
…entry-backend-for-regex-log-esarching
JoshuaKGoldberg
changed the title
…uests The guard sat inside `if context_definition:`, but the timeseries branch above clears context_definition after remapping, so it never fired. The pattern was then looked up in the context's value map as a literal, surfacing as "Unknown value ^sen" instead of the intended error.
Three visitor sites repeated the same validate-then-mark block; SearchValue branched on is_regex in three places where one condition covers it.
The mypy config points django_settings_module at sentry.conf.server_mypy, so mypy cannot construct the django plugin and exits before checking anything. make test-tools runs under uv sync --only-dev in the dev env workflow, where the four tests asserting diagnostics have been failing since they were added in #123923 - that PR touched no path that triggers the workflow, so it never ran there.
… installed" This reverts commit 4ff7e94. #124057 fixed the same devenv/test-tools failure at the root: the checks now run under an isolated mypy config with the vendored stubs on MYPYPATH, so the django plugin is never constructed and sentry does not need to be installed. The skip is no longer needed, and would drop the coverage that fix restores.
…58-implement-getsentrysentry-backend-for-regex-log-esarching
Snuba lowercases the pattern along with the value when a comparison filter sets ignore_case, which rewrites classes like [A-Z] and inverts escapes like \D. Prefixing the pattern with (?i) instead keeps it intact.
ClickHouse only reports an uncompilable pattern once the query runs, and Snuba surfaces that as a 500, so \Z, atomic groups, conditionals, inline comments and named backreferences are now caught alongside backreferences and lookaround.
Exercises matching, negation, IN lists, attribute keys and case insensitivity against Snuba now that OP_REGEXP has landed there.
Member
Author
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a7d246e. Configure here.
The unsupported-syntax scan treated any backslash-digit pair as a backreference, so a pattern matching a literal backslash followed by a digit was rejected before it reached Snuba.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds regex search to the EAP query syntax, so logs can be filtered by pattern rather than by literal or wildcard. This is done with a new
Matchesoperator, using the same reserved-unicode marker encoding as the existingContains/StartsWith/EndsWithwildcard operators:Resolution produces a
ComparisonFilterwithOP_REGEXP, which Snuba compiles to a ClickHousematch(). There is noOP_NOT_REGEXP, so!=andNOT INwrap the match in aNotFilter, andIN/NOT INlists become anOrFilterover the patterns.Case insensitivity is expressed by prefixing the pattern with RE2's
(?i)rather than by settingignoreCaseon the filter: Snuba implements that flag by lowercasing the pattern along with the value, which rewrites[A-Z]and inverts escapes like\D.Patterns are validated before the query is sent. ClickHouse only reports an uncompilable pattern once the query runs, and Snuba surfaces that as a 500, so the syntax RE2 lacks — backreferences, lookaround, atomic groups, conditionals, inline comments,
\Z— is rejected up front as an invalid search.Both upstreams this needed have landed:
OP_REGEXPOP_REGEXPto ClickHousematch()Closes LOGS-958.