Skip to content

Add optional PCRE2 runtime support for bundled NetEase rules - #26

Closed
jacklilyhello wants to merge 1 commit into
mainfrom
codex/pcre2-2026-06-23-07-54-16
Closed

jacklilyhello wants to merge 1 commit into
mainfrom
codex/pcre2-2026-06-23-07-54-16

Conversation

@jacklilyhello

Copy link
Copy Markdown
Owner

Motivation

  • Enable operators to opt in to a more-expressive PCRE2 regex backend so NetEase rules that are incompatible with Go RE2 can be activated when explicitly requested.
  • Keep default behavior unchanged: RE2 remains the default engine, default builds remain CGO-free, and NetEase bundled data remains default-disabled with no runtime downloads.
  • Provide safe, auditable activation with explicit config, clear unsupported errors for non-PCRE2 builds, preserved provenance, and richer compatibility statistics.

Description

  • Add a small regex backend abstraction and RE2/PCRE2 implementations in internal/matcher (regex_backend.go, updated regex.go, pcre2.go, pcre2_stub.go, PCRE2-tagged tests), plus compile-time selection CompileRegexRulesWithEngine.
  • Extend bundled NetEase runtime loading in internal/bundled/runtime.go to select the effective regex_engine, evaluate PCRE2 compatibility when available, and surface new stats (selected engine, total examined, RE2/PCRE2 compatible/incompatible, backend-unavailable skipped, configuration-disabled, activated counts).
  • Wire config support bundled_rules.netease.regex_engine with env override OPENAUDIT_BUNDLED_RULES_NETEASE_REGEX_ENGINE, preserve legacy mode as an alias, and validate allowed values; updates in internal/config/* and config.example.yml.
  • Make PCRE2 optional via a pcre2 build tag that adds a direct cgo binding to libpcre2-8 (PCRE2 8-bit API) requiring CGO_ENABLED=1 and system libpcre2-8 development files; default builds include a stub that returns a clear unsupported error; documentation and Makefile targets for test-pcre2/build-pcre2 added; GPL/provenance and reload atomicity preserved.

Modified notable files (high level): internal/matcher/*, internal/bundled/runtime.go, internal/engine/engine.go, internal/config/*, config.example.yml, Makefile, README.md, docs/bundled-rules-phase-b-runtime.md, CHANGELOG.md, CODEX.md.

PCRE2 dependency and license note: the implementation uses direct cgo to the PCRE2 8-bit API via pkg-config: libpcre2-8 (requires system libpcre2-dev); PCRE2 upstream license is documented and no third-party Go binding was added.

Safety and reload semantics: patterns are compiled once during prepare/reload, PCRE2 match/depth limits are set in the match context, compile/match errors are deterministic and sanitized, and failed reloads or unsupported PCRE2 requests preserve previous active engine state and stats.

Testing

  • Ran unit and package tests (default builds): go test ./internal/bundled/... ./internal/engine/... ./internal/matcher/... ./internal/rules/... ./internal/config/... — PASS; go test ./... — PASS.
  • Ran static/build checks: go vet ./... && go build ./... && CGO_ENABLED=0 go build ./... && make fmt-check && make verify-bundled-netease — PASS.
  • Concurrency/race/smoke/e2e: go test -race ./internal/bundled/... ./internal/engine/... ./internal/matcher/... && make smoke && make e2e — PASS.
  • PCRE2-tagged builds/tests: go test -tags pcre2 ./... and go build -tags pcre2 ./... / make test-pcre2 / make build-pcre2 could not complete in this environment because pkg-config could not find libpcre2-8 (system dependency missing); PCRE2 code and tag are present and tests are added but were skipped/failed for lack of libpcre2 in CI environment.
  • Security tooling: attempted make gosec started installing gosec and began dependency fetch; installation was interrupted after tool install exceeded time budget in this environment and govulncheck was not run here for the same reason.

Branch and PR

Confirmations

  • Default RE2 behavior unchanged: yes.
  • CGO_ENABLED=0 go build ./... passes: yes.
  • PCRE2 support implemented behind -tags pcre2 and CGO_ENABLED=1: implemented, but PCRE2-tagged tests/builds were skipped in this environment due to missing libpcre2-8 (pkg-config).

If you want, I can: open an optional CI job guidance for running -tags pcre2 with libpcre2-dev installed, or refine the PCRE2 cgo wrapper (timeouts, error messages) further before merge.


Codex Task

@jacklilyhello

Copy link
Copy Markdown
Owner Author

This is a duplicate PR. The title and content are the same, and #26 even references #25 in its description. #26 should be closed, and #25 should be fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant