Migrate to a full native extension wrapping protovalidate-cc

[anuraaga]

Performance matters here and popular validation libraries are largely written in native. We took one step there by replacing our CEL engine with cel-expr-python, but that left several problems. The idea of utilizing protovalidate-cc as a native core came up, but a Python project must not require Bazel to allow smooth sdist builds and to improve contributor experience. At first this seemed like a blocker, but this takes an approach that I think is viable.

Instead of running Bazel on every build, this uses it as a preprocessing step to vendor in C++ sources for us to wrap into a standard PyO3 + cargo build. This relies on certain observations

• There are only a handful of C++ libraries used here, not hundreds like something like Envoy
• These are all core business logic libraries, not I/O or syscall heavy. They don't require complex autotools scripts to setup, the C++ source just needs to be passed to the compiler
• Rust relies on -sys crates heavily to expose existing native libraries and includes robust support for building them with it's cc crate. This isn't random spawning of gcc

So this adds a script, extract_native_sources, which fetches the protovalidate-cc repo and uses Bazel to build it, which fetches in its dependency sources, and we use bazel aquery to analyse the build graph for the exact source files used. We vendor the sources and a manifest of the files list to feed to standard cargo cc machinery, to have a simple cargo build for the libraries that sidesteps Bazel completely.

This isn't a simple approach by any means but it is systematic and I believe robust - we don't have any shady regex matching, we use bazel's queries to get real info on the build. IMO it's the best way to follow our standing directive of reusing cel-cpp here. And the approach could be used to implement a protovalidate-rust in the future that still uses the cpp CEL engine - if that happened, protovalidate-python and protovalidate-cpp would both wrap that.

Result, every problem we have is eliminated

• No need for Bazel to build the wheel
• No required dependency on google.protobuf
• protovalidate-cc performance instead of pure python
• Can support all the same platforms as protobuf-py, including Python 3.10, unreleased pythons, free-threaded, alpine linux
• Other platforms can easily (but slowly) build the sdist
• No two protovalidate implementations. In fact, no one implementation either

Improvement, perhaps still a problem though much less of a deal - we only copy once between protobuf-py and protobuf-cpp, no upb copy in between like before.

First bench is from an interim state.

case	celpy	cel-expr	native	pydantic
scalar	1252.71	14.29	2.00	0.42
repeated_scalar	133.75	5.04	1.29	0.50
repeated_message	13556.52	145.90	15.83	2.21
repeated_unique_scalar	90.04	5.63	2.08	0.71
repeated_unique_bytes	87.75	6.04	2.37	0.71
map	115.42	9.52	4.38	0.54
complex_schema	36410.46	542.46	75.08	6.79
int32_gt	16453.04	206.44	45.04	1.46
bytes_matching	1032.17	29.88	4.13	1.46
string_matching	929.13	49.29	4.29	1.96
wrapper_testing	8500.08	107.17	13.83	0.71
multi_rule_error	1531.62	21.54	19.83	0.50
multi_rule_no_error	1356.17	17.96	2.25	0.42

Notice how celpy is just way too slow, so even keeping it as a fallback was somewhat questionable in whether it could actually be used. There are still quite some good wins going from the cel-expr approach to this.

This is the final one after some more optimizations especially when returning errors - didn't even record celpy since doesn't matter much.

case	cel-expr	native	pydantic	× cel-expr	nat/pyd now	was
scalar	13.42	2.08	0.46	6.4×	4.5×	4.8×
repeated_scalar	5.04	1.33	0.58	3.8×	2.3×	2.6×
repeated_message	143.54	17.67	2.46	8.1×	7.2×	6.9×
repeated_unique_scalar	5.29	2.50	0.83	2.1×	3.0×	2.8×
repeated_unique_bytes	5.63	2.58	0.75	2.2×	3.4×	3.3×
map	8.83	4.83	0.62	1.8×	7.7×	7.1×
complex_schema	518.42	84.77	7.42	6.1×	11.4×	10.7×
int32_gt ‡	201.60	34.12	1.58	5.9×	21.6×	30.8×
bytes_matching	29.04	4.71	1.67	6.2×	2.8×	2.7×
string_matching	50.42	4.71	2.17	10.7×	2.2×	2.2×
wrapper_testing	108.50	15.21	0.79	7.1×	19.2×	18.4×
multi_rule_error ‡	21.04	8.13	0.58	2.6×	13.9×	34.2×
multi_rule_no_error	17.29	2.50	0.50	6.9×	5.0×	4.5×

The gap to pydantic is much reduced. We will still need to improve further - this will involve implementing native rules in protovalidate-cc as we have done in some of the other languages. Profiling shows most time is spent in CC, not our bindings (which includes the marshaling cost of the protos to C++).

Just for context, I did some highly experimental native rule evaluation in the vendored protovalidate-cc for a good improvement.

Case	CEL (today)	Native rules (experiment)	pydantic (reference floor)
single int32.gt field	1.71 µs	0.44 µs (3.9x)	0.35 µs
16 numeric range/const/in fields	30.2 µs	5.8 µs (5.2x)	1.35 µs

Fixes #489

[anuraaga]

@ajeetdsouza This now uses git submodules instead of local copies for third party deps, and also merges into two crates, deps-sys and protovalidate-sys which includes the actual rust shim. The extraction script still exists to wire up the needed cc files to the build, and also we do still have some local vendoring for gencode that isn't present in the upstream repos (just cel-cpp, protovalidate proto gencode and generated parser). sdist went from 5MB to 18MB, still quite manageable and I see no issue with it (PyPI's limit is 100MB).

[ajeetdsouza]

Thanks for collapsing the crates + adding submodules, this looks a lot better now.

[ajeetdsouza]

Might be good to include format-cpp here too via clang-format. There's not much C++ code though, so if it's too much of a pain to install, we can leave it out.

[anuraaga]

Yeah since it's small will leave it out for now but look at it later

[ajeetdsouza]

I'm guessing we use the release profile here to avoid a separate Rust toolchain + build path in CI, correct?

[anuraaga]

Yeah, updated the comment

[ajeetdsouza]

protovalidate would be a better name, and is consistent with our Python / JavaScript libraries.

[anuraaga]

Yeah - renamed it