Rust I18N Rewrite - #7052
Conversation
fb73079 to
775f943
Compare
775f943 to
2fee973
Compare
|
The generated output of |
2fee973 to
bfdf2bc
Compare
|
rebasing broke because workerd-cxx got moved, woops |
bfdf2bc to
47bc5b7
Compare
Merging this PR will improve performance by 11.3%
Performance Changes
Tip Curious why this is faster? Comment Comparing 
Footnotes
|
|
Turns out the V8 patch exposing ICU publicly wasn't actually necessary |
|
Ditched the shim file. There were 3 reasons it existed:
|
Ports workerd::api::node::i18n::transcode to a new Rust crate, //src/rust/i18n, selected at runtime by a new NODEJS_I18N_RUST autogate. The C++ implementation is left in place, byte for byte, as the gate-off/rollback path.
The dispatch table, output-size computation, ICU substitute-character setup, empty-input handling, ISOLATE_LIMIT checks, and truncation all move to Rust (src/rust/i18n/dispatch.rs). The underlying codecs stay the same: a new C++ shim (src/rust/i18n/shim.{h,c++}) exposes the ICU ucnv_* primitives and the four simdutf functions the C++ path already uses, so both paths call identical codecs and cannot silently diverge.
i18n::transcode gets a gate branch at the top, following the createNodeException pattern: it maps api::node::Encoding to the bridge's Encoding enum via a fromImpl overload (kj-rs/convert.h idiom) whose switch has no default arm, rejecting BASE64/BASE64URL/HEX on either side (a deliberate divergence from the unmodified C++ dispatch, which only checks the from encoding). The Rust entry point returns a jsg::v8::ffi::MaybeLocal naming a Uint8Array; ffi-inl.h gains the missing Rust-to-C++ MaybeLocal conversion (maybe_local_from_ffi), mirroring the existing local_from_ffi.
10 Rust unit tests in dispatch.rs cover every (from, to) pair, empty input, unmappable-character substitution, the ASCII->UTF16LE latin1-widening quirk, the UTF8->UTF16LE zero-estimate quirk, invalid UTF-8, odd-length UTF-16LE input, and unpaired surrogates.
Point the src/rust/i18n shim's ICU dependency at :icuuc instead of the umbrella :icu target, per R7 (the shim only uses ucnv_* primitives from ICU's common library, not icui18n). The bazel/BUILD.icu file vendored via V8's patch set only marks :icu and :icudata public; :icuuc is package-private. Add a small new V8 patch (0039) granting :icuuc public visibility, following the existing precedent of patch 0027 doing the same for :icudata. Verified: just clippy i18n is clean; bazel test //src/rust/i18n:i18n_test, //src/workerd/api/node/tests:buffer-nodejs-test@, and @all-autogates all pass; full bazel test //... passes 1623/1623, observed running fresh (no cached results).
d55d86b to
f2f30d3
Compare
…ters c_char is unsigned on aarch64, so the explicit i8 annotations caused a type mismatch against the cxx-bridge declaration when building for ARM.
2 participants
Rewrites node I18N in Rust.
Uint8Array::from_bufferto Rust JSG. Necessary to avoid an extra copy in Rust.cxxto supportchar16_tprimitive