Defer Range/Location/Position allocation

[joelhawksley]

Summary

Follow-on to #2328. Herb::Token was 27% of live parse objects, with its backing Range/Location/Position objects accounting for another 33%. Herb.lex alone allocates ~86M objects across the herb-corpus benchmark.

This defers construction of the Range/Location/Position Ruby objects for each token until they're actually accessed via Herb::Token#range / #location, instead of eagerly building them for every token during lex/parse.

Update: rebased onto main after #2234 merged, which independently added a very similar ivar-based (rb_obj_alloc + rb_ivar_set) direct construction path for Position/Location/Range/Token, plus a track_locations parser option (skip range/location entirely) and interned/cached token type strings. This branch now builds on top of that: it keeps #2234's track_locations skip-path and type interning as-is, and adds the lazy deferral on top -- when track_locations is enabled, the raw range/location components are still copied out cheaply, but the Range/Location/Position wrapper objects are only materialized on first access to #range/#location, rather than unconditionally.

Approach

• rb_token_from_c_struct (in ext/herb/extension_helpers.c) allocates the Herb::Token instance directly (rb_obj_alloc) and, when options->track_locations is true, sets ivars from the raw numeric range/location components (token->range.from/to, token->location.start/end.line/column) using cached ivar IDs, instead of eagerly constructing Range/Location/Position objects. When track_locations is false, no range/location ivars are set at all (matching Ruby: Intern token and AST node type strings #2234's skip behavior).
• Herb::Token#range and #location (in lib/herb/token.rb) lazily build and memoize the Range/Location objects from the raw ivars on first access, and return nil when the raw components were never set (i.e. track_locations: false), rather than constructing a bogus Range.new(nil, nil).
• @type is set via rb_token_type_value (interned/cached, from Ruby: Intern token and AST node type strings #2234) rather than re-allocating a string per token.
• No changes to token_T / the C-side token struct, arena lifetime, or value handling -- those remain eagerly copied out as before. This intentionally does not attempt lazy value materialization, which would require keeping the arena alive per-token and was judged too risky for the win.

An earlier iteration routed token construction through a Ruby-level Herb::Token.from_raw class method call from C (rb_funcallv). Benchmarking showed the per-token method dispatch overhead outweighed the allocation savings for Herb.parse, which touches #range/#location on nearly every token (AST node locations, error reporting, etc). Setting ivars directly from C keeps token construction as cheap as the previous eager Token.new call while still avoiding the Range/Location/Position allocations until they're needed.

Benchmarks

Measured against marcoroth/herb-corpus (36,989 .erb files), GC.stat(:total_allocated_objects) deltas and Benchmark.realtime wall clock, after a warmup pass and GC.start (median of 3 runs). Baseline is current upstream/main (f5c0347e, includes #2234's own ivar-based construction, track_locations, and type interning):

	main (post-#2234)	this branch	delta
Herb.lex allocations	73.72M	24.70M	-66%
Herb.lex wall time	~3.05s	~1.87s	-39%
Herb.parse allocations	52.60M	33.71M	-36%
Herb.parse wall time	~4.67s	~4.21s	-10%

(Note: main's own numbers already reflect #2234's independent allocation/interning improvements over the pre-#2234 baseline used in the original benchmark table below -- this branch's deltas are measured on top of that, not in addition to it.)

Testing

• bundle exec rake test -- 2435 runs, 5602 assertions, 0 failures, 0 errors.
• bundle exec rubocop lib/herb/token.rb -- no offenses.

Opening as a draft since this touches a hot path in the C extension and I'd like feedback on the ivar-based construction approach before merging.

[joelhawksley]

@marcoroth thoughts on getting this landed? I'm seeing rough a 2x speedup in per-thread performance with this change.