Ruby: Intern token and AST node type strings

[joelhawksley]

What

Intern the type strings that the Ruby C extension attaches to every Herb::Token and AST node. Token and node type names come from closed enums, so identical names are now shared frozen strings instead of being allocated for every object.

Token types are cached by enum value, while each generated AST node builder caches its constant type. Both paths avoid repeated fstring-table probes after their first use.

Token values are unchanged: each value remains independently allocated and mutable, with the existing read-only Token#value API and RBS contract.

Why

Type interning captures most of the allocation reduction without introducing frozen-value compatibility concerns or changing token value semantics.

Benchmark

Measured over the herb-corpus (36,989 .erb files), building both sides from source and taking the median of 5 runs. Allocations are GC.stat(:total_allocated_objects) deltas.

	allocations	delta	wall clock (median)	delta
Herb.parse, main	61,372,804		6.226s
Herb.parse, this branch	52,596,949	−14.3%	6.024s	−3.2%
Herb.lex, main	85,981,356		4.516s
Herb.lex, this branch	73,724,726	−14.3%	4.034s	−10.7%

21,032,485 fewer allocations (−14.3%) across the corpus.

Lexing gains more wall clock because a token type string is a larger share of the work per token than a node type string is per node.

Testing

• Built the Ruby native extension after regenerating templates and vendoring Prism.
• Added focused coverage that token and AST node type strings are frozen and shared, while token values remain mutable and independently allocated.
• Ran the interning, diagnostics, HTML-safe assertions visitor, and engine test files.

[marcoroth]

Thanks @joelhawksley!

I was curious where the win actually comes from, so I built three variants off this branch and ran parse + lex over all 37,433 files in the corpus:

variant	allocations	vs baseline	median
no interning	147,906,914	-	11.266s
types only	126,795,736	-14.3%	10.729s
types + short values	110,954,929	-25.0%	10.523s

Interning only the type names accounts for 21.1M of the 37.0M saved allocations, so 57% of the win. The short values add the remaining 15.8M.

That makes me lean towards keeping just the type interning. All four frozen-value breakages were on token.value, never on token.type. Type names come from a closed enum and nothing outside the extension writes to them, so it needs no attr_accessor :value and no frozen-string contract to document. It also avoids token.value.frozen? depending on how long the expression is.

Would you be up for splitting it, landing the type interning and then let's see what we can do about the short values as their own PR?

Thank you! 🙏🏼

[joelhawksley]

@marcoroth done: #2328

[marcoroth]

Thank you @joelhawksley! 🙏🏼