Engine: Single-pass optimize_tokens

[joelhawksley]

Summary

Collapses the two-pass whitespace/text optimization in Herb::Engine::Compiler#optimize_tokens into a single pass over the raw token stream.

Previously optimize_tokens did two passes:

1. compact_whitespace_tokens built a whole intermediate array (tokens.map.with_index { ... }.compact), dropping/relabeling whitespace tokens.
2. optimize_tokens re-scanned that array to merge consecutive text tokens, accumulating with current_text += value.

This PR folds both into one pass: whitespace tokens are resolved against their neighbours in the original stream (dropped, or turned into text) and consecutive text is merged inline. compact_whitespace_tokens is removed.

optimize_tokens runs once per template compile, so this is on the hot path for any large ActionView/ReActionView precompile.

Implementation notes

• No intermediate array: the map.with_index + compact allocation is gone; the surviving whitespace-vs-neighbour checks (adjacent_whitespace?, whitespace_before_code_sequence?) run against the original tokens/index, exactly as before.
• Text is accumulated into a single buffer that is mutated with current_text << value instead of current_text += value, which reallocated and copied the whole accumulated string on every text token.
• The buffer is seeded with value.dup so a (possibly frozen) token value is never mutated in place.

Benchmark

Measured end-to-end through ViewPrecompiler.precompile over the github/github monolith template corpus (6,302 templates), ReActionView ON (Herb path), Ruby 4.0.5.

Both sides were run against the same Herb main native extension — only lib/herb/engine/compiler.rb differs between them — so the delta isolates this change. 2 boots × 5 timed iterations each.

Config	Time (median)	Time (min)	Allocated objects (median)
main baseline	19.618s / 19.396s	19.050s / 19.107s	~32,044,000
this PR	18.967s / 19.175s	18.838s / 18.823s	~30,988,000
delta	~−1 to −3%	~−1.2%	−1,056,000 (−3.3%)

The stable, reproducible win is ~1.06M fewer allocated objects per full precompile (−3.3%). Wall-clock is consistently favorable but smaller and noisier (~1–3%).

Benchmark run by Claude Opus 4.8 (GitHub Copilot), acting on behalf of @joelhawksley.

Correctness

Output is behavior-preserving — the full Herb engine test suite passes unchanged (1,139 runs, 3,811 assertions, 0 failures, 0 errors), RuboCop is clean, and sig/herb/engine/compiler.rbs is regenerated via rbs-inline.

---

Split out of #1872.

[marcoroth]

Nice, small change, big win!

Thank you @joelhawksley! 🙏🏼