Skip to content

Ruby: Intern short token values - #2328

Closed
joelhawksley wants to merge 5 commits into
marcoroth:mainfrom
joelhawksley:joelhawksley-intern-short-token-values
Closed

Ruby: Intern short token values#2328
joelhawksley wants to merge 5 commits into
marcoroth:mainfrom
joelhawksley:joelhawksley-intern-short-token-values

Conversation

@joelhawksley

@joelhawksley joelhawksley commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

Follow-up split from #2234, as requested in the maintainer review.

Intern token values of 16 bytes or fewer with rb_enc_interned_str. These common structural values are shared frozen strings; values over 16 bytes remain freshly allocated, independent, mutable strings. This PR intentionally does not include token or AST node type interning.

Because short token values are frozen, this also:

  • makes the two compiler trim paths copy-on-write;
  • makes Token#tree_inspect change encoding on a copy rather than the token value;
  • adds a Token#value= writer (and RBS API) for visitors that replace values;
  • updates the HTML-safe assertions visitor and diagnostics rewrite example to replace token values through that writer.

Benchmarks

Re-benchmarked after merging the latest main (parse + lex over 36,989 corpus files, GC.stat[:total_allocated_objects]):

variant allocations vs baseline
no interning (pre-#2234) 147,354,286 -
main (types only) 126,321,851 -14.3%
types + short values (this PR) 110,507,528 -25.0%

Short-value interning saves an incremental 15.8M allocations beyond the types-only variant on main.

Testing

  • Regenerated templates, re-vendored Prism, and compiled the Ruby native extension.
  • Added focused coverage for the 16-byte boundary, shared/frozen short values, mutable/independent long values, mutation failure, writer replacement, and non-mutating inspection.
  • Ran focused interning, diagnostics, and HTML-safe assertions visitor tests.
  • Ran the full Ruby test suite and RBS generation check.
  • Ran RuboCop on all changed Ruby files and clang-format verification on the changed C file.
  • Resolved the conflict in ext/herb/extension_helpers.c against Ruby: Intern token and AST node type strings聽#2234 by keeping the new rb_interned_string_from_hb_string / rb_token_type_value helpers and routing rb_token_value_from_hb_string through them.

@joelhawksley
Ruby: Intern short token values
a442c7e 
Intern token values up to 16 bytes while preserving mutable, independently allocated long values. Add copy-on-write and replacement compatibility for consumers of frozen short values.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added ruby Ruby source for the gem and its libraries c C source for the core parser, lexer, and AST c-extension Ruby C extension in ext/ rbs RBS type signatures in sig/ engine Herb engine and Rails template compilation rubygem The herb RubyGem and its packaging labels Aug 20, 2026
This was referenced Aug 20, 2026
Merged
Open
@joelhawksley
Merge upstream/main into joelhawksley-intern-short-token-values
68eea2f 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
joelhawksley added a commit to joelhawksley/herb that referenced this pull request Aug 21, 2026
@joelhawksley
Lazy/flyweight Herb::Token: defer Range/Location/Position allocation
843a1eb 
Follow-on to marcoroth#2328. Herb::Token was 27% of live parse objects, with its
backing Range/Location/Position objects accounting for another 33%.
This changes rb_token_from_c_struct to allocate the Herb::Token
instance directly and set its ivars from the raw numeric range/location
components (via cached ivar IDs) instead of eagerly constructing
Range/Location/Position Ruby objects for every token. Those objects are
now materialized lazily, memoized on first access to #range/#location.

Benchmarked against marcoroth/herb-corpus (36,989 .erb files):

  Herb.lex:
    allocations: 85.98M -> 36.95M (-57%)
    wall time:    3.17s ->  2.13s (-33%)

  Herb.parse:
    allocations: 62.22M -> 42.48M (-32%)
    wall time:    4.93s ->  4.95s (~flat)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
joelhawksley and others added 2 commits August 20, 2026 19:42
@joelhawksley
Apply clang-format to extension_helpers.c
9b82f68 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@joelhawksley
Update stale test for short token value interning
fd7be5c 
The types-only test in test/interned_type_test.rb asserted that all
token values remain mutable and unshared. Short-value interning
narrows that guarantee to values over 16 bytes, so rewrite the test
to use a 17-byte tag name and drop the writer-absence assertion.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@marcoroth
marcoroth marked this pull request as draft August 21, 2026 03:31
@joelhawksley
joelhawksley marked this pull request as ready for review August 21, 2026 15:29
@joelhawksley

joelhawksley commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

I'm going to close this for now as re-benchmarking has shown that it slows compilation by about a second on a benchmark I'm working on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

c C source for the core parser, lexer, and AST c-extension Ruby C extension in ext/ engine Herb engine and Rails template compilation rbs RBS type signatures in sig/ ruby Ruby source for the gem and its libraries rubygem The herb RubyGem and its packaging

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@joelhawksley