[smith558]

Summary

The tokeniser now parses regex source directly instead of doing an upfront escaped-string rewrite and slice-based parsing for character classes and {m,n} repetitions.

What Changed

• removed the up-front strToChars() pass from tokenization
• replaced slice-based class parsing with an indexed class scanner
• replaced regex-on-slice repetition parsing with an indexed {m,n} parser
• consolidated escape and number parsing helpers to keep the hot path smaller
• added light comments around the non-obvious tokeniser branches
• added a repo benchmark suite covering:
  • tokenizer
  • reconstruct
  • roundtrip
• documented benchmark usage in the README

Why

The main goal is to reduce per-parse allocations and avoid unnecessary rescans of the input string, especially on patterns with character classes and custom repetitions.

The benchmark suite is included so future tokeniser changes can be measured with representative workloads instead of one-off local scripts.

Benchmark

I compared the current branch against HEAD~2 locally using the same benchmark case set.

Tokeniser results:

• geometric mean speedup: 2.753x
• arithmetic mean speedup: 2.975x

Selected tokeniser cases:

• email-like: 4.922x
• dense-sets: 4.027x
• path-like: 3.447x
• literal: 3.147x
• class-heavy: 3.144x

Roundtrip results:

• geometric mean speedup: 1.826x

reconstruct also benchmarked faster in this harness, but tokenizer is the primary target of the change.

Testing

• npm run build
• npm test
• npm run bench -- --min-ms 200

Notes

The benchmark suite is intentionally dependency-free and runs against the built dist output:

npm run bench
npm run bench -- --suite tokenizer --min-ms 750

[smith558]

Any thoughts on this? @fent

[fent]

Hi, thanks for taking the time to contribute! And sorry for the late reply 😅

Yeah.. removing strToChars is a good change, and recently, #50 surfaced a bunch of errors that are caused by decoding the regex string early. So this will also fix #55 and #56

I see that the code coverage decreased a bit, I think a lot of it is because the util.ts file is still there, but that code is no longer used so it should be removed.

before:

=============================== Coverage summary ===============================
Statements   : 99.66% ( 296/297 )
Branches     : 99.57% ( 229/230 )
Functions    : 100% ( 27/27 )
Lines        : 99.65% ( 283/284 )
================================================================================

after:

=============================== Coverage summary ===============================
Statements   : 92.62% ( 339/366 )
Branches     : 90.24% ( 259/287 )
Functions    : 92.68% ( 38/41 )
Lines        : 92.84% ( 324/349 )
================================================================================

Make sure to merge with master, so it can be reviewed against the latest changes, and so workflows can run. Added a few comments

[fent]

Move the comment above here, or add an additional comment explaining what it does

[fent]

To make it more clear that syntaxError is not a typo of SyntaxError, name this RetSyntaxError, or RetError. Place RepeatErr (rename too?) right after it to keep it more organized

[fent]

Put this after the tests section

[danshome]

Thanks for this. I ran the current test suite against this branch's tokenizer while working on #55, and found two things that may help when you merge master:

• Modifier groups: the branch predates feat: modifier support #50, so (?i:a) and the other modifier forms currently throw "Invalid group" (90 of the modifier tests fail). That will presumably go away with the merge from master that fent asked for.
• \u escapes in group names: (?<\u0061>a) is valid JavaScript (the spec allows \u escapes in names, and Chrome, Node and Safari accept it), and master accepts it today because strToChars decodes it before parsing. readGroupName reads the raw source, so this branch rejects it with "Invalid capture group name, character ''". Decoding \u escapes in readGroupName would keep the current behaviour.

I've opened #58 (escaped characters in group syntax, #55) as a small fix in the meantime. Its (? check becomes unnecessary once this lands, but its group-name tests should carry over and would catch the \u case.