Repository navigation
fix: reject escaped characters in group syntax - #58
Merged
Merged
Conversation
The tokenizer decodes escapes such as \x3D and \x21 before parsing, so escaped characters were read as group syntax. (?\x3Da) was tokenized as the lookahead (?=a), and (?\x21a), (?\x3Aa) and (?\x3Cn>a) as (?!a), (?:a) and (?<n>a). In group names, (?<n\x3Ea) closed the name at the escaped >, and (?<\x61>a) became a group named a. JavaScript rejects all of these, and reconstruct turned them into different, valid regular expressions. The character after (? must now be literal in the source, as the group modifier parser already requires, and is reported the same way: "Invalid group" at its column in the original source. A \ there is never group syntax either, which also covers escapes that decode to an escaped character, such as (?\x28a). The literal check is now a single helper, which the modifier parser uses too. A group name may still use \u escapes, which the specification allows ((?<\u0061>a) is the group a), but no other escape. The closing > may be literal or \u003E: the specification requires a literal >, but V8 and JavaScriptCore accept the escaped form, as does SpiderMonkey, which uses V8's regexp parser, so ret follows the engines. \x3E is rejected, as it is by all of them. Group syntax errors now report the column in the original source rather than in the decoded string, so an earlier escape no longer shifts them, and name errors show the character as written. Patterns without escapes get the same messages as before. Escapes inside a group's body are still decoded as text, so (?=\x3D) is a lookahead for =. Fixes fent#55 Refs fent#56
This was referenced Oct 28, 2026
Owner
|
Hello, thanks for your contribution!
This is the strangest quirk for me, and I agree with the decision made here, following the engine makes sense. Anyway, lgtm! |
|
🎉 This PR is included in version 0.6.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
00o-sh
added a commit
to unabandoned/ret.js
that referenced
this pull request
Nov 8, 2026
Reviews the 13 upstream commits that [unabandoned/.github#53](unabandoned/.github#53) lists for this fork. **4 are ported and 9 declined**, each recorded with its reason under `upstream.reviewed` in `.unabandoned.yml`. **After this PR, `lib/` is identical to upstream master.** ## Ported (lib/ applies unchanged) - **[`b38d6691`](fent@b38d6691) fix: reject escaped characters in group syntax (fent#58).** The tokenizer decodes escapes before it parses, so `(?\x3Da)` was read as the lookahead `(?=a)`. Likewise `(?\x21a)`, `(?\x3Aa)` and `(?\x3Cn>a)` became `(?!a)`, `(?:a)` and a named group. JavaScript rejects all of these, and `reconstruct` turned them into different, valid regexps. They now throw, and group errors report the column in the original source. - **[`720c2ff8`](fent@720c2ff8) feat: group modifier support (fent#50).** `(?im-s:a)` is tokenized as a non-capturing group with a `modifier: { ignoreCase, multiline, dotAll }` object, and `reconstruct` writes it back. Its README section is included. b38d669 builds on the source map this commit adds. - **[`7fbd6e02`](fent@7fbd6e02) fix(types).** `Range` leaves the `Token` union for a new `SetToken`, `reconstruct` accepts a `Range`, and `tokenizeClass`'s `regexpStr` defaults to `''`. - **[`c26bea58`](fent@c26bea58) refactor.** Drops an empty-name check that the first-character check already makes unreachable. ## Tests Upstream's tests for these are TypeScript on mocha. Here they are plain JS on node:test, and they run against the built `dist/`: - `test/modifier-test.js`: upstream's `modifier-test.ts`, with only the imports and type annotations removed. - The `strToChars` source-map tests are appended to `util-test.js`. - The escaped-group tests are appended to `error-test.js` and `main-test.js`. `npm test` gives **342 pass, 0 fail**. With `lib/` reverted, **74 fail**. `dist/` is rebuilt with this repo's TypeScript. **Effect on `@unabandoned/randexp`:** its suite passes against this build. Modifier groups that ret used to reject now parse. randexp generates their body but ignores the modifier. ## Declined | Commits | Why | |---|---| | `0609d0d0` `fcc7fae4` | Test-runner migrations, to mocha and then to TypeScript with ts-node and eslint. This fork already runs the vows suite on node:test through `test/vows-adapter.js`. | | `a7207c2b` | `tsconfig.build.json`. Our `tsconfig.json` already sets `rootDir: ./lib`, and the tests are JS. | | `4b603034` `749035a2` | Upstream's lint, nodejs and release workflows, plus badges for `fent/ret.js`. | | `2b53f4ec` `45242341` `2076ce01` `d8fc932f` | dependabot, npm-audit and eslint, typescript-eslint and @types/node bumps on upstream's lockfile. This fork has no lockfile and no eslint, and Renovate manages `@types/node`. d8fc932's tsconfig.json change only strips comments; the options it keeps are already ours. | Typed `feat`, so it releases as a patch (pre-1.0, `bump-patch-for-minor-pre-major`). --- _Generated by [Claude Code](https://claude.ai/code)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #55. Refs #56 (group-name errors only).
What this fixes
strToCharsdecodes escapes before the tokenizer runs, so an escaped character could be read as group syntax. JavaScript rejects all of these, but ret accepted them, andreconstructturned them into different, valid regular expressions:(?\u0021a),(?\x3Da),(?\x3Aa),(?\x3Cn>a)(?!a),(?=a),(?:a),(?<n>a)Invalid group at column 2(?<n\x3Ea)n, closed at the escaped>(?<\x61>a),(?<a\x62>c)a,abThe four lowercase examples in #55 (
(?\u003aa)etc.) are rejected before and after this change, but only because ret does not decode lowercase hex yet; with this check in place they stay rejected once it does.How
(?must be literal in the source. This is the same check the modifier parser from feat: modifier support #50 already makes, now oneisLiteralhelper that both use. Errors use the existingmodifierErr, so they report the column in the original source.\straight after(?is never group syntax, which also covers escapes that decode to an escaped special character ((?\x28a)becomes(?\(ainternally).\uescape that is still a valid name character ((?<\u0061>a)is the groupa), but not\x,\cor other escapes.>may be literal or\u003E. The specification requires a literal>here, but V8 (Chrome, Node) and JavaScriptCore (Safari) both accept(?<n\u003Ea)as the groupn, and SpiderMonkey uses V8's regexp parser, so ret follows the engines.\x3Eis rejected, as it is by all of them. Happy to switch to the strict spec behaviour if you prefer.\x61(?<1a>b)reports column 8, not 5). Patterns without escapes get exactly the same messages as before.Relation to #51
#51 removes the up-front decoding, which fixes the
(?part of this by design. This is a small fix in the meantime; if #51 lands first, that part can simply be dropped. The group-name rules and the tests still apply: as currently written, #51 rejects valid\unames such as(?<\u0061>a), and these tests would catch that.Not changed (pre-existing)
\u{...}escapes in names (valid JavaScript; ret does not decode\u{...})captureGroupCharsis ASCII only)(?<\u0024>a):$is valid, but the decoder re-escapes it as\$\x3dis not decoded (Clarify supported JavaScript regex syntax in the README #57); I have a follow-up PR ready that builds on this oneTesting
test/error-test.tsandtest/main-test.ts; the error tests fail without the change.tsc, eslint and coverage (100%) clean; CI green on Node 20, 22 and 24 on my fork.