Skip to content

fix: reject escaped characters in group syntax - #58

Merged
fent merged 1 commit into
fent:masterfrom
danshome:fix/escaped-group-syntax
Oct 30, 2026
Merged

fent merged 1 commit into
fent:masterfrom
danshome:fix/escaped-group-syntax

Conversation

@danshome

@danshome danshome commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #55. Refs #56 (group-name errors only).

What this fixes

strToChars decodes escapes before the tokenizer runs, so an escaped character could be read as group syntax. JavaScript rejects all of these, but ret accepted them, and reconstruct turned them into different, valid regular expressions:

Pattern JavaScript ret before ret after
(?\u0021a), (?\x3Da), (?\x3Aa), (?\x3Cn>a) SyntaxError accepted as (?!a), (?=a), (?:a), (?<n>a) Invalid group at column 2
(?<n\x3Ea) SyntaxError group n, closed at the escaped > Unclosed capture group name
(?<\x61>a), (?<a\x62>c) SyntaxError groups a, ab Invalid / Unclosed capture group name

The four lowercase examples in #55 ((?\u003aa) etc.) are rejected before and after this change, but only because ret does not decode lowercase hex yet; with this check in place they stay rejected once it does.

How

  • The character after (? must be literal in the source. This is the same check the modifier parser from feat: modifier support #50 already makes, now one isLiteral helper that both use. Errors use the existing modifierErr, so they report the column in the original source.
  • A \ straight after (? is never group syntax, which also covers escapes that decode to an escaped special character ((?\x28a) becomes (?\(a internally).
  • Group names follow the specification: a name character may be literal or a \u escape that is still a valid name character ((?<\u0061>a) is the group a), but not \x, \c or other escapes.
  • The closing > may be literal or \u003E. The specification requires a literal > here, but V8 (Chrome, Node) and JavaScriptCore (Safari) both accept (?<n\u003Ea) as the group n, and SpiderMonkey uses V8's regexp parser, so ret follows the engines. \x3E is rejected, as it is by all of them. Happy to switch to the strict spec behaviour if you prefer.
  • Group-name errors now report the source column and the character as written, so an earlier escape no longer shifts them (\x61(?<1a>b) reports column 8, not 5). Patterns without escapes get exactly the same messages as before.

Relation to #51

#51 removes the up-front decoding, which fixes the (? part of this by design. This is a small fix in the meantime; if #51 lands first, that part can simply be dropped. The group-name rules and the tests still apply: as currently written, #51 rejects valid \u names such as (?<\u0061>a), and these tests would catch that.

Not changed (pre-existing)

  • \u{...} escapes in names (valid JavaScript; ret does not decode \u{...})
  • Non-ASCII group names (captureGroupChars is ASCII only)
  • (?<\u0024>a): $ is valid, but the decoder re-escapes it as \$
  • Lowercase hex such as \x3d is not decoded (Clarify supported JavaScript regex syntax in the README #57); I have a follow-up PR ready that builds on this one
  • "Invalid group" errors count columns from 0 and name errors from 1; I kept each as it was

Testing

  • New tests in test/error-test.ts and test/main-test.ts; the error tests fail without the change.
  • 364 passing; tsc, eslint and coverage (100%) clean; CI green on Node 20, 22 and 24 on my fork.
  • Independent of my lookbehind PR (feat: add lookbehind support #59): the two merge cleanly in either order, and all tests pass on the combination (387).

The tokenizer decodes escapes such as \x3D and \x21 before parsing, so
escaped characters were read as group syntax. (?\x3Da) was tokenized as
the lookahead (?=a), and (?\x21a), (?\x3Aa) and (?\x3Cn>a) as (?!a),
(?:a) and (?<n>a). In group names, (?<n\x3Ea) closed the name at the
escaped >, and (?<\x61>a) became a group named a. JavaScript rejects all
of these, and reconstruct turned them into different, valid regular
expressions.

The character after (? must now be literal in the source, as the group
modifier parser already requires, and is reported the same way: "Invalid
group" at its column in the original source. A \ there is never group
syntax either, which also covers escapes that decode to an escaped
character, such as (?\x28a). The literal check is now a single helper,
which the modifier parser uses too.

A group name may still use \u escapes, which the specification allows
((?<\u0061>a) is the group a), but no other escape. The closing > may be
literal or \u003E: the specification requires a literal >, but V8 and
JavaScriptCore accept the escaped form, as does SpiderMonkey, which uses
V8's regexp parser, so ret follows the engines. \x3E is rejected, as it
is by all of them.

Group syntax errors now report the column in the original source rather
than in the decoded string, so an earlier escape no longer shifts them,
and name errors show the character as written. Patterns without escapes
get the same messages as before. Escapes inside a group's body are still
decoded as text, so (?=\x3D) is a lookahead for =.

Fixes fent#55
Refs fent#56
@fent
fent merged commit b38d669 into fent:master Oct 30, 2026
6 checks passed
@fent

fent commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Hello, thanks for your contribution!

  • The closing > may be literal or \u003E. The specification requires a literal > here, but V8 (Chrome, Node) and JavaScriptCore (Safari) both accept (?<n\u003Ea) as the group n, and SpiderMonkey uses V8's regexp parser, so ret follows the engines. \x3E is rejected, as it is by all of them. Happy to switch to the strict spec behaviour if you prefer.

This is the strangest quirk for me, and I agree with the decision made here, following the engine makes sense.

Anyway, lgtm!

@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.6.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

00o-sh added a commit to unabandoned/ret.js that referenced this pull request Nov 8, 2026
Reviews the 13 upstream commits that
[unabandoned/.github#53](unabandoned/.github#53)
lists for this fork. **4 are ported and 9 declined**, each recorded with
its reason under `upstream.reviewed` in `.unabandoned.yml`. **After this
PR, `lib/` is identical to upstream master.**

## Ported (lib/ applies unchanged)

- **[`b38d6691`](fent@b38d6691) fix:
reject escaped characters in group syntax (fent#58).** The tokenizer decodes
escapes before it parses, so `(?\x3Da)` was read as the lookahead
`(?=a)`. Likewise `(?\x21a)`, `(?\x3Aa)` and `(?\x3Cn>a)` became
`(?!a)`, `(?:a)` and a named group. JavaScript rejects all of these, and
`reconstruct` turned them into different, valid regexps. They now throw,
and group errors report the column in the original source.
- **[`720c2ff8`](fent@720c2ff8) feat:
group modifier support (fent#50).** `(?im-s:a)` is tokenized as a
non-capturing group with a `modifier: { ignoreCase, multiline, dotAll }`
object, and `reconstruct` writes it back. Its README section is
included. b38d669 builds on the source map this commit adds.
- **[`7fbd6e02`](fent@7fbd6e02)
fix(types).** `Range` leaves the `Token` union for a new `SetToken`,
`reconstruct` accepts a `Range`, and `tokenizeClass`'s `regexpStr`
defaults to `''`.
- **[`c26bea58`](fent@c26bea58)
refactor.** Drops an empty-name check that the first-character check
already makes unreachable.

## Tests

Upstream's tests for these are TypeScript on mocha. Here they are plain
JS on node:test, and they run against the built `dist/`:

- `test/modifier-test.js`: upstream's `modifier-test.ts`, with only the
imports and type annotations removed.
- The `strToChars` source-map tests are appended to `util-test.js`.
- The escaped-group tests are appended to `error-test.js` and
`main-test.js`.

`npm test` gives **342 pass, 0 fail**. With `lib/` reverted, **74
fail**. `dist/` is rebuilt with this repo's TypeScript.

**Effect on `@unabandoned/randexp`:** its suite passes against this
build. Modifier groups that ret used to reject now parse. randexp
generates their body but ignores the modifier.

## Declined

| Commits | Why |
|---|---|
| `0609d0d0` `fcc7fae4` | Test-runner migrations, to mocha and then to
TypeScript with ts-node and eslint. This fork already runs the vows
suite on node:test through `test/vows-adapter.js`. |
| `a7207c2b` | `tsconfig.build.json`. Our `tsconfig.json` already sets
`rootDir: ./lib`, and the tests are JS. |
| `4b603034` `749035a2` | Upstream's lint, nodejs and release workflows,
plus badges for `fent/ret.js`. |
| `2b53f4ec` `45242341` `2076ce01` `d8fc932f` | dependabot, npm-audit
and eslint, typescript-eslint and @types/node bumps on upstream's
lockfile. This fork has no lockfile and no eslint, and Renovate manages
`@types/node`. d8fc932's tsconfig.json change only strips comments; the
options it keeps are already ours. |

Typed `feat`, so it releases as a patch (pre-1.0,
`bump-patch-for-minor-pre-major`).

---
_Generated by [Claude Code](https://claude.ai/code)_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Some encoded characters in groups should throw an error

2 participants