[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

We are porting the Ruby regex library to C++ and trimming it to exactly the grammar C++ std::regex implements (ECMAScript, per [re.grammar] / ECMA-262 5.1).

⚠️ ACCEPTANCE CONTRACT — READ FIRST

This task is accepted only if the final branch contains exactly the numbered commit sequence below — same commits, same order, one concern each. The commit structure is the deliverable. A branch that reaches the right end state via a different set/order/grouping of commits (squashed, reordered, combined, or with extra commits) will be rejected and thrown away.

• Produce exactly the commits below, in order. Commits 1–3 must be genuine git cherry-picks of the named SHAs (message + authorship preserved), not re-implementations.
• Begin by reading the three cherry-picked commits and confirming the facts stated below. If anything does not match — a removal target is absent, \0 is already handled, an extra construct exists, or a "verify" step's outcome differs from what is assumed — STOP and report before writing the affected commit. Do not improvise.

Confirmed facts about the imported code (verified against the three commits)

• Parser: cpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qll. Tree view: cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll. Tests: cpp/ql/test/library-tests/regex/{parse,regexp}.{ql,expected} and corpus cpp/ql/test/library-tests/regex/regexp.cpp (wrapped in an inline std::regex stub). There is no locations.ql and no Consistency.ql.
• Location handling in RegexTreeView.qll::hasLocationInfo currently hardcodes startcolumn = re_start + 1 + start / endcolumn = re_start + 1 + end - 1. The + 1 assumes a single-character opening delimiter and is the off-by-one bug. There is no content-offset predicate yet.
• The imported grammar still recognizes all Ruby constructs: \A \Z \z \G, \h/\H, \p{}/\P{}, (?#...), (?'name'...) and (?<name>...), \k<name>, and lookbehind (?<=)/(?<!). \0 is NOT specially handled. The only POSIX bracket support is posixStyleNamedCharacterProperty, matching [[:name:]] only — [[.a.]] and [[=a=]] are absent. Braced \u{...} is mis-parsed by the \uhhhh-only handling. Lazy quantifiers (*? etc.) are handled by qualifier but untested; \f/\v are handled by the tree view but untested; the meaning of \b inside a character class (should be backspace U+0008, not a word boundary) is untested.

General rules

Each commit compiles and passes codeql test run on cpp/ql/test/library-tests/regex/, except the pre-fix commits (4, 15, 17, 19, 21, and 24 only if a fix is required) which intentionally record incorrect expected output but must still run. Every .expected file must come from codeql test run --learn — never hand-authored.

Commits

Commit 1 — Cherry-pick: copy Ruby regex library and tests verbatim — git cherry-pick fe3c6de7cf7e7956ecd135a0c96438d19827d7cf.

Commit 2 — Cherry-pick: make regex library compile — git cherry-pick 51c94dbd9677561c90d1eb87d2a3eb1ac507e5b4.

Commit 3 — Cherry-pick: make regex test work — git cherry-pick 176f2c1dbe7ba5f932bbbc4d8c56027117ced2f1.

Commit 4 — Add location tests for all literal prefixes (pre-fix). Add a new locations.ql + locations.expected reporting hasLocationInfo columns. Extend regexp.cpp with one regex literal of each spelling: plain "..."; L"...", u8"...", u"...", U"..."; raw R"(...)"; prefixed raw LR"(...)", u8R"(...)", uR"(...)", UR"(...)"; custom-delimiter raw R"x(...)x" and multi-char R"foo(...)foo"; and at least one literal whose content has escapes. Generate .expected with --learn. Columns will be wrong for every non-plain spelling — note commit 5 fixes it. (Regenerate parse.expected/regexp.expected too.)

Commit 5 — Fix the location off-by-one. In hasLocationInfo, replace the hardcoded + 1 with a content offset derived from StringLiteral.getValueText(), accounting for encoding prefix (L/u/U/u8), the R" raw introducer, and the (possibly multi-char, possibly empty) raw delimiter. Regenerate affected .expected with --learn; all columns now correct.

Commits 6–13 each remove one Ruby-only construct absent from the std::regex ECMAScript grammar. Each: edit parser/tree view, remove the construct's cases from regexp.cpp, regenerate affected .expected with --learn.

Commit 6 — Remove \A, \Z, \z, \G anchors (keep \b/\B, ^, $).

Commit 7 — Remove \h / \H (from RegExpCharacterClassEscape's value set).

Commit 8 — Remove \p{...} / \P{...} (pStyleNamedCharacterProperty and all wiring; the RegExpNamedCharacterProperty class). Keep only POSIX [[:...:]] (posixStyleNamedCharacterProperty); [[.a.]]/[[=a=]] are absent and added in commits 18–21.

Commit 9 — Remove (?#...) comment groups (commentGroupStart).

Commit 10 — Remove single-quote named groups (?'name'...) (only the '-branch of namedGroupStart).

Commit 11 — Remove named capture groups (?<name>...) (remaining namedGroupStart branch + getGroupName/numbering; keep (?<=/(?<! detection working).

Commit 12 — Remove named backreferences \k<name> (namedBackreference; \k → identity escape).

Commit 13 — Remove lookbehind (?<=...) / (?<!...) (assertion-start predicates, negativeAssertionGroup/positiveLookbehindAssertionGroup paths, and the tree-view RegExpLookbehind/RegExpPositiveLookbehind/RegExpNegativeLookbehind classes).

Commits 14–21 add std::regex constructs missing from the port, as pre-fix/fix pairs; both regenerate .expected with --learn.

Commit 14 — Add \cX control-escape tests (pre-fix).

Commit 15 — Implement \cX (3-char case in escapedCharacter).

Commit 16 — Add \0 NUL-escape tests (pre-fix) (\0 not followed by a digit).

Commit 17 — Implement \0.

Commit 18 — Add POSIX collating-symbol [[.a.]] tests (pre-fix) (standalone and inside a larger class).

Commit 19 — Implement [[.a.]] (a [.….] bracket-inner predicate analogous to posixStyleNamedCharacterProperty, excluded from charset-delimiter counting; plus tree-view exposure).

Commit 20 — Add POSIX equivalence-class [[=a=]] tests (pre-fix).

Commit 21 — Implement [[=a=]] (mirroring the collating-symbol implementation).

Commit 22 — Clean up non-standard \u{...} corpus entry. Remove the imported \u{9879} entry (braced \u{...} is ES2015+, not in std::regex, currently mis-parsed). Replace with a valid \uhhhh case (e.g. \u0061). Do not add braced-form support or change \uhhhh handling. Regenerate with --learn.

Commit 23 — Add coverage for already-supported constructs (lazy quantifiers, \f, \v). Add corpus cases for lazy quantifiers *?, +?, ??, {2,3}? and for the \f and \v escapes. These are expected to already parse correctly — this is pure test coverage, no parser change. Regenerate .expected with --learn. If any of these do NOT already parse correctly, STOP and report rather than silently fixing here.

Commit 24 — Verify (and if needed fix) \b = backspace inside a character class. In std::regex ECMAScript, \b inside [...] matches backspace (U+0008), not a word boundary. Add a [\b] corpus case and regenerate with --learn. Inspect the result:

• If the parser already treats [\b] as a single backspace character constant (and NOT as RegExpWordBoundary), this commit is test-only and is complete.
• If it is mis-parsed (e.g. as a word boundary, or wrong span/value), first commit the pre-fix .expected here, then STOP and report so a numbered fix commit can be added — do not fold an undocumented parser change into a test commit.

Commit 25 — Change note and housekeeping. Add an unreleased minorAnalysis change note stating the parser targets the std::regex ECMAScript grammar. Remove the temporary RegExpTest extends RegExp { RegExpTest() { any() } } stop-gap only if no longer needed; otherwise leave it and note why. No queries, dataflow, or flow-configs.

Constraints

• The commit sequence is a hard acceptance criterion; keep commits atomic. Commits 4, 14, 16, 18, 20 (and 24 only if a fix is needed) record intentionally-wrong .expected but must run.
• All .expected via codeql test run --learn; never hand-edit.
• Match the std::regex ECMAScript grammar exactly — no supersets, no other dialects.
• Follow the imported code's naming and doc-comment conventions.

The following is the prior conversation context from the user's chat exploration (may be truncated):

Assistant: You're right. Here's the prompt with every commit explicitly numbered, including each removal and each pair.

---

Prompt

Title: C++ ECMAScript regex parser: cherry-pick Ruby import, fix location off-by-one, and align with the std::regex ECMAScript grammar

Repository: github/codeql
Base branch: main

Problem statement:

We are building a C++ ECMAScript regular-expression parser and parse-tree view by porting the Ruby regex library, then trimming it to exactly the grammar C++ std::regex implements (ECMAScript, per [re.grammar] / ECMA-262 5.1). This must land as a clean, reviewable sequence of small, individually-numbered commits. Do not squash. Each commit must compile and pass codeql test run on the regex test directory, except the explicitly-marked "pre-fix" commits (11, 13, 15) which intentionally record incorrect expected output but must still run. Every .expected file must be produced by codeql test run --learn — never hand-authored.

Build the branch as exactly the following commits, in this order:

Commit 1 — Cherry-pick: import Ruby parser
Cherry-pick fe3c6de7cf7e7956ecd135a0c96438d19827d7cf from jketema/ecma-regex, preserving message and authorship.

Commit 2 — Cherry-pick: make it compile against C++
Cherry-pick 51c94dbd9677561c90d1eb87d2a3eb1ac507e5b4.

Commit 3 — Cherry-pick: make the tests work
Cherry-pick 176f2c1dbe7ba5f932bbbc4d8c56027117ced2f1.

If a cherry-pick conflicts, resolve it faithfully to the original commit's intent; add nothing unrelated.

Commit 4 — Add location tests for all literal prefixes (pre-fix)
Add locations.ql + locations.expected, exercising the content-offset computation across every C++ string-literal prefix/delimiter form. test.cpp must include a regex literal of each: plain "..."; L"...", u8"...", u"...", U"..."; raw R"(...)"; prefixed raw LR"(...)", u8R"(...)", uR"(...)", UR"(...)"; custom-delimiter raw R"x(...)x" and multi-char delimiter R"foo(...)foo"; and at least one literal whose content contains metacharacters/escapes so mapping past offset 0 is exercised. Generate locations.expected with --learn. This commit captures the incorrect (off-by-one) columns — note in the message that commit 5 fixes it.

Commit 5 — Fix the location off-by-one
Fix the content-offset predicate (e.g. regexpContentOffset) to derive the true offset from getValueText(), accounting for encoding prefix (L/u/U/u8), the R" raw introducer, and delimiter length. Regenerate locations.expected with --learn; columns from commit 4 must now be correct.

Commit 6 — Remove named capture groups (?<name>...)
Remove namedGroupStart and its use in groupStart/getGroupName/group numbering; ensure (?<=/(?<! detection no longer depends on it. Remove the corresponding test.cpp cases. Regenerate all affected .expected with --learn. (Not supported by std::regex — confirmed against libstdc++ and libc++.)

Commit 7 — Remove named backreferences \k<name>
Remove namedBackreference and its wiring; \k becomes an identity escape (literal k). Remove corpus cases. Regenerate affected .expected with --learn.

Commit 8 — Remove lookbehind assertions (?<=...) / (?<!...)
Remove lookbehindAssertionStart / negativeLookbehindAssertionStart, the associated negativeAssertionGroup / positiveLookbehindAssertionGroup paths, and the tree-view RegExpLookbehind hierarchy. Remove corpus cases. Regenerate affected .expected with --learn. (Not in the ECMA-262 grammar std::regex implements — confirmed against libstdc++ and libc++.)

(Ruby-only constructs that the cherry-picked commits 1–3 should already have removed — \A \Z \z \G, \h \H, \p{}/\P{}, (?#...) comments, (?'name'...) single-quote groups — do not get their own removal commit. Verify they are gone. If any survived the port, add a numbered removal commit for it immediately after commit 8, continuing the numbering, and shift the subsequent commit numbers accordingly.)

Commit 9 — Add \cX control-escape tests (pre-fix)
Add test.cpp cases for \cA, \cM, etc. Regenerate affected .expected with --learn, capturing the incorrect pre-fix parse. Note that commit 10 fixes it.

Commit 10 — Implement \cX control escape
Add a \cX case to escapedCharacter consuming 3 characters. Regenerate affected .expected with --learn; the parse must now be correct.

Commit 11 — Add \0 NUL-escape tests (pre-fix)
Add test.cpp cases for \0 (not followed by another digit). Regenerate affected .expected with --learn, capturing the pre-fix parse. If the cherry-picked parser already handles \0 correctly, skip commits 11 and 12 entirely and record in the PR description why (with the observed correct output).

Commit 12 — Implement \0 NUL escape
Add \0 handling to escapedCharacter. Regenerate affected .expected with --learn; the parse must now be correct.

Commit 13 — Change note and housekeeping
Add an unreleased minorAnalysis change note describing the C++ ECMAScript regex parse library and that it targets the std::regex ECMAScript grammar exactly. No queries, dataflow, or flow-configs.

Constraints:

• Preserve the exact commit numbering/ordering above; keep commits atomic. Commits 4, 9, and 11 intentionally record pre-fix (incorrect) expected output but must still run cleanly.
• Reproduce all .expected via codeql test run --learn; never edit them by hand.
• Match the standard std::regex ECMAScript grammar exactly — no supersets, no other dialects (BRE/ERE out of scope).
• Follow existing predicate-naming and doc-comment conventions from the ported code.

---

Note the two numbering contingencies baked in: (a) if commits 1–3 left extra Ruby-only features in, an additional numbered removal commit gets inserted after commit 8 and everything shifts; (b) if \0 already works, commits 11–12 drop out. Both are the kind of thing that's only knowable from the actual cherry-pick diffs — which I still haven't been able to read. Want me to fetch those three diffs now so we can lock the numbering (confirm the remove-list and whether \0 survives) before you launch?

User: This is not true: Ruby-only constructs that the cherry-picked commits 1–3 should already have removed

Nothing was removed. Update the prompt accordingly

Assistant: You're right — the cherry-picked commits only import, compile, and get tests passing; they remove nothing. So every Ruby-only construct not in the std::regex ECMAScript grammar needs its own numbered removal commit. Here's the corrected prompt.

---

Prompt

Title: C++ ECMAScript regex parser: cherry-pick Ruby import, fix location off-by-one, and align with the std::regex ECMAScript grammar

Repository: github/codeql
Base branch: main

Problem statement:

We are building a C++ ECMAScript regular-expression parser and parse-tree view by porting the Ruby regex library, then trimming it to exactly the grammar C++ std::regex implements (ECMAScript, per [re.grammar] / ECMA-262 5.1). This must land as a clean, reviewable sequence of small, individually-numbered commits. Do not squash. Each commit must compile and pass codeql test run on the regex test directory, except the explicitly-marked "pre-fix" commits (4, 15, 17) which intentionally record incorrect expected output but must still run. Every .expected file must be produced by codeql test run --learn — never hand-authored.

The three cherry-picked commits only import the Ruby parser, make it compile against C++, and get the tests passing. They remove nothing: the imported parser still recognizes the full Ruby grammar. Every Ruby-only construct therefore needs its own explicit removal commit below.

Build the branch as exactly the following commits, in this order:

Commit 1 — Cherry-pick: import Ruby parser
Cherry-pick fe3c6de7cf7e7956ecd135a0c96438d19827d7cf from jketema/ecma-regex, preserving message and authorship.

Commit 2 — Cherry-pick: make it compile against C++
Cherry-pick 51c94dbd9677561c90d1eb87d2a3eb1ac507e5b4.

Commit 3 — Cherry-pick: make the tests work
Cherry-pick 176f2c1dbe7ba5f932bbbc4d8c56027117ced2f1.

If a cherry-pick conflicts, resolve it faithfully to the original commit's intent; add nothing unrelated.

Commit 4 — Add location tests for all literal prefixes (pre-fix)
Add locations.ql + locations.expected, exercising the content-offset computation across every C++ string-literal prefix/delimiter form. test.cpp must include a regex literal of each: plain "..."; L"...", u8"...", u"...", U"..."; raw R"(...)"; prefixed raw LR"(...)", u8R"(...)", uR"(...)", UR"(...)"; custom-delimiter raw R"x(...)x" and multi-char delimiter R"foo(...)foo"; and at least one literal whose content contains metacharacters/escapes so mapping past offset 0 is exercised. Generate locations.expected with --learn. This commit captures the incorrect (off-by-one) columns — note in the message that commit 5 fixes it.

Commit 5 — Fix the location off-by-one
Fix the content-offset predicate (e.g. regexpContentOffset) to derive the true offset from getValueText(), accounting for encoding prefix (L/u/U/u8), the R" raw introducer, and delimiter length. Regenerate locations.expected with --learn; columns from commit 4 must now be correct.

Commits 6–13 each remove one Ruby-only construct that is not part of the std::regex ECMAScript grammar. For each: adjust the parser/tree view, remove the corresponding test.cpp corpus cases, and regenerate all affected .expected files with --learn. One construct per commit:

Commit 6 — Remove \A, \Z, \z, \G anchors
Remove these Ruby-only anchors from specialCharacter and from the firstPart/lastPart anchor handling.

Commit 7 — Remove \h / \H hex-digit class escapes
Remove the Ruby-only \h/\H escapes.

Commit 8 — Remove \p{...} / \P{...} Unicode property escapes
Remove pStyleNamedCharacterProperty and its wiring (namedCharacterProperty, baseItem, character, inversion handling). Keep POSIX [[:class:]], [[.a.]], [[=a=]] bracket expressions — those are valid in std::regex.

Commit 9 — Remove (?#...) comment groups
Remove commentGroupStart and its use in groupStart.

Commit 10 — Remove single-quote named groups (?'name'...)
Remove the single-quote branch of namedGroupStart (leave the angle-bracket form for now; it is removed in commit 11).

Commit 11 — Remove named capture groups (?<name>...)
Remove the remaining (angle-bracket) namedGroupStart and its use in groupStart/getGroupName/group numbering; ensure (?<=/(?<! detection no longer depends on it. (Not supported by std::regex — confirmed against libstdc++ and libc++.)

Commit 12 — Remove named backreferences \k<name>
Remove namedBackreference and its wiring; \k becomes an identity escape (literal k).

Commit 13 — Remove lookbehind assertions (?<=...) / (?<!...)
Remove lookbehindAssertionStart / negativeLookbehindAssertionStart, the associated negativeAssertionGroup / positiveLookbehindAssertionGroup paths, and the tree-view RegExpLookbehind hierarchy. (Not in the ECMA-262 grammar std::regex implements — confirmed against libstdc++ and libc++.)

After each removal, confirm Consistency.expected (if present) stays empty; if a removal makes a retained corpus regex fail to parse, fix the corpus.

Commits 14–17 add ECMAScript constructs missing from the Ruby port, as pairs (tests pre-fix, then fix). Both commits in each pair regenerate .expected with --learn.

Commit 14 — Add \cX control-escape tests (pre-fix)
Add test.cpp cases for \cA, \cM, etc. Regenerate affected .expected with --learn, capturing the incorrect pre-fix parse. Note that commit 15 fixes it.

Commit 15 — Implement \cX control escape
Add a \cX case to escapedCharacter consuming 3 characters. Regenerate affected .expected with --learn; the parse must now be correct.

Commit 16 — Add \0 NUL-escape tests (pre-fix)
Add test.cpp cases for \0 (not followed by another digit). Regenerate affected .expected with --learn, capturing the pre-fix parse. Note that commit 17 fixes it.

Commit 17 — Implement \0 NUL escape
Add \0 handling to escapedCharacter. Regenerate affected .expected with --learn; the parse must now be correct.

Commit 18 — Change note and housekeeping
Add an unreleased minorAnalysis change note describing the C++ ECMAScript regex parse library and that it targets the std::regex ECMAScript grammar exactly. No queries, dataflow, or flow-configs.

Constraints:

• Preserve the exact commit numbering/ordering above; keep commits atomic. Commits 4, 14, and 16 intentionally record pre-fix (incorrect) expected output but must still run cleanly.
• Reproduce all .expected via codeql test run --learn; never edit them by hand.
• Match the standard std::regex ECMAScript grammar exactly — no supersets, no other dialects (BRE/ERE out of scope).
• Follow existing predicate-naming and doc-comment conventions from the ported code.

---

Two things I'd still verify before launching, both from the actual cherry-pick diffs (which I haven't been able to read):

• The removal list (commits 6–13) assumes the imported parser matches the Ruby ParseRegExp.qll we read at base 5690ec7. If commits 1–3 altered the grammar surface at all, some of these commits may be no-ops or need adjustment.
• \0 (commits 16–17) — the Ruby original doesn't handle \0, so this pair should be real, but worth confirming the imported escapedCharacter still lacks it.

Want me to pull those three diffs to lock the list before you run it?

User: Update the prompt to make sure that the result is only accepted by me if the exact commit sequence occur. I don't want to have throw away the work.