Skip to content

fix(lexer): implement ANSI-C string quoting support ($'...') - #1024

Merged
HalFrgrd merged 4 commits into
HalFrgrd:masterfrom
ravvenox:master
Oct 10, 2026
Merged

HalFrgrd merged 4 commits into
HalFrgrd:masterfrom
ravvenox:master

Conversation

@ravvenox

@ravvenox ravvenox commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

The root cause of this syntax-trapping behavior is located entirely in src/grammar/lexer.rs due to a fundamental blind spot regarding Bash’s ANSI-C Quoting ($'...') syntax.

Currently, the lexer handles a standalone $ token by setting self.after_dollar = true and emitting a TokenKind::Dollar. However, because the lexer doesn't recognize the $' sequence as an atomic ANSI-C initialization block, the state tracking falls apart across token boundaries:

  1. The Issue with after_dollar Persistence: The lexer consumes after_dollar inside read_var_name() or drops it entirely before realizing that the next sequence is an opening single quote. By the time it processes the quote character ', it treats it as a standard POSIX single quote literal.
  2. The Escape Sequence Blind Spot: Inside read_quoted_content(), backslash character processing is heavily guarded behind an is_double_quote check. Because the lexer assumes it is parsing a standard single-quoted string, it completely strips the backslash's escaping power. In a string like $'aa\'bb', the quote following the backslash is mistakenly evaluated as the structural termination quote. The remaining text bb' is left unquoted, and the final quote at the very end opens a brand new, unclosed string context.

This causes DParser::walk to mark the final token as OpeningState::Unmatched, which trips needs_more_input() into incorrectly returning true and forcing a line continuation prompt (> ).

Solution Applied

Introduced a persistent state flag is_ansi_c_quote to the lexer structure. This flag is:

  1. Activated at quote-opening time if self.after_dollar is present.
  2. Utilized inside read_quoted_content() to conditionally enable backslash escape evaluation for \' and \\.
  3. Reset at quote-closing time when the string naturally terminates.

This ensures that echo $'aa\'bb' bypasses the multi-line continuation prompt entirely and executes seamlessly, while preserving pristine POSIX behavior for standalone single quotes where backslashes must remain literal text.

- Added  state tracking to the Lexer struct.
- Intercepted token boundaries in  to catch the '$' prefix.
- Updated  to handle backslash escapes inside ANSI-C strings.
- Fixes multi-line continuation prompt trap when typing single-quote escapes.
@HalFrgrd

HalFrgrd commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Thanks for the fix! Some PR feedback: ravvenox#1

@HalFrgrd
HalFrgrd merged commit 54922cf into HalFrgrd:master Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants