Repository navigation
fix(lexer): implement ANSI-C string quoting support ($'...') - #1024
Merged
Merged
Conversation
- Added state tracking to the Lexer struct. - Intercepted token boundaries in to catch the '$' prefix. - Updated to handle backslash escapes inside ANSI-C strings. - Fixes multi-line continuation prompt trap when typing single-quote escapes.
Owner
|
Thanks for the fix! Some PR feedback: ravvenox#1 |
Add tests and more fixes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The root cause of this syntax-trapping behavior is located entirely in
src/grammar/lexer.rsdue to a fundamental blind spot regarding Bash’s ANSI-C Quoting ($'...') syntax.Currently, the lexer handles a standalone
$token by settingself.after_dollar = trueand emitting aTokenKind::Dollar. However, because the lexer doesn't recognize the$'sequence as an atomic ANSI-C initialization block, the state tracking falls apart across token boundaries:after_dollarPersistence: The lexer consumesafter_dollarinsideread_var_name()or drops it entirely before realizing that the next sequence is an opening single quote. By the time it processes the quote character', it treats it as a standard POSIX single quote literal.read_quoted_content(), backslash character processing is heavily guarded behind anis_double_quotecheck. Because the lexer assumes it is parsing a standard single-quoted string, it completely strips the backslash's escaping power. In a string like$'aa\'bb', the quote following the backslash is mistakenly evaluated as the structural termination quote. The remaining textbb'is left unquoted, and the final quote at the very end opens a brand new, unclosed string context.This causes
DParser::walkto mark the final token asOpeningState::Unmatched, which tripsneeds_more_input()into incorrectly returningtrueand forcing a line continuation prompt (>).Solution Applied
Introduced a persistent state flag
is_ansi_c_quoteto the lexer structure. This flag is:self.after_dollaris present.read_quoted_content()to conditionally enable backslash escape evaluation for\'and\\.This ensures that
echo $'aa\'bb'bypasses the multi-line continuation prompt entirely and executes seamlessly, while preserving pristine POSIX behavior for standalone single quotes where backslashes must remain literal text.