gh-149277: Fix error position for invalid numeric literals by anujbharambe · Pull Request #149456 · python/cpython

anujbharambe · 2026-05-06T12:54:23Z

Summary

Fix the SyntaxError caret position for invalid numeric literals. Previously,
the caret pointed at the last valid digit instead of the first invalid character.

For example, 0x9g now correctly shows:
0x9g
^
SyntaxError: invalid hexadecimal literal

Instead of the previous incorrect output:
0x9g
^
SyntaxError: invalid hexadecimal literal

The issue was a spurious tok_backup(tok, c) call in verify_end_of_number()
in the error branch. Since _PyTokenizer_syntaxerror computes the column offset
from tok->cur, backing up one character caused the caret to point at the
preceding (valid) character. Removing the backup keeps tok->cur at the correct
position.

Fixes #149277

Issue: Error display for invalid literal is incorrect #149277

picnixz

Please add tests.

bedevere-app · 2026-05-06T18:08:49Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

read-the-docs-community · 2026-05-06T19:33:22Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #32571271 | 📁 Comparing 20996e1 against main (ff422bd)

🔍 Preview build

18 files changed · ± 18 modified

± Modified

anujbharambe · 2026-05-06T19:51:23Z

I have made the requested changes; please review again.

Added test_end_of_numerical_literals_offset in Lib/test/test_grammar.py that verifies SyntaxError.offset points at the first invalid character for various invalid numeric literal cases (hex, binary, octal, decimal, float, scientific notation, and imaginary literals).

bedevere-app · 2026-05-06T19:51:29Z

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

picnixz · 2026-05-09T10:03:22Z

+            with self.subTest(source=source):
+                with self.assertRaises(SyntaxError) as cm:
+                    compile(source, "<test>", "eval")
+                self.assertEqual(cm.exception.offset, expected_offset)


Please also test the rendered carret not just the carret offset. Use check_syntax_error for that.

picnixz · 2026-05-09T10:03:38Z

+    def test_end_of_numerical_literals_offset(self):
+        # gh-149277: verify the error caret points at the first invalid
+        # character, not the last valid digit.
+        cases = [
+            ("0xfg", 4),
+            ("0x9g", 4),
+            ("0b1z", 4),
+            ("0o7q", 4),
+            ("9spam", 2),
+            ("0xfspam", 4),
+            ("1.0x", 4),
+            ("1e3w", 4),
+            ("1jz", 3),
+        ]
+        for source, expected_offset in cases:


Use @subTests decorator for cases instead of this loop and subTest().

bedevere-app · 2026-05-09T10:05:50Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pablogsal · 2026-07-11T16:00:41Z

+    def test_end_of_numerical_literals_offset(self):
+        # gh-149277: verify the error caret points at the first invalid
+        # character, not the last valid digit.
+        cases = [


Nit: the cases here all go through verify_end_of_number. Can we add 0xI/0bz too so we lock the behaviour for the prefix-only path, and maybe assert .msg as well?

skirpichev

@anujbharambe, there is a merge conflict now. Could you fix?

Also, to address @pablogsal review you need more changes. I believe that following patch will be complete:

diff --git a/Parser/lexer/number.c b/Parser/lexer/number.c
index 8bca8cbb9ad..02e11d12616 100644
--- a/Parser/lexer/number.c
+++ b/Parser/lexer/number.c
@@ -80,7 +80,6 @@ verify_end_of_number(struct tok_state *tok, int c, const char *kind) {
     }
     else /* In future releases, only error will remain. */
     if (c < 128 && is_potential_identifier_char(c)) {
-        tok_backup(tok, c);
         _PyTokenizer_syntaxerror(tok, "invalid %s literal", kind);
         return 0;
     }
@@ -130,7 +129,6 @@ _PyLexer_scan_number(struct tok_state *tok, struct token *token, int c,
                     c = tok_nextc(tok);
                 }
                 if (!Py_ISXDIGIT(c)) {
-                    tok_backup(tok, c);
                     return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "invalid hexadecimal literal"));
                 }
                 do {
@@ -154,7 +152,6 @@ _PyLexer_scan_number(struct tok_state *tok, struct token *token, int c,
                                 "invalid digit '%c' in octal literal", c));
                     }
                     else {
-                        tok_backup(tok, c);
                         return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "invalid octal literal"));
                     }
                 }
@@ -182,7 +179,6 @@ _PyLexer_scan_number(struct tok_state *tok, struct token *token, int c,
                         return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "invalid digit '%c' in binary literal", c));
                     }
                     else {
-                        tok_backup(tok, c);
                         return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "invalid binary literal"));
                     }
                 }

skirpichev · 2026-07-29T07:23:40Z

+    def test_end_of_numerical_literals_offset(self):
+        # gh-149277: verify the error caret points at the first invalid
+        # character, not the last valid digit.
+        cases = [
+            ("0xfg", 4),
+            ("0x9g", 4),
+            ("0b1z", 4),
+            ("0o7q", 4),
+            ("9spam", 2),
+            ("0xfspam", 4),
+            ("1.0x", 4),
+            ("1e3w", 4),
+            ("1jz", 3),
+        ]
+        for source, expected_offset in cases:
+            with self.subTest(source=source):
+                with self.assertRaises(SyntaxError) as cm:
+                    compile(source, "<test>", "eval")
+                self.assertEqual(cm.exception.offset, expected_offset)
+


Suggested change

def test_end_of_numerical_literals_offset(self):

# gh-149277: verify the error caret points at the first invalid

# character, not the last valid digit.

cases = [

("0xfg", 4),

("0x9g", 4),

("0b1z", 4),

("0o7q", 4),

("9spam", 2),

("0xfspam", 4),

("1.0x", 4),

("1e3w", 4),

("1jz", 3),

]

for source, expected_offset in cases:

with self.subTest(source=source):

with self.assertRaises(SyntaxError) as cm:

compile(source, "<test>", "eval")

self.assertEqual(cm.exception.offset, expected_offset)

@subTests('source,offset,msg',

[("0xfg", 4, "hexadecimal"),

("0x9g", 4, "hexadecimal"),

("0b1z", 4, "binary"),

("0o7q", 4, "octal"),

("9spam", 2, " decimal"),

("0xfspam", 4, "hexadecimal"),

("1.0x", 4, " decimal"),

("1e3w", 4, " decimal"),

("1jz", 3, "imaginary"),

("0xI", 3, "hexadecimal"),

("0bz", 3, "binary"),

])

def test_end_of_numerical_literals_offset(self, source, offset, msg):

# gh-149277: verify the error caret points at the first invalid

# character, not the last valid digit.

with self.assertRaises(SyntaxError) as cm:

compile(source, "<test>", "eval")

self.assertEqual(cm.exception.offset, offset)

self.assertIn(msg, cm.exception.msg)

pablogsal · 2026-07-30T14:53:40Z

+    def test_end_of_numerical_literals_offset(self):
+        # gh-149277: verify the error caret points at the first invalid
+        # character, not the last valid digit.
+        cases = [


Nit: the cases here all go through verify_end_of_number. Can we add 0xI/0bz too so we lock the behaviour for the prefix-only path, and maybe assert .msg as well?

skirpichev · 2026-08-11T05:47:25Z

Ok, this still has merge conflicts. Closing.

Continued in #155534. @pablogsal, I hope your review was addressed.

pythongh-149277: Fix error position for invalid numeric literals

88d52b8

anujbharambe requested review from lysnikolaou and pablogsal as code owners May 6, 2026 12:54

bedevere-app Bot mentioned this pull request May 6, 2026

Error display for invalid literal is incorrect #149277

Open

bedevere-app Bot added the awaiting review label May 6, 2026

pythongh-149277: Fix error position for invalid numeric literals

5bf3295

picnixz requested changes May 6, 2026

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting review labels May 6, 2026

Add tests for invalid numeric literal error offset

20996e1

bedevere-app Bot added awaiting change review and removed awaiting changes labels May 6, 2026

bedevere-app Bot requested a review from picnixz May 6, 2026 19:51

picnixz requested changes May 9, 2026

View reviewed changes

bedevere-app Bot removed the awaiting change review label May 9, 2026

bedevere-app Bot added the awaiting changes label May 9, 2026

pablogsal reviewed Jul 11, 2026

View reviewed changes

skirpichev reviewed Jul 29, 2026

View reviewed changes

skirpichev self-assigned this Jul 29, 2026

skirpichev added the pending The issue will be closed if no feedback is provided label Jul 29, 2026

pablogsal reviewed Jul 30, 2026

View reviewed changes

skirpichev closed this Aug 11, 2026

skirpichev removed the pending The issue will be closed if no feedback is provided label Aug 11, 2026

skirpichev removed their assignment Aug 11, 2026

Uh oh!

Uh oh!

Conversation

anujbharambe commented May 6, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented May 6, 2026

Uh oh!

read-the-docs-community Bot commented May 6, 2026

Documentation build overview

Uh oh!

anujbharambe commented May 6, 2026

Uh oh!

bedevere-app Bot commented May 6, 2026

Uh oh!

picnixz May 9, 2026

Choose a reason for hiding this comment

Uh oh!

picnixz May 9, 2026

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented May 9, 2026

Uh oh!

pablogsal Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

skirpichev left a comment

Choose a reason for hiding this comment

Uh oh!

skirpichev Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

pablogsal Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

skirpichev commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anujbharambe commented May 6, 2026 •

edited by bedevere-app Bot

Loading