Error display for invalid literal is incorrect

[that-random-programmer]

Bug report

Bug description:

If you run

0x9g

Anywhere (I did it in the REPL), the pointer that shows where the error happened is on 9, even though it is valid:

>>> 0x9g
  File "<stdin>", line 1
    0x9g
      ^
SyntaxError: invalid hexadecimal literal

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-149277: Fix error position for invalid numeric literals #149456
• gh-149277: Fix error position for invalid numeric literals #155534

[picnixz]

It should be a known parser issue somewhere (it happens with non-decimal literals) but I can't find the issue again.

[skirpichev]

It happens with decimal literals too:

Python 3.15.0a8+ (heads/main-dirty:39f123c587a, May  2 2026, 18:34:11) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 123f
  File "<python-input-0>", line 1
    123f
      ^
SyntaxError: invalid decimal literal

[skirpichev]

This patch "works for me":

diff --git a/Parser/lexer/lexer.c b/Parser/lexer/lexer.c
index 7f25afec302..dad8d800617 100644
--- a/Parser/lexer/lexer.c
+++ b/Parser/lexer/lexer.c
@@ -352,7 +352,6 @@ verify_end_of_number(struct tok_state *tok, int c, const char *kind) {
     }
     else /* In future releases, only error will remain. */
     if (c < 128 && is_potential_identifier_char(c)) {
-        tok_backup(tok, c);
         _PyTokenizer_syntaxerror(tok, "invalid %s literal", kind);
         return 0;
     }

[serhiy-storchaka]

This is not so simple. Look st SyntaxWraning turned into SyntaxError when run with -Werror.

>>> 1or 0
  File "<python-input-0>", line 1
    1or 0
    ^
SyntaxError: invalid decimal literal
>>> 0or 1
  File "<python-input-1>", line 1
    0or 1
     ^
SyntaxError: invalid octal literal

Currently the cursor points to the end of the invalid decimal literal and the end of the invalid octal literal. The patch will change only one of them.

Other consideration is that c can be a start of a multi-byte UTF-8 sequence. tok_backup() restores the position to boundary between such sequences. With it removed, the position can point inside a multi-byte UTF-8 sequence.

We should also plan what is the best representation of such errors.

[skirpichev]

Other consideration is that c can be a start of a multi-byte UTF-8 sequence. tok_backup() restores the position to boundary between such sequences. With it removed, the position can point inside a multi-byte UTF-8 sequence.

Could you be more explicit here, any example?

I got something like this on the pr:

Python 3.16.0a0 (heads/better-lit-errors/149277:f4632ca662f, Sep  4 2026, 09:16:31) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 123абырвалг
  File "<python-input-0>", line 1
    123абырвалг
       ^^^^^^^^
SyntaxError: invalid syntax
>>> 123١٢٣٤
  File "<python-input-1>", line 1
    123١٢٣٤
       ^
SyntaxError: invalid character '١' (U+0661)

I don't think it's worse: same as in the master.

[serhiy-storchaka]

0xф

[skirpichev]

0xф

In the master:

>>> 0xф
  File "<python-input-0>", line 1
    0xф
     ^
SyntaxError: invalid hexadecimal literal

On the PR:

>>> 0xф
  File "<python-input-0>", line 1
    0xф
      ^
SyntaxError: invalid hexadecimal literal

So, what's wrong?

Edit:
I fixed SyntaxWarning's case above:

>>> 1or 0
  File "<python-input-0>", line 1
    1or 0
     ^
SyntaxError: invalid decimal literal
>>> 0or 1
  File "<python-input-1>", line 1
    0or 1
      ^
SyntaxError: invalid octal literal