Heap-buffer-overflow in the `hz` incremental decoder on a trailing `~`

[djarfluka]

Bug report

Bug description:

import codecs
d = codecs.getincrementaldecoder('hz')()
d.decode(b'~', False)   
d.decode(b'', True)

=================================================================
==1==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5020000040d1 at pc 0xffff7feff068 bp 0xffffd608df50 sp 0xffffd608df40
READ of size 1 at 0x5020000040d1 thread T0
    #0 0xffff7feff064 in hz_decode Modules/cjkcodecs/_codecs_cn.c:417
    #1 0xffff7fe7426c in decoder_feed_buffer Modules/cjkcodecs/multibytecodec.c:921
    #2 0xffff7fe76238 in _multibytecodec_MultibyteIncrementalDecoder_decode_impl Modules/cjkcodecs/multibytecodec.c:1214
    #3 0xffff7fe76238 in _multibytecodec_MultibyteIncrementalDecoder_decode Modules/cjkcodecs/clinic/multibytecodec.c.h:395
    #4 0xaaaad3656320 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:144
    #5 0xaaaad3656320 in PyObject_Vectorcall Objects/call.c:327
    #6 0xaaaad3ca1bc8 in _Py_VectorCallInstrumentation_StackRefSteal Python/ceval.c:768
    #7 0xaaaad33d1de8 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1906
    #8 0xaaaad3cc0b1c in _PyEval_EvalFrame Include/internal/pycore_ceval.h:122
    #9 0xaaaad3cc0b1c in _PyEval_Vector Python/ceval.c:2174
    #10 0xaaaad3cc0b1c in PyEval_EvalCode Python/ceval.c:679
    #11 0xaaaad3ebfeb4 in run_mod Python/pythonrun.c:1509
    #12 0xaaaad3ec0d38 in _PyRun_File Python/pythonrun.c:1332
    #13 0xaaaad3ec4ffc in _PyRun_SimpleFile Python/pythonrun.c:544
    #14 0xaaaad3ec8dd0 in _PyRun_AnyFile Python/pythonrun.c:92
    #15 0xaaaad3f9390c in pymain_run_file_obj Modules/main.c:478
    #16 0xaaaad3f9390c in pymain_run_file Modules/main.c:494
    #17 0xaaaad3f9390c in pymain_run_python Modules/main.c:804
    #18 0xaaaad3f9390c in Py_RunMain Modules/main.c:891
    #19 0xaaaad3f95c38 in pymain_main Modules/main.c:921
    #20 0xaaaad3f95c38 in Py_BytesMain Modules/main.c:945
    #21 0xffff82c484c0  (/lib/aarch64-linux-gnu/libc.so.6+0x284c0) (BuildId: 27027b96e5b8c475fc327aa445bea1c71d37b4e2)
    #22 0xffff82c48594 in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x28594) (BuildId: 27027b96e5b8c475fc327aa445bea1c71d37b4e2)
    #23 0xaaaad343beec in _start (/src/cpython/python+0x183beec) (BuildId: eba909a3db0adabacf151d7dc3081dda7999191a)

0x5020000040d1 is located 0 bytes after 1-byte region [0x5020000040d0,0x5020000040d1)
allocated by thread T0 here:
    #0 0xffff835e76d0 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xffff7fe76090 in _multibytecodec_MultibyteIncrementalDecoder_decode_impl Modules/cjkcodecs/multibytecodec.c:1201
    #2 0xffff7fe76090 in _multibytecodec_MultibyteIncrementalDecoder_decode Modules/cjkcodecs/clinic/multibytecodec.c.h:395
    #3 0xaaaad3656320 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:144
    #4 0xaaaad3656320 in PyObject_Vectorcall Objects/call.c:327
    #5 0xaaaad3ca1bc8 in _Py_VectorCallInstrumentation_StackRefSteal Python/ceval.c:768
    #6 0xaaaad33d1de8 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1906
    #7 0xaaaad3cc0b1c in _PyEval_EvalFrame Include/internal/pycore_ceval.h:122
    #8 0xaaaad3cc0b1c in _PyEval_Vector Python/ceval.c:2174
    #9 0xaaaad3cc0b1c in PyEval_EvalCode Python/ceval.c:679
    #10 0xaaaad3ebfeb4 in run_mod Python/pythonrun.c:1509
    #11 0xaaaad3ec0d38 in _PyRun_File Python/pythonrun.c:1332
    #12 0xaaaad3ec4ffc in _PyRun_SimpleFile Python/pythonrun.c:544
    #13 0xaaaad3ec8dd0 in _PyRun_AnyFile Python/pythonrun.c:92
    #14 0xaaaad3f9390c in pymain_run_file_obj Modules/main.c:478
    #15 0xaaaad3f9390c in pymain_run_file Modules/main.c:494
    #16 0xaaaad3f9390c in pymain_run_python Modules/main.c:804
    #17 0xaaaad3f9390c in Py_RunMain Modules/main.c:891
    #18 0xaaaad3f95c38 in pymain_main Modules/main.c:921
    #19 0xaaaad3f95c38 in Py_BytesMain Modules/main.c:945
    #20 0xffff82c484c0  (/lib/aarch64-linux-gnu/libc.so.6+0x284c0) (BuildId: 27027b96e5b8c475fc327aa445bea1c71d37b4e2)
    #21 0xffff82c48594 in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x28594) (BuildId: 27027b96e5b8c475fc327aa445bea1c71d37b4e2)
    #22 0xaaaad343beec in _start (/src/cpython/python+0x183beec) (BuildId: eba909a3db0adabacf151d7dc3081dda7999191a)

SUMMARY: AddressSanitizer: heap-buffer-overflow Modules/cjkcodecs/_codecs_cn.c:417 in hz_decode
Shadow bytes around the buggy address:
  0x502000003e00: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x502000003e80: fa fa fd fa fa fa fd fd fa fa fd fa fa fa fd fa
  0x502000003f00: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x502000003f80: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x502000004000: fa fa fd fa fa fa fd fd fa fa fd fa fa fa fd fa
=>0x502000004080: fa fa fd fa fa fa fd fa fa fa[01]fa fa fa fa fa
  0x502000004100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000004180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000004200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000004280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000004300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1==ABORTING

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-157325: Fix an OOB read in the hz incremental decoder on a trailing ~ #157333

[picnixz]

@djarfluka We appreciate bug reports, but please could you improve the title of your issues in the future?