Skip to content

gh-160307: Complete final multibyte incremental decoding - #160629

Open
kinqsradio wants to merge 1 commit into
python:mainfrom
kinqsradio:fix/multibyte-final-decoding
Open

kinqsradio wants to merge 1 commit into
python:mainfrom
kinqsradio:fix/multibyte-final-decoding

Conversation

@kinqsradio

@kinqsradio kinqsradio commented Oct 24, 2026 •

Copy link
Copy Markdown

Fixes #160307.

Final incremental decoding could return with input still buffered when an error handler consumed only part of an incomplete sequence. For example, decoding b'\x8f\x00' as EUC-JP with surrogateescape and final=True returned '\udc8f' instead of '\udc8f\x00'. The same issue affected EUC-JIS variants and GB18030, including reads through TextIOWrapper.

Resume ordinary decoding after each final error-handler call, repeating final error handling if another incomplete fragment remains. Add regression coverage for chunk boundaries, partial custom error handlers, exception recovery, and file reads, plus a NEWS entry.

Validation on a macOS ARM64 debug build:

  • Regression tests produced 21 failing subcases before the fix and pass afterward.
  • Relevant codec and I/O suites pass: 1,812 tests run, 44 skipped.
  • ./python.exe -m test test_multibytecodec -R 3:3 passes with no leaks in measured repetitions.
  • 924,296 incremental-versus-stateless comparisons pass, covering every one- and two-byte input for the four affected codecs with surrogateescape, plus deterministic longer inputs with four error handlers.
  • make patchcheck and git diff --check pass.

AI assistance was used to investigate the issue, prepare the patch and tests, and run local validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

euc_jp and gb18030 incremental decoders drop input after an incomplete trailing sequence under surrogateescape

1 participant