Skip to content

gh-160518: Fix hz encoder dropping half of the ~~ escape - #160519

Open
gpshead wants to merge 1 commit into
python:mainfrom
gpshead:scanfix-codecscn-fxw1xykr
Open

gpshead wants to merge 1 commit into
python:mainfrom
gpshead:scanfix-codecscn-fxw1xykr

Conversation

@gpshead

@gpshead gpshead commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

hz_encode() wrote the first byte of the ~~ escape, consumed the input character, and only then wrote the second byte. When the output buffer was full at that point, multibytecodec_encode() enlarged it and resumed at the next character, so the second ~ was lost. The escape is now written with WRITEBYTE2 before NEXT(1, 2), like the other multi-byte writes in this file, so the retry starts at the tilde again.

The test sweeps the text length across the initial buffer size and its first enlargement; it fails at 4 of the 120 lengths without the fix. Backports to branches older than 3.14 would need assertEndsWith replaced, but 3.14 and 3.15 have it.

Fixes #160518

ANT-2026-FXW1XYKR

Write both bytes of the escape before consuming the input character, so
that a retry after the output buffer is enlarged starts at the tilde.
@gpshead gpshead added needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Oct 23, 2026
@gpshead gpshead self-assigned this Oct 23, 2026
@gpshead
gpshead marked this pull request as ready for review October 23, 2026 22:00
@gpshead
gpshead requested a review from corona10 as a code owner October 23, 2026 22:00
@gpshead gpshead added the sprint label Oct 23, 2026
@github-project-automation github-project-automation Bot moved this to Todo in Sprint Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

hz codec encodes ~ as a single ~ when the output buffer fills mid-escape

1 participant