Skip to content

Read GPX files as UTF-8 on every platform - #933

Merged
caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:read-gpx-as-utf8
Oct 27, 2026
Merged

caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:read-gpx-as-utf8

Conversation

@caglarpir

Copy link
Copy Markdown
Contributor

A GPX file was decoded in the encoding of the locale, as v0.14.7 did with open("r"): cp1252 on Western Windows, cp936 on Chinese Windows. GPX files are almost always UTF-8. There, a UTF-8 GPX with "Á", "Í", "Ð", "Ý" or most CJK characters in its names failed with UnicodeDecodeError, as their bytes are undefined in cp1252, and the user was told "X.gpx is not a valid GPX file. Replace it with a valid GPX file."; other accented letters only came out garbled. On Linux and macOS, a GPX in another encoding, such as ISO-8859-1 with "Café" in a name, failed the same way.

Decode it as UTF-8 with errors="replace", as NMEA logs are (#902), so that a byte that is not UTF-8 only turns a letter of a name into U+FFFD: positions and times are ASCII. gpxpy still reads the file opened in text mode itself (#912's memory fix), and a byte order mark is still ignored. A UTF-16 GPX still fails as MapillaryInvalidGPXError.

  • A GPX written in ISO-8859-1 from the track of a BlackVue video, with "Café" in its name, with process VIDEO --video_geotag_source gpx --geotag_source_path X.gpx --interpolation_use_gpx_start_time: ready, with the same 60 points as its UTF-8 copy. Main: MapillaryInvalidGPXError, exit 6.
  • The tests read a UTF-8 GPX with "Árbol 北京" in its name, with and without a byte order mark, as if the locale were cp1252, which failed on main; on Windows they run in its own locale too.

Compared with main on process, video_process and process_and_upload --dry_run, on images and videos of many cameras with the default and other sources, the outputs and exit codes are the same.

Unit and integration tests, doctests, mypy --python-version 3.9, ruff check, ruff format --check and usort diff are clean.

A GPX file was decoded in the encoding of the locale, as v0.14.7 did
with open("r"): cp1252 on Western Windows, cp936 on Chinese Windows.
GPX files are almost always UTF-8. There, a UTF-8 GPX with "Á", "Í",
"Ð", "Ý" or most CJK characters in its names failed with
UnicodeDecodeError, as their bytes are undefined in cp1252, and the
user was told that it is not a valid GPX file; other accented letters
only came out garbled. On Linux and macOS, a GPX in another encoding,
such as ISO-8859-1 with "Café" in a name, failed the same way.

Decode it as UTF-8 with errors="replace", as NMEA logs are, so that a
byte that is not UTF-8 only turns a letter of a name into U+FFFD:
positions and times are ASCII. gpxpy still reads the file opened in
text mode itself, and a byte order mark is still ignored. A UTF-16 GPX
still fails as not a valid GPX file.
@meta-cla meta-cla Bot added the cla signed label Oct 27, 2026
@caglarpir
caglarpir merged commit 7f823e7 into mapillary:main Oct 27, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant