# UTF-8 by Default Fixes a Silent Bug and Adds a Loud One > Under a Latin-1 locale, CPython 3.13 reads a UTF-8 file as mojibake and raises nothing. 3.15 reads it correctly and raises on the Latin-1 file 3.13 read fine. - Published: 2026-10-11 - Category: runtime - Tags: versions, benchmarking - Sources: https://github.com/CognatePress/erperiments.pydepth.com/blob/main/evalloop/encoding.py, https://peps.python.org/pep-686/, https://peps.python.org/pep-540/ - Source: https://pydepth.com/blog/utf8-by-default/ - Language: en-US - Author: Elliot Sayer --- PEP 686 makes UTF-8 the default encoding in 3.15, and the argument for it is correctness: the same program should read the same file the same way on every machine. What that argument leaves out is which direction the errors run. The behaviour being removed fails silently and the behaviour replacing it fails loudly, and an upgrade turns one into the other. Under a Latin-1 locale, CPython 3.13 reads a UTF-8 file and returns `'caf\xc3\xa9 na\xc3\xafve'` without raising anything. 3.15 reads it correctly — and raises on the Latin-1 file that 3.13 handled without complaint. ## What open() picks ``` $ python3 experiments/evalloop/encoding.py ~/.pyenv/versions/3.15.0rc1/bin/python3 platform darwin, Latin-1 locale en_US.ISO8859-1 what open() picks python locale utf8_mode locale.getencoding open().encoding 3.13.2 UTF-8 (this shell) 0 UTF-8 UTF-8 3.13.2 en_US.ISO8859-1 0 ISO8859-1 ISO8859-1 3.15.0rc1 UTF-8 (this shell) 1 UTF-8 utf-8 3.15.0rc1 en_US.ISO8859-1 1 ISO8859-1 utf-8 the same two files, opened with no encoding= under en_US.ISO8859-1 utf8.txt is 63 6166c3a9 206e61c3 af... 3.13.2 ISO8859-1 ok 'caf\xc3\xa9 na\xc3\xafve' 3.15.0rc1 utf-8 ok 'caf\xe9 na\xefve' latin1.txt is 636166 e9206e61 ef76650... 3.13.2 ISO8859-1 ok 'caf\xe9 na\xefve' 3.15.0rc1 utf-8 raised 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte read throughput, MB/s, 16 MB file, best of 5 python file binary utf-8 latin-1 ascii 3.13.2 ascii.txt 28702 11950 10446 11528 3.13.2 mixed.txt 29960 2126 15015 859 3.15.0rc1 ascii.txt 29696 10334 10322 11888 3.15.0rc1 mixed.txt 29376 2938 15099 1369 every decode uses errors='replace', so the ascii column on mixed.txt is the error handler running once per non-ASCII byte, not a decode rate ``` The first block is the whole change in one row. On 3.15 under a Latin-1 locale, `locale.getencoding()` still answers `ISO8859-1` — the locale is real and Python knows what it is — and `open()` no longer asks. The flag that decides is `sys.flags.utf8_mode`, and it goes from 0 to 1 with no environment variable set. The C locale cannot be used to show this. CPython has coerced it to UTF-8 since PEP 540, so under `LC_ALL=C` every version behaves like 3.15 and the comparison disappears. That is also why the shell's own UTF-8 locale gives identical rows on both interpreters: on a machine that was already UTF-8 everywhere, PEP 686 changes nothing at all. ## The two files that change `utf8.txt` holds `café naïve` encoded as UTF-8, so `é` is the two bytes `c3 a9`. Read under a Latin-1 locale, each of those bytes becomes its own character, and the program gets `'caf\xc3\xa9 na\xc3\xafve'` — twelve characters where the file held ten, and no exception. A length check passes, a `startswith` passes, and the string is wrong. Latin-1 has a character for all 256 byte values, so a Latin-1 decode cannot fail. That is the property that makes the old default dangerous: it does not misread loudly, it misreads quietly and hands the result on. `latin1.txt` holds the same text encoded as Latin-1, where `é` is the single byte `e9`. 3.13 under that locale reads it correctly. 3.15 reads it as UTF-8, finds `e9` where a multi-byte sequence should start, and raises `UnicodeDecodeError` at position 3. Both rows are the same upgrade. Whether it fixes a bug or introduces one depends on which of the two files the codebase has been reading, and nothing in the code says which. ## What a decode costs The throughput block answers a question the correctness argument never raises. A 16 MB ASCII file, already in the page cache, reads at 28,702 MB/s as bytes and 11,950 MB/s as UTF-8 text. Decoding is 58% of the time the text read takes. Put a handful of non-ASCII characters on every line and UTF-8 drops to 2,126 MB/s. The same codec, the same file size, 5.6 times slower — the ASCII-only fast path is gone and every multi-byte sequence is decoded individually. `latin-1` sits at about 15,000 MB/s on both files because it has no multi-byte case to leave. None of that is a reason to avoid UTF-8. It is a reason to know that "just read the file" has a per-megabyte price that varies by a factor of six with the content, and that a program parsing gigabytes of mostly-ASCII log lines is on the fast path only until someone writes a name in it. ## When to write encoding= anyway The default is now the right default, and it is still the wrong thing to rely on. Two cases: A file whose encoding is a fact about the file, not about the machine — a fixture written in `cp1252`, an export from a system that emits Latin-1 — needs `encoding=` because the correct answer is not UTF-8 and never was. The new default does not fix those; it changes which machines they break on. A file read on 3.15 and on an older interpreter, which is every library supporting more than one version. There the explicit argument is the only way to get one behaviour. That is the second thing in this archive an upgrade changes about *when* code raises rather than how fast it runs. PEP 649 does it to annotations — the expression that ran at definition time [stops running there](/blog/type-hints-at-runtime/) — and PEP 686 does it to reads. Neither shows up in a benchmark. Finding them is mechanical. `-X warn_default_encoding` raises an `EncodingWarning` reading `'encoding' argument not specified` for every text-mode `open()` without one, and `-W error::EncodingWarning` turns them into failures. `PYTHONUTF8=0` restores the old behaviour on 3.15, which is a way to ship on a deadline rather than a fix. ## Where to stop One machine, macOS on Apple silicon, 3.13.2 against 3.15.0rc1 built from source on 2 September 2026. The final release is dated 1 October and these are not its numbers. Nothing here charges the encoding change against import cost either. `-X importtime` attributes by order rather than by work, so a locale change that moves where a decode happens moves the blame with it — [which number that report is actually answering](/blog/import-time-is-not-import-cost/) is its own entry. Nothing here is a Windows measurement, and Windows is where PEP 686 does most of its work. The code page there is typically `cp1252` or a regional variant, so the default differs from UTF-8 without anyone changing a locale — but that is the PEP's claim, repeated, not this entry's. The throughput figures are reads of a cached file, so they measure decoding and memory rather than storage. Every decode in that table uses `errors='replace'`, which is why the `ascii` column on the mixed file is 859 MB/s: that is an error handler running once per non-ASCII byte, not a decode rate.