PEP 686 makes UTF-8 the default encoding in 3.15, and the argument for it is correctness: the same program should read the same file the same way on every machine. What that argument leaves out is which direction the errors run. The behaviour being removed fails silently and the behaviour replacing it fails loudly, and an upgrade turns one into the other.
Under a Latin-1 locale, CPython 3.13 reads a UTF-8 file and returns 'caf\xc3\xa9 na\xc3\xafve' without raising anything. 3.15 reads it correctly — and raises on the Latin-1
file that 3.13 handled without complaint.
What open() picks
$ python3 experiments/evalloop/encoding.py ~/.pyenv/versions/3.15.0rc1/bin/python3
platform darwin, Latin-1 locale en_US.ISO8859-1
what open() picks
python locale utf8_mode locale.getencoding open().encoding
3.13.2 UTF-8 (this shell) 0 UTF-8 UTF-8
3.13.2 en_US.ISO8859-1 0 ISO8859-1 ISO8859-1
3.15.0rc1 UTF-8 (this shell) 1 UTF-8 utf-8
3.15.0rc1 en_US.ISO8859-1 1 ISO8859-1 utf-8
the same two files, opened with no encoding= under en_US.ISO8859-1
utf8.txt is 63 6166c3a9 206e61c3 af...
3.13.2 ISO8859-1 ok 'caf\xc3\xa9 na\xc3\xafve'
3.15.0rc1 utf-8 ok 'caf\xe9 na\xefve'
latin1.txt is 636166 e9206e61 ef76650...
3.13.2 ISO8859-1 ok 'caf\xe9 na\xefve'
3.15.0rc1 utf-8 raised 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte
read throughput, MB/s, 16 MB file, best of 5
python file binary utf-8 latin-1 ascii
3.13.2 ascii.txt 28702 11950 10446 11528
3.13.2 mixed.txt 29960 2126 15015 859
3.15.0rc1 ascii.txt 29696 10334 10322 11888
3.15.0rc1 mixed.txt 29376 2938 15099 1369
every decode uses errors='replace', so the ascii column on mixed.txt is
the error handler running once per non-ASCII byte, not a decode rateThe first block is the whole change in one row. On 3.15 under a Latin-1 locale,
locale.getencoding() still answers ISO8859-1 — the locale is real and Python knows what it
is — and open() no longer asks. The flag that decides is sys.flags.utf8_mode, and it goes
from 0 to 1 with no environment variable set.
The C locale cannot be used to show this. CPython has coerced it to UTF-8 since PEP 540, so
under LC_ALL=C every version behaves like 3.15 and the comparison disappears. That is also
why the shell’s own UTF-8 locale gives identical rows on both interpreters: on a machine that
was already UTF-8 everywhere, PEP 686 changes nothing at all.
The two files that change
utf8.txt holds café naïve encoded as UTF-8, so é is the two bytes c3 a9. Read under a
Latin-1 locale, each of those bytes becomes its own character, and the program gets
'caf\xc3\xa9 na\xc3\xafve' — twelve characters where the file held ten, and no exception. A
length check passes, a startswith passes, and the string is wrong. Latin-1
has a character for all 256 byte values, so a Latin-1 decode cannot fail. That is the property
that makes the old default dangerous: it does not misread loudly, it misreads quietly and hands
the result on.
latin1.txt holds the same text encoded as Latin-1, where é is the single byte e9. 3.13
under that locale reads it correctly. 3.15 reads it as UTF-8, finds e9 where a multi-byte
sequence should start, and raises UnicodeDecodeError at position 3.
Both rows are the same upgrade. Whether it fixes a bug or introduces one depends on which of the two files the codebase has been reading, and nothing in the code says which.
What a decode costs
The throughput block answers a question the correctness argument never raises. A 16 MB ASCII file, already in the page cache, reads at 28,702 MB/s as bytes and 11,950 MB/s as UTF-8 text. Decoding is 58% of the time the text read takes.
Put a handful of non-ASCII characters on every line and UTF-8 drops to 2,126 MB/s. The same
codec, the same file size, 5.6 times slower — the ASCII-only fast path is gone and every
multi-byte sequence is decoded individually. latin-1 sits at about 15,000 MB/s on both files
because it has no multi-byte case to leave.
None of that is a reason to avoid UTF-8. It is a reason to know that “just read the file” has a per-megabyte price that varies by a factor of six with the content, and that a program parsing gigabytes of mostly-ASCII log lines is on the fast path only until someone writes a name in it.
When to write encoding= anyway
The default is now the right default, and it is still the wrong thing to rely on. Two cases:
A file whose encoding is a fact about the file, not about the machine — a fixture written in
cp1252, an export from a system that emits Latin-1 — needs encoding= because the correct
answer is not UTF-8 and never was. The new default does not fix those; it changes which
machines they break on.
A file read on 3.15 and on an older interpreter, which is every library supporting more than one version. There the explicit argument is the only way to get one behaviour.
That is the second thing in this archive an upgrade changes about when code raises rather than how fast it runs. PEP 649 does it to annotations — the expression that ran at definition time stops running there — and PEP 686 does it to reads. Neither shows up in a benchmark.
Finding them is mechanical. -X warn_default_encoding raises an EncodingWarning reading
'encoding' argument not specified for every text-mode open() without one, and
-W error::EncodingWarning turns them into failures. PYTHONUTF8=0 restores the old behaviour
on 3.15, which is a way to ship on a deadline rather than a fix.
Where to stop
One machine, macOS on Apple silicon, 3.13.2 against 3.15.0rc1 built from source on 2 September 2026. The final release is dated 1 October and these are not its numbers.
Nothing here charges the encoding change against import cost either. -X importtime attributes
by order rather than by work, so a locale change that moves where a decode happens moves the
blame with it — which number that report is actually
answering is its own entry.
Nothing here is a Windows measurement, and Windows is where PEP 686 does most of its work. The
code page there is typically cp1252 or a regional variant, so the default differs from UTF-8
without anyone changing a locale — but that is the PEP’s claim, repeated, not this entry’s.
The throughput figures are reads of a cached file, so they measure decoding and memory rather
than storage. Every decode in that table uses errors='replace', which is why the ascii
column on the mixed file is 859 MB/s: that is an error handler running once per non-ASCII byte,
not a decode rate.
Frequently asked
Does this affect Linux and macOS at all?
Only where the locale is not already UTF-8. On the machine measured here the shell locale is UTF-8, and 3.13 and 3.15 agree exactly until the locale is changed. PEP 686 is aimed mostly at Windows, where the code page is typically cp1252 or a regional variant and the default therefore differs from UTF-8 out of the box.
Why not demonstrate this with the C locale?
Because CPython coerces it. Under LC_ALL=C the interpreter enables UTF-8 mode on its own, which it has done since PEP 540, so every version shows the new behaviour and the comparison collapses. The demonstration needs a locale that is real and not UTF-8, which is why en_US.ISO8859-1 is used here.
How do I find the calls that will change?
Run the test suite with -X warn_default_encoding. Every text-mode open() that omits encoding= raises an EncodingWarning reading "encoding argument not specified", and adding -W error::EncodingWarning turns each one into a failure you can fix. That flag has existed since 3.10, so this work can be done before the upgrade rather than during it.
Should I set PYTHONUTF8=0 and move on?
As a way to unblock a deployment, yes. As a fix, no: it re-enables the behaviour where the same code reads a file differently on two machines, which is the thing being removed. The durable version is an explicit encoding= on the calls that need one.
Is the throughput difference worth designing around?
Rarely. At 2,126 MB/s the decode of a 16 MB file is 7.5 ms, and any program reading files that large is usually doing something more expensive with them afterwards. It matters where the file is enormous and the work per line is small, and there the answer is latin-1 or binary, chosen deliberately rather than inherited from a locale.
Where this came from
Run it yourself — every figure above came out of these:
- evalloop/encoding.py — What open() picks under a Latin-1 locale, and decode throughput
Read, rather than assumed:
