Skip to content

gh-148603: Update docs now UTF-8 is default (PEP 686) #157372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hugovk wants to merge 4 commits into
base: main
Choose a base branch
from
+55 −65
Open

gh-148603: Update docs now UTF-8 is default (PEP 686) #157372

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
02ad240
Update docs now UTF-8 is default (PEP 686)
hugovk Sep 12, 2026
043b203
Sentence case for headers
hugovk Sep 12, 2026
c3986bc
Clarify for Windows with Python UTF-8 Mode disabled
hugovk Sep 13, 2026
0823961
Merge remote-tracking branch 'upstream/main' into 3.15-utf8-docs
hugovk Sep 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions Doc/builtins/functions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. XXX document all delegations to __special__ methods
.. _built-in-funcs:

Built-in Functions
Built-in functions
==================

The Python interpreter has a number of functions and types built into it that
Expand Down Expand Up @@ -1459,7 +1459,8 @@ are always available. They are listed here in alphabetical order.
already exists), ``'x'`` for exclusive creation, and ``'a'`` for appending
(which on *some* Unix systems, means that *all* writes append to the end of
the file regardless of the current seek position). In text mode, if
*encoding* is not specified the encoding used is platform-dependent:
*encoding* is not specified, UTF-8 is used by default; if
:ref:`Python UTF-8 Mode <utf8-mode>` is disabled,
:func:`locale.getencoding` is called to get the current locale encoding.
(For reading and writing raw bytes use binary mode and leave
*encoding* unspecified.) The available modes are:
Expand Down Expand Up @@ -1490,7 +1491,7 @@ are always available. They are listed here in alphabetical order.
argument) return contents as :class:`bytes` objects without any decoding. In
text mode (the default, or when ``'t'`` is included in the *mode* argument),
the contents of the file are returned as :class:`str`, the bytes having been
first decoded using a platform-dependent encoding or using the specified
first decoded using the default encoding or using the specified
*encoding* if given.

.. note::
Expand Down Expand Up @@ -1519,9 +1520,11 @@ are always available. They are listed here in alphabetical order.
described above for binary files.

*encoding* is the name of the encoding used to decode or encode the file.
This should only be used in text mode. The default encoding is platform
dependent (whatever :func:`locale.getencoding` returns), but any
:term:`text encoding` supported by Python can be used.
This should only be used in text mode. The default encoding is UTF-8;
if :ref:`Python UTF-8 Mode <utf8-mode>` is disabled, the default is
platform-dependent (whatever :func:`locale.getencoding` returns).
Any :term:`text encoding` supported by Python can be used, and
``encoding="locale"`` specifies the current locale encoding explicitly.
See the :mod:`codecs` module for the list of supported encodings.

*errors* is an optional string that specifies how encoding and decoding
Expand Down Expand Up @@ -1638,6 +1641,10 @@ are always available. They are listed here in alphabetical order.
.. versionchanged:: 3.11
The ``'U'`` mode has been removed.

.. versionchanged:: 3.15
UTF-8 is now the default encoding, instead of the
platform-dependent locale encoding (:pep:`686`).

.. function:: ord(character, /)

Return the ordinal value of a character.
Expand Down
17 changes: 8 additions & 9 deletions Doc/library/csv.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
:mod:`!csv` --- CSV File Reading and Writing
:mod:`!csv` --- CSV file reading and writing
============================================

.. module:: csv
Expand Down Expand Up @@ -42,7 +42,7 @@ using the :class:`DictReader` and :class:`DictWriter` classes.

.. _csv-contents:

Module Contents
Module contents
---------------

The :mod:`!csv` module defines the following functions:
Expand Down Expand Up @@ -451,7 +451,7 @@ The :mod:`!csv` module defines the following exception:

.. _csv-fmt-params:

Dialects and Formatting Parameters
Dialects and formatting parameters
----------------------------------

To make it easier to specify the format of input and output records, specific
Expand Down Expand Up @@ -557,7 +557,7 @@ with the specified formatting parameters replaced.

.. _reader-objects:

Reader Objects
Reader objects
--------------

Reader objects (:class:`DictReader` instances and objects returned by the
Expand Down Expand Up @@ -594,7 +594,7 @@ DictReader objects have the following public attribute:



Writer Objects
Writer objects
--------------

:class:`writer` objects (:class:`DictWriter` instances and objects returned by
Expand Down Expand Up @@ -673,17 +673,16 @@ The corresponding simplest possible writing example is::
writer.writerows(someiterable)

Since :func:`open` is used to open a CSV file for reading, the file
will by default be decoded into unicode using the system default
encoding (see :func:`locale.getencoding`). To decode a file
will by default be decoded into Unicode using UTF-8. To decode a file
Comment thread
hugovk marked this conversation as resolved.
using a different encoding, use the ``encoding`` argument of open::

import csv
with open('some.csv', newline='', encoding='utf-8') as f:
with open('some.csv', newline='', encoding='latin-1') as f:
reader = csv.reader(f)
for row in reader:
print(row)

The same applies to writing in something other than the system default
The same applies to writing in something other than the default
encoding: specify the encoding argument when opening the output file.

Registering a new dialect::
Expand Down
43 changes: 15 additions & 28 deletions Doc/library/io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,34 +115,21 @@ The raw stream API is described in detail in the docs of :class:`RawIOBase`.

.. _io-text-encoding:

Text Encoding
Text encoding
-------------

The default encoding of :class:`TextIOWrapper` and :func:`open` is
locale-specific (:func:`locale.getencoding`).

However, many developers forget to specify the encoding when opening text files
encoded in UTF-8 (e.g. JSON, TOML, Markdown, etc...) since most Unix
platforms use UTF-8 locale by default. This causes bugs because the locale
encoding is not UTF-8 for most Windows users. For example::

# May not work on Windows when non-ASCII characters in the file.
with open("README.md") as f:
long_description = f.read()

Accordingly, it is highly recommended that you specify the encoding
explicitly when opening text files. If you want to use UTF-8, pass
``encoding="utf-8"``. To use the current locale encoding,
``encoding="locale"`` is supported since Python 3.10.
The default encoding of :class:`TextIOWrapper` and :func:`open` is UTF-8.
If :ref:`Python UTF-8 Mode <utf8-mode>` is disabled, the default encoding
is locale-specific (:func:`locale.getencoding`).

.. seealso::

:ref:`utf8-mode`
Python UTF-8 Mode can be used to change the default encoding to
UTF-8 from locale-specific encoding.
Python UTF-8 Mode ignores the locale encoding and forces the use
of UTF-8.

:pep:`686`
Python 3.15 will make :ref:`utf8-mode` default.
Python 3.15 made :ref:`utf8-mode` the default.

.. _io-encoding-warning:

Expand All @@ -152,7 +139,7 @@ Opt-in EncodingWarning
.. versionadded:: 3.10
See :pep:`597` for more details.

To find where the default locale encoding is used, you can enable
To find where the default encoding is used, you can enable
the :option:`-X warn_default_encoding <-X>` command line option or set the

@vstinner vstinner Sep 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@methane: It now looks wrong to emit EncodingWarning if open() is called with no encoding argument, since it defaults to UTF-8 which is fine, no? For me, the purpose of PEP 597 "Add optional EncodingWarning" is to detect when platform dependent locale encoding is used. Am I wrong?

At least, I see one advantage of the current behavior, it helps writing new code which is not platform dependent when running on older Python versions.

@methane methane Sep 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EncodingWarning is a tool for people who want to identify all the places that will be affected by the transition to UTF-8 Mode and verify that the transition will not cause any problems.

Considering that some people may not start thinking about enabling UTF-8 Mode until Python 3.15, I think removing EncodingWarning immediately in Python 3.15 would be premature. How about removing it in Python 3.17 instead?

@vstinner vstinner Sep 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since -X warn_default_encoding is an opt-in option to emit EncodingWarning, I'm fine with emitting EncodingWarning on Python 3.15 when the encoding parameter is omitted. As I wrote, it should help writing new code on Python 3.15 which is compatible (behave as expected) on older Python versions.

:envvar:`PYTHONWARNDEFAULTENCODING` environment variable, which will
emit an :exc:`EncodingWarning` when the default encoding is used.
Expand All @@ -165,7 +152,7 @@ please consider using UTF-8 by default (i.e. ``encoding="utf-8"``) for
new APIs.


High-level Module Interface
High-level module interface
---------------------------

.. data:: DEFAULT_BUFFER_SIZE
Expand Down Expand Up @@ -315,7 +302,7 @@ ABC Inherits Stub Methods Mixin M
========================= ================== ======================== ==================================================


I/O Base Classes
I/O base classes
^^^^^^^^^^^^^^^^

.. class:: IOBase
Expand Down Expand Up @@ -660,7 +647,7 @@ I/O Base Classes
so the implementation should only access *b* during the method call.


Raw File I/O
Raw file I/O
^^^^^^^^^^^^

.. class:: FileIO(name, mode='r', closefd=True, opener=None)
Expand Down Expand Up @@ -728,7 +715,7 @@ Raw File I/O
given in the constructor.


Buffered Streams
Buffered streams
^^^^^^^^^^^^^^^^

Buffered I/O streams provide a higher-level interface to an I/O device
Expand Down Expand Up @@ -1004,8 +991,8 @@ Text I/O
:class:`TextIOBase`.

*encoding* gives the name of the encoding that the stream will be decoded or
encoded with. In :ref:`UTF-8 Mode <utf8-mode>`, this defaults to UTF-8.
Otherwise, it defaults to :func:`locale.getencoding`.
encoded with. This defaults to UTF-8; if :ref:`UTF-8 Mode <utf8-mode>` is
disabled, it defaults to :func:`locale.getencoding`.
``encoding="locale"`` can be used to specify the current locale's encoding
explicitly. See :ref:`io-text-encoding` for more information.

Expand Down Expand Up @@ -1187,7 +1174,7 @@ Text I/O
It inherits from :class:`codecs.IncrementalDecoder`.


Static Typing
Static typing
-------------

The following protocols can be used for annotating function and method
Expand Down
30 changes: 13 additions & 17 deletions Doc/tutorial/inputoutput.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _tut-io:

****************
Input and Output
Input and output
****************

There are several ways to present the output of a program; data can be printed
Expand All @@ -11,7 +11,7 @@ discuss some of the possibilities.

.. _tut-formatting:

Fancier Output Formatting
Fancier output formatting
=========================

So far we've encountered two ways of writing values: *expression statements* and
Expand Down Expand Up @@ -111,7 +111,7 @@ This syntax is easy to use, although it offers much less control for formatting.

.. _tut-f-strings:

Formatted String Literals
Formatted string literals
-------------------------

:ref:`Formatted string literals <f-strings>` (also called f-strings for
Expand Down Expand Up @@ -163,7 +163,7 @@ the reference guide for the :ref:`formatspec`.

.. _tut-string-format:

The String format() Method
The string format() method
--------------------------

Basic usage of the :meth:`str.format` method looks like this::
Expand Down Expand Up @@ -240,7 +240,7 @@ For a complete overview of string formatting with :meth:`str.format`, see
:ref:`formatstrings`.


Manual String Formatting
Manual string formatting
------------------------

Here's the same table of squares and cubes, formatted manually::
Expand Down Expand Up @@ -303,20 +303,19 @@ More information can be found in the :ref:`old-string-formatting` section.

.. _tut-files:

Reading and Writing Files
Reading and writing files
=========================

.. index::
pair: built-in function; open
pair: object; file

:func:`open` returns a :term:`file object`, and is most commonly used with
two positional arguments and one keyword argument:
``open(filename, mode, encoding=None)``
two positional arguments: ``open(filename, mode)``

::

>>> f = open('workfile', 'w', encoding="utf-8")
>>> f = open('workfile', 'w')

.. XXX str(f) is <io.TextIOWrapper object at 0x82e8dc4>

Expand All @@ -334,10 +333,7 @@ omitted.

Normally, files are opened in :dfn:`text mode`, that means, you read and write
strings from and to the file, which are encoded in a specific *encoding*.
If *encoding* is not specified, the default is platform dependent
(see :func:`open`).
Because UTF-8 is the modern de-facto standard, ``encoding="utf-8"`` is
recommended unless you know that you need to use a different encoding.
If *encoding* is not specified, the default is UTF-8 (see :func:`open`).
Appending a ``'b'`` to the mode opens the file in :dfn:`binary mode`.
Binary mode data is read and written as :class:`bytes` objects.
You can not specify *encoding* when opening file in binary mode.
Expand All @@ -356,7 +352,7 @@ after its suite finishes, even if an exception is raised at some
point. Using :keyword:`!with` is also much shorter than writing
equivalent :keyword:`try`\ -\ :keyword:`finally` blocks::

>>> with open('workfile', encoding="utf-8") as f:
>>> with open('workfile') as f:
... read_data = f.read()

>>> # We can check that the file has been automatically closed.
Expand Down Expand Up @@ -389,7 +385,7 @@ automatically fail. ::

.. _tut-filemethods:

Methods of File Objects
Methods of file objects
-----------------------

The rest of the examples in this section will assume that a file object called
Expand Down Expand Up @@ -532,8 +528,8 @@ To decode the object again, if ``f`` is a :term:`binary file` or
x = json.load(f)

.. note::
JSON files must be encoded in UTF-8. Use ``encoding="utf-8"`` when opening
JSON file as a :term:`text file` for both of reading and writing.
JSON files must be encoded in UTF-8, the default encoding for
:term:`text files <text file>`.

This simple serialization technique can handle lists and dictionaries, but
serializing arbitrary class instances in JSON requires a bit of extra effort.
Expand Down
11 changes: 6 additions & 5 deletions Doc/using/windows.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -943,7 +943,7 @@ checking for that feed.
}


Proxy Settings
Proxy settings
--------------

.. versionadded:: 26.4
Expand Down Expand Up @@ -1345,14 +1345,15 @@ UTF-8 mode
Python UTF-8 mode is now enabled by default (:pep:`686`).

Windows still uses legacy encodings for the system encoding (the ANSI Code
Page). Python uses it for the default encoding of text files (e.g.
:func:`locale.getencoding`).
Page). When the :ref:`Python UTF-8 Mode <utf8-mode>` is disabled, Python
uses the ANSI Code Page as the default encoding of text files, as
returned by :func:`locale.getencoding`.

This may cause issues because UTF-8 is widely used on the internet
and most Unix systems, including WSL (Windows Subsystem for Linux).

The :ref:`Python UTF-8 Mode <utf8-mode>`, enabled by default, can help by
changing the default text encoding to UTF-8.
The :ref:`Python UTF-8 Mode <utf8-mode>`, enabled by default, ignores the
system encoding and uses UTF-8 as the default text encoding.
When the :ref:`UTF-8 mode <utf8-mode>` is enabled, you can still use the
system encoding (the ANSI Code Page) via the "mbcs" codec.

Expand Down
Loading