Skip to content

Read and write text files as UTF-8 in tests - #7764

Merged
albertvillanova merged 1 commit into
mainfrom
explicit-utf8-text-io-tests
Oct 19, 2026
Merged

albertvillanova merged 1 commit into
mainfrom
explicit-utf8-text-io-tests

Conversation

@albertvillanova

@albertvillanova albertvillanova commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

This PR makes the tests read and write every text file as UTF-8, instead of relying on the platform's default encoding.

Follow-up to #7755, which did the same for the library, examples and scripts.

Motivation

Python's default text encoding depends on the platform until Python 3.15 (PEP 686): on Windows it is the ANSI code page (e.g. cp1252), and on Linux it follows the locale. test_examples_index.py reads docs/source/example_overview.md, which contains non-ASCII characters (…, 🤗), so it depends on that default. This already fails on Linux CI: torch 2.6.0 ships NVRTC 12.4, which can leave an xdist worker's locale as C (#7720), and the read then raises UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2. The failure depends on which xdist worker the test lands on, so it is not deterministic. It failed in the minimum-versions job of #7741 (https://github.com/huggingface/trl/actions/runs/38909894519/job/117002780666) and earlier of #7642. The other reads and writes in the tests handle ASCII content today, and are changed for consistency with #7755.

Changes

  • Pass encoding="utf-8" to the text reads and writes in the tests: example index, CLI and config parsing, callbacks, SFT and Reward chat template checks, async GRPO and async distillation rollout state, Harbor task files, and the invariant trajectories

Note

Low Risk
Test-only encoding annotations with no runtime or library behavior changes.

Overview
Tests now explicitly use UTF-8 for text file I/O instead of the platform default encoding, matching the library change in #7755.

open(), Path.read_text(), Path.write_text(), and NamedTemporaryFile calls across the test suite pass encoding="utf-8" when reading or writing JSON, YAML, markdown, chat templates, Harbor task fixtures, rollout checkpoint state, invariant trajectories, and W&B completion logs. No production code changes—behavior is unchanged on typical Linux UTF-8 setups; on Windows or non-UTF-8 locales, tests that touch non-ASCII docs (e.g. example_overview.md) become reliable.

Reviewed by Cursor Bugbot for commit afb7347. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions github-actions Bot added the 🔧 maintenance CI, tooling, tests, or repo upkeep label Oct 16, 2026
@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@qgallouedec qgallouedec left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@albertvillanova
albertvillanova merged commit c979641 into main Oct 19, 2026
12 checks passed
@albertvillanova
albertvillanova deleted the explicit-utf8-text-io-tests branch October 19, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🔧 maintenance CI, tooling, tests, or repo upkeep

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants