Skip to content

Pin UTF-8 encoding when reading the shadow model's config.json - #3772

Closed
dltsum wants to merge 1 commit into
huggingface:mainfrom
dltsum:fix-shadow-config-utf8
Closed

dltsum wants to merge 1 commit into
huggingface:mainfrom
dltsum:fix-shadow-config-utf8

Conversation

@dltsum

@dltsum dltsum commented Sep 20, 2026

Copy link
Copy Markdown

Problem

ShadowModel._load_shadow_backbone reads the shadow checkpoint's config.json with the locale's preferred encoding:

config_file = cached_file(config.shadow_model, "config.json")
with open(config_file) as f:
    raw_config = json.load(f)

On Windows with a non-UTF-8 locale (e.g. cp936/GBK), a config containing non-ASCII content crashes:

UnicodeDecodeError: 'gbk' codec can't decode byte 0xa4 in position 52: illegal multibyte sequence

A realistic source: a locally fine-tuned checkpoint whose config.json carries a CJK/emoji path in _name_or_path and was saved with ensure_ascii=False.

Reproducer (Windows 11, cp936, Python 3.14)

Write {'model_type': 'bert', '_name_or_path': 'D:/模型🤗hub'} as UTF-8 JSON into a local checkpoint dir and call _load_shadow_backbone → UnicodeDecodeError before this change; the read succeeds after it (the call then proceeds to model loading as expected).

Fix

Pin encoding="utf-8" on the read.

🤖 Generated with Claude Code

`ShadowModel._load_shadow_backbone` read the shadow checkpoint's `config.json`
with the locale's preferred encoding. On Windows with a non-UTF-8 locale (e.g.
cp936), a config containing non-ASCII content (for example a CJK or emoji path
in `_name_or_path`, as produced by a local fine-tune saved with
`ensure_ascii=False`) crashes with `UnicodeDecodeError`.

Pin `encoding="utf-8"` on the read.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@BenjaminBossan

Copy link
Copy Markdown
Member

Please fold this PR into #3771. Similarly, if you find this kind of error elsewhere in the code, please fold it into one PR.

dltsum added a commit to dltsum/peft that referenced this pull request Sep 21, 2026
As requested in review:

- Pin UTF-8 on the shadow backbone's config.json read as well (folded
  in from huggingface#3772, which will be closed). A repo-wide scan found no other
  user-facing config reads missing the pin.
- Add a Windows-gated regression test: a config file containing literal
  UTF-8 non-ASCII values (as written by other tools or by hand) used to
  crash or decode incorrectly under a non-UTF-8 locale (e.g. cp936). The
  test fails without the fix and passes with it.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@dltsum

dltsum commented Sep 21, 2026

Copy link
Copy Markdown
Author

Folded into #3771 as commit 216bc6a (same change, plus the requested regression test).

dltsum added a commit to dltsum/peft that referenced this pull request Oct 16, 2026
As requested in review:

- Pin UTF-8 on the shadow backbone's config.json read as well (folded
  in from huggingface#3772, which will be closed). A repo-wide scan found no other
  user-facing config reads missing the pin.
- Add a Windows-gated regression test: a config file containing literal
  UTF-8 non-ASCII values (as written by other tools or by hand) used to
  crash or decode incorrectly under a non-UTF-8 locale (e.g. cp936). The
  test fails without the fix and passes with it.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants