Repository navigation
Conversation
`ShadowModel._load_shadow_backbone` read the shadow checkpoint's `config.json` with the locale's preferred encoding. On Windows with a non-UTF-8 locale (e.g. cp936), a config containing non-ASCII content (for example a CJK or emoji path in `_name_or_path`, as produced by a local fine-tune saved with `ensure_ascii=False`) crashes with `UnicodeDecodeError`. Pin `encoding="utf-8"` on the read. Co-Authored-By: Claude Code <noreply@anthropic.com>
Member
|
Please fold this PR into #3771. Similarly, if you find this kind of error elsewhere in the code, please fold it into one PR. |
dltsum
added a commit
to dltsum/peft
that referenced
this pull request
Sep 21, 2026
As requested in review: - Pin UTF-8 on the shadow backbone's config.json read as well (folded in from huggingface#3772, which will be closed). A repo-wide scan found no other user-facing config reads missing the pin. - Add a Windows-gated regression test: a config file containing literal UTF-8 non-ASCII values (as written by other tools or by hand) used to crash or decode incorrectly under a non-UTF-8 locale (e.g. cp936). The test fails without the fix and passes with it. Co-Authored-By: Claude Code <noreply@anthropic.com>
Author
dltsum
added a commit
to dltsum/peft
that referenced
this pull request
Oct 16, 2026
As requested in review: - Pin UTF-8 on the shadow backbone's config.json read as well (folded in from huggingface#3772, which will be closed). A repo-wide scan found no other user-facing config reads missing the pin. - Add a Windows-gated regression test: a config file containing literal UTF-8 non-ASCII values (as written by other tools or by hand) used to crash or decode incorrectly under a non-UTF-8 locale (e.g. cp936). The test fails without the fix and passes with it. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ShadowModel._load_shadow_backbonereads the shadow checkpoint'sconfig.jsonwith the locale's preferred encoding:On Windows with a non-UTF-8 locale (e.g. cp936/GBK), a config containing non-ASCII content crashes:
A realistic source: a locally fine-tuned checkpoint whose
config.jsoncarries a CJK/emoji path in_name_or_pathand was saved withensure_ascii=False.Reproducer (Windows 11, cp936, Python 3.14)
Write
{'model_type': 'bert', '_name_or_path': 'D:/模型🤗hub'}as UTF-8 JSON into a local checkpoint dir and call_load_shadow_backbone→UnicodeDecodeErrorbefore this change; the read succeeds after it (the call then proceeds to model loading as expected).Fix
Pin
encoding="utf-8"on the read.🤖 Generated with Claude Code