Skip to content

Add paired cp1252 and UTF-8 native model controls - #34

Merged
PyYoshi merged 1 commit into
devfrom
v3/paired-encoding-controls
Sep 20, 2026
Merged

PyYoshi merged 1 commit into
devfrom
v3/paired-encoding-controls

Conversation

@PyYoshi

@PyYoshi PyYoshi commented Sep 20, 2026

Copy link
Copy Markdown
Owner

概要

  • 同じvalidation/tuning文章のcp1252/UTF-8を対にして3モデルを比較
  • ASCII等の同一bytes、cp1252へdecode可能/不能、filter後の途中終了を分離
  • 有限値のpair順序比較。正解率・false-positive rateとは扱わず、閾値調整なし
  • 日本語仕様・8テスト。既存native比較のbuild/run処理を共通化

検証

  • モデル78 tests成功(native toolsあり)、なし69成功/9skip
  • Clangで追加8 tests成功、Ruff/diff check成功
  • Paris16対・Rust book1対のreportをfresh buildから2回生成し全byte一致
  • RustのUTF-8はcp1252としてdecode不能、nativeは5138 filtered bytes中263で停止。統計的識別と分離
  • engine/API/既定model/P01は変更なし

Related: PyYoshi/cChardet#123

@PyYoshi
PyYoshi marked this pull request as ready for review September 20, 2026 21:35
@PyYoshi
PyYoshi merged commit 6e1e9d9 into dev Sep 20, 2026
11 checks passed
@PyYoshi
PyYoshi deleted the v3/paired-encoding-controls branch September 20, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant