Skip to content

build: generate OpenCC data with a host CMake build - #2235

Merged
Bambooin merged 1 commit into
osfans:developfrom
WhiredPlanck:opencc-data
Nov 20, 2026
Merged

Bambooin merged 1 commit into
osfans:developfrom
WhiredPlanck:opencc-data

Conversation

@WhiredPlanck

Copy link
Copy Markdown
Collaborator

Pull request

Issue tracker

Fixes #2225

Feature

OpenCCDataPlugin used to reimplement OpenCC's data/CMakeLists.txt in Kotlin:
glob data/config/*.json, scrape the .ocd2 names out of them with a regex, copy
data/dictionary/*.txt, and re-implement upstream's Python generators
(reverse.py, extract_tofu_risk.py) for the derived dictionaries. Every OpenCC
bump meant re-checking that copy, and one dictionary could not be produced this
way at all: STPhrases_GeneratedFromRegionalPhrases needs a runnable opencc
CLI, so s2t, s2tw, s2twp, s2hk and s2hkp were broken at runtime.

This replaces the copy with a host build of the bundled OpenCC source, the same
approach fcitx5-android's prebuilder uses:

  1. buildHostOpenCCData configures a private tree in app/build/opencc/host and
    builds opencc_dict plus the Dictionaries target, which pulls in opencc
    for the dictionaries derived from regional phrases.
  2. cmake -P host/data/cmake_install.cmake installs dictionaries and configs
    only into app/build/opencc/host-prefix. No upstream patch, no
    --component data, and none of the command line tools.
  3. installOpenCCData syncs that prefix into app/src/main/assets/shared/opencc,
    then verifies that every "file" reference of the installed configs resolves,
    so a change in upstream's install rules fails loudly instead of shipping a
    config that OpenCC cannot load.

The assets now carry .ocd2 instead of .txt, which also removes the 21
dictionary compilations OpenCCDictManager did on the first deploy.

Along the way this fixes a latent cross-compile bug:
add_subdirectory(librime/deps/opencc) made the Android build tree build OpenCC's
data/ target, whose commands run the freshly built aarch64 opencc_dict on the
build host (Exec format error). That directory is now EXCLUDE_FROM_ALL.

Verification

  • 38 files installed: 22 .ocd2 + 16 .json, 1.4 MB, including
    STPhrases_GeneratedFromRegionalPhrases.ocd2.
  • Cold host build 11–20 s, no-op run under 1 s; rebuilding after a dictionary
    change only regenerates the affected .ocd2.
  • Output is reproducible across the toolchains that matter: byte-identical with
    gcc 16 and clang 23, and with Ninja and Unix Makefiles (22/22 identical).
  • opencc -c s2twp.json: 内存→記憶體, 方便面→泡麵, 打印机→印表機; t2s.json:
    漢字轉換測試→汉字转换测试.
  • Rime deployment: assets contain no leftover .txt, and built-in dictionaries
    are no longer converted at deploy time.
  • Host tools are optional overrides (HOST_CMAKE, HOST_NINJA, HOST_PYTHON,
    or -PhostCmake= … ); CMake falls back to the SDK copy, then PATH. Ninja is
    optional and the default generator is used when it is absent, which keeps the
    build working on hosts that only have make.

Code of conduct

Code style

Build pass

  • make debug

Manually test

  • Done

Code Review

  1. No wildcards import
  2. Manual build and test pass
  3. GitHub Action CI pass
  4. At least one contributor review and approve
  5. Merged clean without conflicts
  6. PR will be merged by rebase upstream base

Daily build

Login and download artifact at https://github.com/osfans/trime/actions

Additional Info

F-Droid: the host binary is compiled from the bundled source during the build and
never reaches the APK, which matches "binary dependencies must originate from
source compilation". No download is added — marisa is vendored under
deps/marisa-0.3.1, and the Python generators only read local files. The build
server already installs g++/make/python3 for this app via sudo, so the
only optional addition would be ninja-build. Its sdkmanager 'cmake;X' pin may
differ from Versions.DEFAULT_CMAKE; the SDK lookup tolerates that by falling
back to any installed version.

APK size: assets grow from 1.32 MB (gzip 530 KB) to 1.27 MB (gzip 610 KB), about
+80 KB after compression. The first update after this change performs a full
DataDiff sync (21 deletions, 22 creations), which is expected.

Replace the hand-written copy of OpenCC's data rules with a host build of the
bundled source, so bumping the submodule no longer needs plugin changes.

buildHostOpenCCData configures a private tree, builds opencc_dict and the
Dictionaries target, installs the data into a local prefix and syncs it into the
assets. Installed configs are then verified against the dictionaries they
reference. That also produces STPhrases_GeneratedFromRegionalPhrases, which
could not be built before and left s2t, s2tw, s2twp, s2hk and s2hkp broken.

Dictionaries now ship as .ocd2, so the app no longer compiles them at deploy
time. OpenCC's data target is excluded from the cross-compile tree, where it
would try to run an Android opencc_dict on the build host.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cross-platform Gradle/CMake/OpenCC pipeline cannot be independently verified because the review snapshot lacks the vendored submodule and supported host toolchain coverage.

No open issues 🎉

What changed in this PR

This pull request replaces Kotlin-side OpenCC data generation with a host-side CMake build of the bundled OpenCC source, ensuring all dictionaries, including regional-phrase-derived dictionaries, are generated using upstream rules before being packaged into app assets.

Changes:

  • Adds configurable host CMake, Ninja, and Python tool resolution with Android SDK fallback.
  • Builds and installs .ocd2 dictionaries and configurations into assets with reference validation and synchronization.
  • Excludes OpenCC data generation from the Android native build to avoid executing target-architecture tools on the host.
File Description
build-logic/​convention/​src/​main/​kotlin/​ProjectExtensions.kt Adds Android SDK location discovery for resolving host build tools.
build-logic/​convention/​src/​main/​kotlin/​OpenCCDataPlugin.kt Implements host OpenCC configuration, dictionary generation, installation, asset synchronization, and output validation.
app/​src/​main/​jni/​CMakeLists.txt Excludes OpenCC data targets from the Android native build because dictionary generation is handled by the host build.

Review effort: Lite

@Bambooin
Bambooin merged commit 8ef9ff8 into osfans:develop Nov 20, 2026
5 checks passed
@WhiredPlanck
WhiredPlanck deleted the opencc-data branch November 21, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3.3.13版本後簡繁轉換失敗

3 participants