Repository navigation
build: generate OpenCC data with a host CMake build - #2235
Merged
Merged
Conversation
Replace the hand-written copy of OpenCC's data rules with a host build of the bundled source, so bumping the submodule no longer needs plugin changes. buildHostOpenCCData configures a private tree, builds opencc_dict and the Dictionaries target, installs the data into a local prefix and syncs it into the assets. Installed configs are then verified against the dictionaries they reference. That also produces STPhrases_GeneratedFromRegionalPhrases, which could not be built before and left s2t, s2tw, s2twp, s2hk and s2hkp broken. Dictionaries now ship as .ocd2, so the app no longer compiles them at deploy time. OpenCC's data target is excluded from the cross-compile tree, where it would try to run an Android opencc_dict on the build host.
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-platform Gradle/CMake/OpenCC pipeline cannot be independently verified because the review snapshot lacks the vendored submodule and supported host toolchain coverage.
No open issues 🎉
What changed in this PR
This pull request replaces Kotlin-side OpenCC data generation with a host-side CMake build of the bundled OpenCC source, ensuring all dictionaries, including regional-phrase-derived dictionaries, are generated using upstream rules before being packaged into app assets.
Changes:
- Adds configurable host CMake, Ninja, and Python tool resolution with Android SDK fallback.
- Builds and installs
.ocd2dictionaries and configurations into assets with reference validation and synchronization. - Excludes OpenCC data generation from the Android native build to avoid executing target-architecture tools on the host.
| File | Description |
|---|---|
| build-logic/convention/src/main/kotlin/ProjectExtensions.kt | Adds Android SDK location discovery for resolving host build tools. |
| build-logic/convention/src/main/kotlin/OpenCCDataPlugin.kt | Implements host OpenCC configuration, dictionary generation, installation, asset synchronization, and output validation. |
| app/src/main/jni/CMakeLists.txt | Excludes OpenCC data targets from the Android native build because dictionary generation is handled by the host build. |
Review effort: Lite
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request
Issue tracker
Fixes #2225
Feature
OpenCCDataPluginused to reimplement OpenCC'sdata/CMakeLists.txtin Kotlin:glob
data/config/*.json, scrape the.ocd2names out of them with a regex, copydata/dictionary/*.txt, and re-implement upstream's Python generators(
reverse.py,extract_tofu_risk.py) for the derived dictionaries. Every OpenCCbump meant re-checking that copy, and one dictionary could not be produced this
way at all:
STPhrases_GeneratedFromRegionalPhrasesneeds a runnableopenccCLI, so s2t, s2tw, s2twp, s2hk and s2hkp were broken at runtime.
This replaces the copy with a host build of the bundled OpenCC source, the same
approach fcitx5-android's prebuilder uses:
buildHostOpenCCDataconfigures a private tree inapp/build/opencc/hostandbuilds
opencc_dictplus theDictionariestarget, which pulls inopenccfor the dictionaries derived from regional phrases.
cmake -P host/data/cmake_install.cmakeinstalls dictionaries and configsonly into
app/build/opencc/host-prefix. No upstream patch, no--component data, and none of the command line tools.installOpenCCDatasyncs that prefix intoapp/src/main/assets/shared/opencc,then verifies that every
"file"reference of the installed configs resolves,so a change in upstream's install rules fails loudly instead of shipping a
config that OpenCC cannot load.
The assets now carry
.ocd2instead of.txt, which also removes the 21dictionary compilations
OpenCCDictManagerdid on the first deploy.Along the way this fixes a latent cross-compile bug:
add_subdirectory(librime/deps/opencc)made the Android build tree build OpenCC'sdata/target, whose commands run the freshly built aarch64opencc_dicton thebuild host (
Exec format error). That directory is nowEXCLUDE_FROM_ALL.Verification
.ocd2+ 16.json, 1.4 MB, includingSTPhrases_GeneratedFromRegionalPhrases.ocd2.change only regenerates the affected
.ocd2.gcc 16 and clang 23, and with Ninja and Unix Makefiles (22/22 identical).
opencc -c s2twp.json: 内存→記憶體, 方便面→泡麵, 打印机→印表機;t2s.json:漢字轉換測試→汉字转换测试.
.txt, and built-in dictionariesare no longer converted at deploy time.
HOST_CMAKE,HOST_NINJA,HOST_PYTHON,or
-PhostCmake=… ); CMake falls back to the SDK copy, then PATH. Ninja isoptional and the default generator is used when it is absent, which keeps the
build working on hosts that only have make.
Code of conduct
Code style
make style-lintBuild pass
make debugManually test
Code Review
Daily build
Login and download artifact at https://github.com/osfans/trime/actions
Additional Info
F-Droid: the host binary is compiled from the bundled source during the build and
never reaches the APK, which matches "binary dependencies must originate from
source compilation". No download is added — marisa is vendored under
deps/marisa-0.3.1, and the Python generators only read local files. The buildserver already installs
g++/make/python3for this app viasudo, so theonly optional addition would be
ninja-build. Itssdkmanager 'cmake;X'pin maydiffer from
Versions.DEFAULT_CMAKE; the SDK lookup tolerates that by fallingback to any installed version.
APK size: assets grow from 1.32 MB (gzip 530 KB) to 1.27 MB (gzip 610 KB), about
+80 KB after compression. The first update after this change performs a full
DataDiffsync (21 deletions, 22 creations), which is expected.