feat(sms): read message text after undoing disguises

Homoglyphs and invisible characters survive SMS and RCS delivery and
hide spam words from rules that read raw text (one Android filter fell
from 89.6% to 41.2% accuracy in arXiv 2609.01171). SmsTextNormalizer
applies NFKC, drops format characters (zero-width, bidi, soft hyphen,
tags), variation selectors and blank-looking letters, and folds
Cyrillic, Greek and IPA letters that render as Latin.

- Folding applies to the whole message when it reads as Latin (Latin
  outnumbers Cyrillic and Greek, or every such letter has a Latin twin)
  and otherwise only inside words that mix scripts, so Russian and
  Greek messages keep their text.
- URL and host checks read the unfolded form, so a Cyrillic look-alike
  host is never scored or reported as the real domain; invisible
  characters no longer split a host out of the URL pattern.
- A URL host or brand name that hid characters or used look-alikes adds
  disguised_text (+30), labeled in en and zh.
- User keyword rules match the text as written or folded on both sides,
  keeping their case rules; the checker folds the message once.
- Invisible padding no longer stretches a short message with a link.

Tests: SmsTextNormalizerTest (JVM) and SmsTextNormalizerIcuTest
(Robolectric) share one case table; the ICU test also compares the JDK's
NFKC with ICU's Normalizer2 and the format-character set with
UCharacter over every code point both know. SmsEvaluationCorpusTest
gains zero-width, look-alike, fullwidth and hidden-in-link copies of
every flagged spam example (all stay flagged; run against the previous
analyzer, dozens escaped), and pins false alarms at none, the count
before this change. No existing assertion changed. The corpus README
paragraph said seven locales; it has eight.

Author: SysAdminDoc

CHANGELOG.md

@@ -53,6 +53,12 @@ All notable changes to CallShield will be documented in this file.
   the community report waits until that's over, so an undone block is never
   reported. Restore with Replace keeps a copy of the sections it's about to
   clear, and Undo next to the result puts them back for 30 seconds.
+- Text rules see through disguised spam. Fullwidth and math-style letters,
+  zero-width and other invisible characters, and Cyrillic or Greek letters
+  that look Latin are undone before the SMS content checks and your keyword
+  rules run. A link or brand name with hidden or look-alike characters in it
+  counts against the message. Russian and Greek messages are read as written,
+  and a look-alike web address is never mistaken for the real one.
 
 ## v1.10.0 (2026-09-28)
 

README.md

@@ -12,7 +12,7 @@
 <p align="center">
   <a href="https://github.com/SysAdminDoc/CallShield/releases/latest"><img src="https://img.shields.io/github/v/release/SysAdminDoc/CallShield?style=flat-square&color=a6e3a1" alt="Release"></a>
   <img src="https://img.shields.io/badge/Spam%20Numbers-51%2C363-f38ba8?style=flat-square" alt="51,363 Numbers">
-  <img src="https://img.shields.io/badge/JVM%20unit%20tests-1675-94e2d5?style=flat-square" alt="1675 JVM unit tests">
+  <img src="https://img.shields.io/badge/JVM%20unit%20tests-1688-94e2d5?style=flat-square" alt="1688 JVM unit tests">
   <img src="https://img.shields.io/badge/Android-10%2B-89b4fa?style=flat-square" alt="Android 10+">
   <img src="https://img.shields.io/badge/License-MIT-cba6f7?style=flat-square" alt="MIT License">
   <img src="https://img.shields.io/badge/API%20Keys-None-fab387?style=flat-square" alt="No required API keys">
@@ -140,7 +140,7 @@ All detection layers implement a shared `IChecker` interface and run in priority
 |  2000 | **ML Spam Scorer** | Block | 20-feature on-device gradient-boosted tree model |
 |  1500 | **Meeting Mode** | Silence | Opt-in. While a meeting app you pick shows a call in progress, calls from outside your contacts go quietly to voicemail. Runs on notification access, with no calendar permission |
 
-SMS-specific layers run after the shared chain, in their own priority order: **SMS Keyword Rules** (5400, with schedules) → **Carrier Scam Label** (4750, a sender your carrier renamed to its scam label, such as Singapore's Likely-SCAM) → **SMS Context Trust** (4700, trusted-sender allow) → **SMS Burst Protection** (4650) → **SMS Content Analysis** (1900, with 30+ regex patterns, URL shorteners, suspicious TLDs and the spam-domain list).
+SMS-specific layers run after the shared chain, in their own priority order: **SMS Keyword Rules** (5400, with schedules) → **Carrier Scam Label** (4750, a sender your carrier renamed to its scam label, such as Singapore's Likely-SCAM) → **SMS Context Trust** (4700, trusted-sender allow) → **SMS Burst Protection** (4650) → **SMS Content Analysis** (1900, with 30+ regex patterns, URL shorteners, suspicious TLDs and the spam-domain list). Both SMS rule layers read the text after fullwidth letters, invisible characters and Cyrillic or Greek letters posing as Latin are undone, and a link or brand name hiding such characters counts against the message.
 
 > **A flagged text still reaches your inbox.** Calls are different from
 > messages here, and the table above is about verdicts, not delivery. CallShield
@@ -214,11 +214,7 @@ An on-device **gradient-boosted tree** of 50 trees over 20 features, with a logi
 
 The maintainer trains it by hand from the CallShield database (50K positive and 50K negative samples). The scorer uses the threshold stored in the weights file, which is 0.648 for the model that ships today. A quality gate scores the 20% of rows training held out, with the same inference the app runs, and fails when F1 drops below 0.45.
 
-SMS content regressions use a separate CC0, CallShield-authored synthetic
-corpus covering seven locales, sender forms, link classes, legitimate messages,
-and hard negatives. The test reports precision, recall, and false-positive rate
-by locale and message type without shipping personal data:
-`./gradlew :app:testDebugUnitTest --tests com.sysadmindoc.callshield.data.SmsEvaluationCorpusTest`.
+SMS content regressions use a separate CC0 corpus that CallShield wrote itself. It covers eight languages, the three sender forms, each kind of link, legitimate messages and hard negatives, and the test reports precision, recall and false-positive rate by language and message type without shipping personal data. Every spam message it catches is checked again in disguise (zero-width characters between letters, Cyrillic look-alike letters, fullwidth text, a hidden character in its link) and each copy has to be caught too: `./gradlew :app:testDebugUnitTest --tests com.sysadmindoc.callshield.data.SmsEvaluationCorpusTest`.
 
 ## Features
 
@@ -509,12 +505,12 @@ RELEASE_KEY_PASSWORD=...
 ## Testing
 
 ```bash
-./gradlew testDebugUnitTest   # 1675 tests
+./gradlew testDebugUnitTest   # 1688 tests
 ./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.sysadmindoc.callshield.platform.TargetSdkBehaviorSmokeTest
 ./gradlew verifyPipelineTests # Cloudflare Worker (node) + data-pipeline and translation checks (python)
 ```
 
-The suite is **1675 total JVM unit tests**, run with Robolectric wherever a screen, a service or the database is involved.
+The suite is **1688 total JVM unit tests**, run with Robolectric wherever a screen, a service or the database is involved.
 
 Two GitHub workflows run without building the app. **Validation** runs the Worker and
 pipeline suites on every push except report-only ones (`run-pipeline-tests.ps1 -CorrectnessOnly`),
@@ -551,8 +547,8 @@ language in [issue #7](https://github.com/SysAdminDoc/CallShield/issues/7).
 | Community API | Cloudflare Workers |
 | URL Safety | Local spam-domain data, with optional PhishTank and OpenPhish |
 | Verification | Local Gradle, lint, and release-artifact checks |
-| Tests | 1675 JVM unit tests (JUnit) |
-| Strings | 1715 string resources and 37 plural groups (translation-ready) |
+| Tests | 1688 JVM unit tests (JUnit) |
+| Strings | 1716 string resources and 37 plural groups (translation-ready) |
 | Accessibility | 100+ content descriptions, 48dp touch targets |
 | Min SDK | 29 (Android 10) |
 | Target SDK | 36 |

app/src/main/java/com/sysadmindoc/callshield/data/SmsContentAnalyzer.kt

@@ -301,7 +301,8 @@ class SmsContentAnalyzer
             val domains = linkedSetOf<String>()
             val indicators = linkedSetOf<String>()
 
-            urlPattern.findAll(analysisBody).forEach { match ->
+            // Invisible characters would split a host and hide it from the pattern.
+            urlPattern.findAll(SmsTextNormalizer.normalize(analysisBody).visible).forEach { match ->
                 val url = match.value.lowercase()
                 val domain = normalizeDomainCandidate(extractDomain(url))
                 indicators.add("url_present")
@@ -352,11 +353,17 @@ class SmsContentAnalyzer
                 } else {
                     body
                 }
+            // Hosts are read without invisible characters but keep their own
+            // letters, so a Cyrillic look-alike of a real domain doesn't pass
+            // for it. Every other rule reads look-alikes as Latin.
+            val normalized = SmsTextNormalizer.normalize(analysisBody)
+            val visibleText = normalized.visible
+            val ruleText = normalized.folded
 
             // Check for URL shorteners (high spam signal)
             val urls =
-                if ('.' in analysisBody || analysisBody.contains("http", ignoreCase = true)) {
-                    urlPattern.findAll(analysisBody).map { it.value.lowercase() }.toList()
+                if ('.' in visibleText || visibleText.contains("http", ignoreCase = true)) {
+                    urlPattern.findAll(visibleText).map { it.value.lowercase() }.toList()
                 } else {
                     emptyList()
                 }
@@ -372,10 +379,15 @@ class SmsContentAnalyzer
                 }
             }
 
+            if (normalized.disguised) {
+                score += 30
+                reasons.add("disguised_text")
+            }
+
             // Check spam keyword patterns
             var patternHits = 0
             for (pattern in spamPatterns) {
-                if (pattern.containsMatchIn(analysisBody)) {
+                if (pattern.containsMatchIn(ruleText)) {
                     patternHits++
                     if (patternHits == 1) {
                         score += 25
@@ -388,32 +400,31 @@ class SmsContentAnalyzer
             }
 
             // All caps text (>50% of message) — shouting is a spam signal
-            val alphaChars = analysisBody.filter { it.isLetter() }
+            val alphaChars = ruleText.filter { it.isLetter() }
             if (alphaChars.length > 10 && alphaChars.count { it.isUpperCase() }.toFloat() / alphaChars.length > 0.5f) {
                 score += 15
                 reasons.add("excessive_caps")
             }
 
             // Contains phone number in body (callback scam) — regex is
             // case-insensitive so we match the original body instead of lowercasing.
-            if (phoneInBody.containsMatchIn(analysisBody)) {
+            if (phoneInBody.containsMatchIn(ruleText)) {
                 score += 10
                 reasons.add("callback_number")
             }
 
             // Excessive special characters / emoji (common in spam)
             val specialRatio =
-                analysisBody.count { !it.isLetterOrDigit() && !it.isWhitespace() }.toFloat() /
-                    analysisBody.length.coerceAtLeast(1)
-            if (specialRatio > 0.15f && analysisBody.length > 20) {
+                ruleText.count { !it.isLetterOrDigit() && !it.isWhitespace() }.toFloat() /
+                    ruleText.length.coerceAtLeast(1)
+            if (specialRatio > 0.15f && ruleText.length > 20) {
                 score += 10
                 reasons.add("special_chars")
             }
 
-            // Very short message with URL (likely phishing).
-            // Uses original body length so a 20-char SMS still triggers when
-            // [analysisBody] hasn't been truncated.
-            if (body.length < 50 && urls.isNotEmpty()) {
+            // Very short message with URL (likely phishing). Counted without
+            // invisible characters, which would otherwise pad it past the limit.
+            if (visibleText.length < 50 && urls.isNotEmpty()) {
                 score += 20
                 reasons.add("short_msg_with_url")
             }