Skip to content

Commit 13fbfa2

Browse files
committed
Fix the named-sender rule: no regex blow-up, no lost link lures
A review of 9d77391 found three faults. The "this is Dana from" pattern let spaces sit inside the name class and around it, so a text of "this is" plus 16,000 spaces took 72 seconds to score, against the screener's 5-second budget. Each name word is now its own run of letters, so spaces can't be split more than one way. The brand-prefix pattern took up to 39 characters of anything before the first colon. "Are you free Saturday at 3:30? Reply STOP" read as a brand, and so did a mid-text https: link, tel:, "Valued customer:", "Alert:", "Hiya:" and "Quick question:". The colon must now be followed by a space, the name must be at most five capitalized words (digits and connectors like "of" and "&" allowed), and its first word can't be a greeting or a notice word. "7-Eleven:", "24 Hour Fitness:" and "[Lakeside Clinic]" count as names now; they didn't before. The footer was skipped whenever no keyword pattern hit, even when a link, shouting, a callback number or reply bait had already scored. A branded smish with a suspicious link dropped from 70 to 30, under the default bar. The footer now counts next to any other signal, so the only texts that score differently from v1.10.0 are named senders whose footer is their only signal. The signals are worked out before the pattern block and scored in the same order as before, so reasons keep their order. IMC 2025 recall: all 0.1206 (was 0.1204), en 0.1611 (was 0.1603), precision 1.0. Tests: four new SmsContentAnalyzerTest cases (any other signal brings the footer back, a time/link/notice word isn't a name, digit/connector/bracket names are, and a timing guard on long runs of spaces). No existing test changed. Ten mutants of the new rules were all killed.
1 parent 2448d31 commit 13fbfa2

4 files changed

Lines changed: 144 additions & 28 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,13 @@ All notable changes to CallShield will be documented in this file.
1818
footer alone. "Reply STOP to opt out" scored 40 against aggressive mode's
1919
bar of 25, so a clinic's appointment reminder or a pharmacy's refill notice
2020
was blocked on its last line. Answer and opt-out wording now counts only
21-
next to a real spam pattern, or in a text that doesn't say who sent it. The
22-
carrier rules make a business open with its name ("Lakeside Family Clinic:
23-
..." or "this is Dana from Maple Street Dental"), while scams paste the
24-
footer onto a chatty opener that never says. Recall on the IMC 2025
25-
smishing sample stays at its floors.
21+
next to another spam signal (a lure, a suspicious link, shouting or a
22+
callback number), or in a text that doesn't say who sent it. The carrier
23+
rules make a business open with its name ("Lakeside Family Clinic: ...",
24+
"[Corner Market] ..." or "this is Dana from Maple Street Dental"), while
25+
scams paste the footer onto a chatty opener that never says. A time like
26+
3:30, a link, a greeting or a word like Alert in front of the colon isn't
27+
a name. Recall on the IMC 2025 smishing sample went up slightly.
2628
- A number the maintainer approved can now be taken back. Its entry in
2729
`spam_numbers_approved.json` gets `"revoked": true` and a date, and the next
2830
merge strips the review from the row. A row the review alone was holding up

‎README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
<p align="center">
1313
<a href="https://github.com/SysAdminDoc/CallShield/releases/latest"><img src="https://img.shields.io/github/v/release/SysAdminDoc/CallShield?style=flat-square&color=a6e3a1" alt="Release"></a>
1414
<img src="https://img.shields.io/badge/Spam%20Numbers-51%2C363-f38ba8?style=flat-square" alt="51,363 Numbers">
15-
<img src="https://img.shields.io/badge/JVM%20unit%20tests-1827-94e2d5?style=flat-square" alt="1827 JVM unit tests">
15+
<img src="https://img.shields.io/badge/JVM%20unit%20tests-1831-94e2d5?style=flat-square" alt="1831 JVM unit tests">
1616
<img src="https://img.shields.io/badge/Android-10%2B-89b4fa?style=flat-square" alt="Android 10+">
1717
<img src="https://img.shields.io/badge/License-MIT-cba6f7?style=flat-square" alt="MIT License">
1818
<img src="https://img.shields.io/badge/API%20Keys-None-fab387?style=flat-square" alt="No required API keys">
@@ -519,12 +519,12 @@ RELEASE_KEY_PASSWORD=...
519519
## Testing
520520

521521
```bash
522-
./gradlew testDebugUnitTest # 1827 tests
522+
./gradlew testDebugUnitTest # 1831 tests
523523
./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.sysadmindoc.callshield.platform.TargetSdkBehaviorSmokeTest
524524
./gradlew verifyPipelineTests # Cloudflare Worker (node) + data-pipeline and translation checks (python)
525525
```
526526

527-
The suite is **1827 total JVM unit tests**, run with Robolectric wherever a screen, a service or the database is involved.
527+
The suite is **1831 total JVM unit tests**, run with Robolectric wherever a screen, a service or the database is involved.
528528

529529
Two GitHub workflows run without building the app. **Validation** runs the Worker and
530530
pipeline suites on every push except report-only ones (`run-pipeline-tests.ps1 -CorrectnessOnly`),
@@ -561,7 +561,7 @@ language in [issue #7](https://github.com/SysAdminDoc/CallShield/issues/7).
561561
| Community API | Cloudflare Workers |
562562
| URL Safety | Local spam-domain data, with optional PhishTank and OpenPhish |
563563
| Verification | Local Gradle, lint, and release-artifact checks |
564-
| Tests | 1827 JVM unit tests (JUnit) |
564+
| Tests | 1831 JVM unit tests (JUnit) |
565565
| Strings | 1738 string resources and 37 plural groups (translation-ready) |
566566
| Accessibility | 100+ content descriptions, 48dp touch targets |
567567
| Min SDK | 29 (Android 10) |

‎app/src/main/java/com/sysadmindoc/callshield/data/SmsContentAnalyzer.kt‎

Lines changed: 62 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -157,8 +157,8 @@ class SmsContentAnalyzer
157157
// confirm"), and alone they blocked a clinic's appointment reminder in
158158
// aggressive mode. Scams copy the footer onto a chatty opener too
159159
// ("Hey Sam, are you free for coffee? Reply STOP to unsubscribe"), so
160-
// they still count after a pattern above, or when the text doesn't say
161-
// who sent it (see [namedSender]).
160+
// they still count next to any other signal, or when the text doesn't
161+
// say who sent it (see [namesItsSender]).
162162
private val responseCues =
163163
listOf(
164164
Regex("(?i)(unsubscribe|opt.?out|stop to (end|cancel|quit|unsubscribe))"),
@@ -167,13 +167,47 @@ class SmsContentAnalyzer
167167
)
168168

169169
// The carrier rules also make a business say who it is first:
170-
// "Lakeside Family Clinic: your visit is..." or "this is Dana from
171-
// Maple Street Dental". A greeting or a link before the colon isn't a name.
172-
private val namedSender =
173-
Regex(
174-
"^[^\\p{L}\\p{N}]*(?!(?i:hi|hey|hello|dear|good|https?)(?![\\p{L}\\p{N}]))\\p{L}[\\p{L}\\p{N}&'’. -]{0,38}:" +
175-
"|(?i:this is)$SPACE+[\\p{L}'’. ]{1,30}$SPACE+(?i:from|at|with)$SPACE+\\p{Lu}",
176-
)
170+
// "Lakeside Family Clinic: your visit is...", "[Corner Market] your
171+
// order..." or "this is Dana from Maple Street Dental". The colon has to
172+
// be followed by a space, so "3:30", "https://" and "tel:" aren't names.
173+
private val senderPrefix =
174+
Regex("^[^\\p{L}\\p{N}\\[]*(?:\\[([^\\]\\n]{1,40})]|([\\p{L}\\p{N}][\\p{L}\\p{N}&'’.\\- ]{0,39}):(?:$SPACE|\$))")
175+
176+
// Each name word is its own run of letters, never spaces, so a long run
177+
// of spaces after "this is" can't make the engine try every split.
178+
private val senderIntroduction =
179+
Regex("(?i:\\bthis is)$SPACE+(?:\\p{L}[\\p{L}'’.-]*$SPACE+){1,3}?(?i:from|at|with)$SPACE+\\p{Lu}")
180+
181+
private val nameWordSeparator = Regex("$SPACE+")
182+
183+
// Openers that start a notice or a greeting, not a business's name:
184+
// "Alert:", "Valued customer:", "Hi Sam:", "Quick question:".
185+
private val notANameOpener =
186+
(
187+
"hi hey hello hiya howdy yo greetings dear good valued customer member user client friend sir madam " +
188+
"alert notice warning urgent important attention final reminder update info notification action " +
189+
"security account message msg re fwd fw your our my the a congrats congratulations quick question " +
190+
"note ps http https www tel sms"
191+
).split(' ').toSet()
192+
193+
// Lowercase words a business name can carry: "Bank of America", "Smith & Sons".
194+
private val nameConnectors = "of the and for at on de del la le du des da do y & + -".split(' ').toSet()
195+
196+
// A sender's name is a few capitalized words or digits ("Rosa's Kitchen",
197+
// "Bank of America", "7-Eleven", "24 Hour Fitness") that doesn't open
198+
// with a greeting or a notice word.
199+
private fun looksLikeName(name: String): Boolean {
200+
val words = name.split(nameWordSeparator).filter { it.isNotEmpty() }
201+
if (words.isEmpty() || words.size > 5 || name.none { it.isLetter() }) return false
202+
if (words.first().lowercase().trim { !it.isLetterOrDigit() } in notANameOpener) return false
203+
return words.all { word -> word.lowercase() in nameConnectors || word.any { it.isUpperCase() || it.isDigit() } }
204+
}
205+
206+
private fun namesItsSender(text: String): Boolean {
207+
val prefix = senderPrefix.find(text)
208+
val name = prefix?.let { it.groupValues[1].ifEmpty { it.groupValues[2] } }
209+
return (name != null && looksLikeName(name)) || senderIntroduction.containsMatchIn(text)
210+
}
177211

178212
// Phone number in SMS body (common in callback scams)
179213
private val phoneInBody =
@@ -418,6 +452,19 @@ class SmsContentAnalyzer
418452
reasons.add("disguised_text")
419453
}
420454

455+
// The signals scored below, worked out first so the footer decision
456+
// can see every one of them.
457+
val alphaChars = ruleText.filter { it.isLetter() }
458+
val shouting = alphaChars.length > 10 && alphaChars.count { it.isUpperCase() }.toFloat() / alphaChars.length > 0.5f
459+
val callback = phoneInBody.containsMatchIn(ruleText)
460+
val specialRatio =
461+
ruleText.count { !it.isLetterOrDigit() && !it.isWhitespace() }.toFloat() /
462+
ruleText.length.coerceAtLeast(1)
463+
val specialHeavy = specialRatio > 0.15f && ruleText.length > 20
464+
val shortWithUrl = visibleText.length < 50 && urls.isNotEmpty()
465+
val replyBait = firstContact && SmsReplyBait.matches(ruleText, urls, visibleText.length)
466+
val otherSignal = score > 0 || shouting || callback || specialHeavy || shortWithUrl || replyBait
467+
421468
// Check spam keyword patterns
422469
var patternHits = 0
423470
val countPatternHit = {
@@ -433,44 +480,40 @@ class SmsContentAnalyzer
433480
if (patternHits >= 3) break // Cap pattern contribution
434481
if (pattern.containsMatchIn(ruleText)) countPatternHit()
435482
}
436-
if (patternHits > 0 || !namedSender.containsMatchIn(ruleText)) {
483+
if (patternHits > 0 || otherSignal || !namesItsSender(ruleText)) {
437484
for (cue in responseCues) {
438485
if (patternHits >= 3) break
439486
if (cue.containsMatchIn(ruleText)) countPatternHit()
440487
}
441488
}
442489

443490
// All caps text (>50% of message) — shouting is a spam signal
444-
val alphaChars = ruleText.filter { it.isLetter() }
445-
if (alphaChars.length > 10 && alphaChars.count { it.isUpperCase() }.toFloat() / alphaChars.length > 0.5f) {
491+
if (shouting) {
446492
score += 15
447493
reasons.add("excessive_caps")
448494
}
449495

450496
// Contains phone number in body (callback scam) — regex is
451497
// case-insensitive so we match the original body instead of lowercasing.
452-
if (phoneInBody.containsMatchIn(ruleText)) {
498+
if (callback) {
453499
score += 10
454500
reasons.add("callback_number")
455501
}
456502

457503
// Excessive special characters / emoji (common in spam)
458-
val specialRatio =
459-
ruleText.count { !it.isLetterOrDigit() && !it.isWhitespace() }.toFloat() /
460-
ruleText.length.coerceAtLeast(1)
461-
if (specialRatio > 0.15f && ruleText.length > 20) {
504+
if (specialHeavy) {
462505
score += 10
463506
reasons.add("special_chars")
464507
}
465508

466509
// Very short message with URL (likely phishing). Counted without
467510
// invisible characters, which would otherwise pad it past the limit.
468-
if (visibleText.length < 50 && urls.isNotEmpty()) {
511+
if (shortWithUrl) {
469512
score += 20
470513
reasons.add("short_msg_with_url")
471514
}
472515

473-
if (firstContact && SmsReplyBait.matches(ruleText, urls, visibleText.length)) {
516+
if (replyBait) {
474517
score += REPLY_BAIT_SCORE
475518
reasons.add("reply_bait")
476519
}

‎app/src/test/java/com/sysadmindoc/callshield/data/SmsContentAnalyzerTest.kt‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -785,6 +785,77 @@ class SmsContentAnalyzerTest {
785785
assertEquals(listOf("spam_keywords"), footed.reasons)
786786
}
787787

788+
@Test
789+
fun `any other signal brings a named sender's footer back`() {
790+
// A brand name up front doesn't make a link or a callback lure safe.
791+
listOf(
792+
"USPS: your parcel is on hold. Reschedule at https://usps-parcel.top/r" to 70,
793+
"Lakeside Clinic: confirm your visit at https://bit.ly/3xYz" to 75,
794+
"CVS: YOUR PRESCRIPTION IS READY FOR PICKUP TODAY" to 55,
795+
"Lakeside Clinic: call 800-555-0100 about your visit." to 50,
796+
).forEach { (lure, expected) ->
797+
val footed = SmsContentAnalyzer.analyze("$lure Reply STOP to opt out.")
798+
assertEquals(lure, expected, footed.score)
799+
assertTrue(lure, "spam_keywords" in footed.reasons)
800+
}
801+
}
802+
803+
@Test
804+
fun `a time, a link or a notice word before a colon doesn't name a sender`() {
805+
listOf(
806+
"Are you free Saturday at 3:30? Reply STOP to unsubscribe.",
807+
// Capitalized and short, but the colon sits inside a time.
808+
"Saturday 3:30 works for me? Reply STOP to unsubscribe.",
809+
"Meet me at 5:30 by the fountain. Reply STOP to unsubscribe.",
810+
"Look at these pics https://photos.example.com/abc Reply STOP to unsubscribe.",
811+
"tel:8005551234 is our new line. Reply STOP to opt out.",
812+
"Valued customer: we miss you. Reply STOP to unsubscribe.",
813+
"Valued Customer: we miss you. Reply STOP to unsubscribe.",
814+
"Hiya: long time no see! Reply STOP to unsubscribe.",
815+
"Greetings: we have news for you. Reply STOP to unsubscribe.",
816+
"Quick question: are you free this week? Reply STOP to unsubscribe.",
817+
"Hi Sam: are you free this week? Reply STOP to unsubscribe.",
818+
"[1/2] Hey, are you free? Reply STOP to unsubscribe.",
819+
// A sentence isn't a name: lowercase words, or more than five.
820+
"are you around tonight: text me back. Reply STOP to unsubscribe.",
821+
"We Want You Back At The Gym: Reply STOP to unsubscribe.",
822+
).forEach { body ->
823+
assertEquals(body, 40, SmsContentAnalyzer.analyze(body).score)
824+
}
825+
assertEquals(25, SmsContentAnalyzer.analyze("Alert: your parcel is waiting. Reply Y to confirm.").score)
826+
assertEquals(25, SmsContentAnalyzer.analyze("Notice: your package is at the depot. Reply 1 for help.").score)
827+
}
828+
829+
@Test
830+
fun `a business named with digits, connectors or brackets is a sender`() {
831+
listOf(
832+
"7-Eleven: your order is ready for pickup. Reply STOP to opt out.",
833+
"24 Hour Fitness: your class starts at 6 PM. Reply STOP to opt out.",
834+
"Smith & Sons Plumbing: we're on our way. Reply STOP to opt out.",
835+
"[Lakeside Clinic] Your visit is tomorrow at 9:00. Reply Y to confirm.",
836+
).forEach { body ->
837+
assertEquals(body, 0, SmsContentAnalyzer.analyze(body).score)
838+
}
839+
}
840+
841+
@Test
842+
fun `a long run of spaces can't stall the sender check`() {
843+
// The first version of the "this is Dana from" rule let spaces sit
844+
// inside and around the name, and 16,000 of them took over a minute.
845+
listOf(
846+
"this is" + " ".repeat(16_000) + "x",
847+
"this is Dana" + " ".repeat(16_000) + "from",
848+
"this is " + "a ".repeat(8_000),
849+
"[" + " ".repeat(16_000),
850+
"Lakeside" + " ".repeat(16_000) + ":",
851+
).forEach { body ->
852+
val started = System.nanoTime()
853+
SmsContentAnalyzer.analyze(body)
854+
val millis = (System.nanoTime() - started) / 1_000_000
855+
assertTrue("${body.take(12).trim()}... took $millis ms", millis < 2_000)
856+
}
857+
}
858+
788859
@Test
789860
fun `interview and payslip texts stay clean`() {
790861
listOf(

0 commit comments

Comments
 (0)