fix: retire URLhaus and honor OpenPhish feed cadence

URLhaus requires an Auth-Key and has no usable keyless path. Its parser test described a removed source, so ETag, 304 and six-hour expiry tests now cover the live feed. Cache and pin tests use active sources. Legacy stored preference and backup names remain readable.

Author: SysAdminDoc

CHANGELOG.md

@@ -4,6 +4,9 @@ All notable changes to CallShield will be documented in this file.
 
 ## Unreleased
 
+- Removed the inactive URLhaus lookup and its Settings claim. Optional link
+  checks now name PhishTank and OpenPhish. The OpenPhish feed refreshes every
+  six hours and reuses its local copy when the server reports no change.
 - Cached prefix ranges now expire on time, even when no feed sync runs.
 - Frequency auto-block now counts prior incoming calls that reached the phone's
   call log as answered or missed. Calls stopped or silenced by CallShield, and

README.md

@@ -147,7 +147,7 @@ SMS-specific layers (append after the shared chain, in their own priority order)
 - **Caller ID Overlay**. Suspicious calls (heuristic score 30-59) can use an explicit, default-off live enrichment option that checks SkipCalls. Clean calls never trigger it
 - **Region & caller-name rules**. Opt-in offline blocking outside the US states, Canadian provinces, area codes (`+1809`) and country calling codes (`+39`) you allow, plus bounded `*`/`?` trust and block patterns for carrier-presented caller names. Explicit number, system, prefix and wildcard blocks, and all allow layers, keep priority. Area codes absent from the pinned NANP table pass through this regional rule
 - **Opt-in message notification screening**. Google/Samsung Messages are enabled by default. AOSP Messages, SMS Organizer, Signal, WhatsApp, WhatsApp Business, Gmail, Outlook, and Thunderbird can be enabled individually. Private-messenger/email matches show a separate warning without removing the original notification.
-- **URL Safety**. Local spam-domain checks stay on-device. Optional URLhaus (abuse.ch) checks are off by default and disclose only the registrable domain
+- **URL Safety**. Local spam-domain checks stay on-device. Optional link checks use PhishTank and a six-hour OpenPhish feed. PhishTank receives only the site's base domain
 - **STIR/SHAKEN**. Blocks calls failing carrier caller ID verification (Android 11+)
 - **Outgoing call check**. Opt-in, through Android's call redirection role. A call you start to a number on your blocklist, in the spam database, on a premium-rate line or on a callback-scam country code is held, and a notification tells you why and offers Call anyway. Contacts and trusted numbers ring through, and a slow check never delays the call. If you wouldn't see the notification, in car mode or under Do Not Disturb for example, the call goes through.
 - **Answer & hang up**. Opt-in. A call CallShield would reject is answered without video and hung up after a delay you set (1 to 10 seconds), so spam can't leave a voicemail. Silent voicemail mode, auto-mute and a meeting-mode silence still go to voicemail, and the short answered call never counts toward trusting the caller. Contributed by tikkamasalla
@@ -342,7 +342,9 @@ scrape Nomorobo's restricted carrier feed.
 ### URL Safety (post-decision)
 | Source | What It Checks |
 |--------|---------------|
-| **URLhaus** (abuse.ch) | Optional, default-off malware-domain checks after local spam-domain matching; only the registrable domain is shared |
+| **Local spam-domain list** | Checks known spam domains on the phone |
+| **PhishTank** | Optional link lookup; receives only the site's base domain |
+| **OpenPhish** | Optional feed checked for updates after six hours of use; matches stay on the phone |
 
 ## Security
 
@@ -364,11 +366,11 @@ scrape Nomorobo's restricted carrier feed.
 
 ## Privacy
 
-All detection runs on-device. No personal data is collected. Network requests:
+Call and message decisions run on-device. No personal data is collected. Network requests:
 - Syncing spam database from GitHub (public)
 - Optional live caller enrichment is off by default and runs only for locally suspicious calls. The setting names every destination host before a number is shared
 - Community reports to Cloudflare Worker (phone number only, no identity)
-- Local spam-domain checks don't disclose SMS/RCS links. Optional URLhaus checks are opt-in and send only the registrable domain
+- Local spam-domain checks don't disclose SMS or RCS links. Optional PhishTank lookups send only the site's base domain, and the OpenPhish feed is downloaded for local matching. Neither receives message text
 
 No API keys. None required, none optional, no credential entry anywhere in the app. No accounts. No analytics. No ads.
 
@@ -533,7 +535,7 @@ language in [issue #7](https://github.com/SysAdminDoc/CallShield/issues/7).
 | Settings | DataStore Preferences 1.2.1 |
 | Background | WorkManager 2.11.2 |
 | Community API | Cloudflare Workers |
-| URL Safety | Local spam-domain data; optional URLhaus (abuse.ch) |
+| URL Safety | Local spam-domain data; optional PhishTank and OpenPhish |
 | Verification | Local Gradle, lint, and release-artifact checks |
 | Tests | 1556 JVM unit tests (JUnit) |
 | Strings | 1583 string resources and 38 plural groups (translation-ready) |

app/src/main/java/com/sysadmindoc/callshield/data/BackupSettingsPreferences.kt

@@ -29,8 +29,8 @@ internal fun Preferences.toBackupSettings(): BackupSettings =
         heuristicsEnabled = this[SpamRepository.KEY_HEURISTICS] ?: true,
         smsContentEnabled = this[SpamRepository.KEY_SMS_CONTENT] ?: true,
         smsBurstEnabled = this[SpamRepository.KEY_SMS_BURST] ?: true,
-        urlhausStripQueryEnabled = this[SpamRepository.KEY_URLHAUS_STRIP_QUERY] ?: true,
-        urlhausRemoteLookupEnabled = this[SpamRepository.KEY_URLHAUS_REMOTE_LOOKUP] ?: false,
+        urlhausStripQueryEnabled = this[SpamRepository.KEY_URL_STRIP_QUERY] ?: true,
+        urlhausRemoteLookupEnabled = this[SpamRepository.KEY_REMOTE_URL_LOOKUP] ?: false,
         liveCallerEnrichmentEnabled = this[SpamRepository.KEY_LIVE_CALLER_ENRICHMENT] ?: false,
         contactWhitelistEnabled = this[SpamRepository.KEY_CONTACT_WHITELIST] ?: true,
         contactsOnlyEnabled = this[SpamRepository.KEY_CONTACTS_ONLY] ?: false,
@@ -148,8 +148,8 @@ internal fun BackupSettings.writeTo(preferences: MutablePreferences) {
     preferences[SpamRepository.KEY_HEURISTICS] = heuristicsEnabled
     preferences[SpamRepository.KEY_SMS_CONTENT] = smsContentEnabled
     preferences[SpamRepository.KEY_SMS_BURST] = smsBurstEnabled
-    preferences[SpamRepository.KEY_URLHAUS_STRIP_QUERY] = urlhausStripQueryEnabled
-    preferences[SpamRepository.KEY_URLHAUS_REMOTE_LOOKUP] = urlhausRemoteLookupEnabled
+    preferences[SpamRepository.KEY_URL_STRIP_QUERY] = urlhausStripQueryEnabled
+    preferences[SpamRepository.KEY_REMOTE_URL_LOOKUP] = urlhausRemoteLookupEnabled
     preferences[SpamRepository.KEY_LIVE_CALLER_ENRICHMENT] = liveCallerEnrichmentEnabled
     preferences[SpamRepository.KEY_CONTACT_WHITELIST] = contactWhitelistEnabled
     preferences[SpamRepository.KEY_CONTACTS_ONLY] = contactsOnlyEnabled

app/src/main/java/com/sysadmindoc/callshield/data/SpamRepository.kt

@@ -182,8 +182,10 @@ class SpamRepository(
         val KEY_HEURISTICS = booleanPreferencesKey("heuristics_enabled")
         val KEY_SMS_CONTENT = booleanPreferencesKey("sms_content_analysis_enabled")
         val KEY_SMS_BURST = booleanPreferencesKey("sms_burst_detection_enabled")
-        val KEY_URLHAUS_STRIP_QUERY = booleanPreferencesKey("urlhaus_strip_query_enabled")
-        val KEY_URLHAUS_REMOTE_LOOKUP = booleanPreferencesKey("urlhaus_remote_lookup_enabled")
+
+        // Keep the old storage names so an app update preserves both settings.
+        val KEY_URL_STRIP_QUERY = booleanPreferencesKey("urlhaus_strip_query_enabled")
+        val KEY_REMOTE_URL_LOOKUP = booleanPreferencesKey("urlhaus_remote_lookup_enabled")
         val KEY_LIVE_CALLER_ENRICHMENT = booleanPreferencesKey("live_caller_enrichment_enabled")
         val KEY_CONTACT_WHITELIST = booleanPreferencesKey("contact_whitelist_enabled")
         val KEY_CONTACTS_ONLY = booleanPreferencesKey("contacts_only_mode_enabled")
@@ -393,8 +395,8 @@ class SpamRepository(
     val heuristicsEnabled: Flow<Boolean> = settingsRepository.heuristicsEnabled
     val smsContentEnabled: Flow<Boolean> = settingsRepository.smsContentEnabled
     val smsBurstEnabled: Flow<Boolean> = settingsRepository.smsBurstEnabled
-    val urlhausStripQueryEnabled: Flow<Boolean> = settingsRepository.urlhausStripQueryEnabled
-    val urlhausRemoteLookupEnabled: Flow<Boolean> = settingsRepository.urlhausRemoteLookupEnabled
+    val urlStripQueryEnabled: Flow<Boolean> = settingsRepository.urlStripQueryEnabled
+    val remoteUrlLookupEnabled: Flow<Boolean> = settingsRepository.remoteUrlLookupEnabled
     val liveCallerEnrichmentEnabled: Flow<Boolean> = settingsRepository.liveCallerEnrichmentEnabled
     val contactWhitelistEnabled: Flow<Boolean> = settingsRepository.contactWhitelistEnabled
     val contactsOnlyEnabled: Flow<Boolean> = settingsRepository.contactsOnlyEnabled
@@ -575,9 +577,9 @@ class SpamRepository(
 
     suspend fun setSmsBurst(enabled: Boolean) = settingsRepository.setSmsBurst(enabled)
 
-    suspend fun setUrlhausStripQuery(enabled: Boolean) = settingsRepository.setUrlhausStripQuery(enabled)
+    suspend fun setUrlStripQuery(enabled: Boolean) = settingsRepository.setUrlStripQuery(enabled)
 
-    suspend fun setUrlhausRemoteLookup(enabled: Boolean) = settingsRepository.setUrlhausRemoteLookup(enabled)
+    suspend fun setRemoteUrlLookup(enabled: Boolean) = settingsRepository.setRemoteUrlLookup(enabled)
 
     suspend fun setLiveCallerEnrichment(enabled: Boolean) = settingsRepository.setLiveCallerEnrichment(enabled)
 

app/src/main/java/com/sysadmindoc/callshield/data/remote/HttpClient.kt

@@ -71,12 +71,6 @@ object HttpClient {
                     "sha256/kIdp6NNEd8wsugYyyIYFsi1ylMCED3hZbSR8ZFsa/A4=",
                     "sha256/mEflZT5enoR1FuXLgYYGqnVEoZvmf9c2bVBpiOjYQ0c=",
                 ),
-            "urlhaus-api.abuse.ch" to
-                listOf(
-                    "sha256/Yz0ts4M9B9b1XBQTQtITniseuxd86RWgvl5+aekJM8Q=",
-                    "sha256/A7AXWj1rjKywVBFqQcQvoHEEWHeViDOFXrwzRs984Xc=",
-                    "sha256/68l4rg3Z5YItaxllJZb2IMk9fK76lSGRywUKYyypAF8=",
-                ),
         )
 
     internal val certificatePinner: CertificatePinner =

app/src/main/java/com/sysadmindoc/callshield/data/remote/UrlSafetyChecker.kt

@@ -23,8 +23,8 @@ object UrlSafetyChecker {
         val isMalicious: Boolean,
         val threat: String = "",
         val tags: List<String> = emptyList(),
-        val source: UrlThreatSource = UrlThreatSource.URLHAUS,
-        val sourceVersion: String = UrlThreatSource.URLHAUS.defaultVersion,
+        val source: UrlThreatSource = UrlThreatSource.NONE,
+        val sourceVersion: String = UrlThreatSource.NONE.defaultVersion,
         val category: UrlThreatCategory = UrlThreatCategory.UNKNOWN,
         val verdict: UrlThreatVerdict = if (isMalicious) UrlThreatVerdict.MALICIOUS else UrlThreatVerdict.CLEAN,
         val expiresAtMillis: Long = 0L,

app/src/main/java/com/sysadmindoc/callshield/data/remote/UrlThreatAdapters.kt

@@ -13,7 +13,6 @@ import java.time.Instant
 import java.util.concurrent.TimeUnit
 
 private const val URL_THREAT_USER_AGENT = "CallShield/1.0 (URL threat lookup)"
-private const val URLHAUS_ENDPOINT = "https://urlhaus-api.abuse.ch/v1/url/"
 private const val PHISHTANK_ENDPOINT = "https://checkurl.phishtank.com/checkurl/"
 private const val OPENPHISH_FEED_URL =
     "https://raw.githubusercontent.com/openphish/public_feed/refs/heads/main/feed.txt"
@@ -23,7 +22,7 @@ private const val WEB_RISK_ENDPOINT = "https://webrisk.googleapis.com/v1/uris:se
 private const val MAX_URL_LOOKUP_RESPONSE_BYTES = 256L * 1024L
 private const val MAX_OPENPHISH_FEED_BYTES = 16L * 1024L * 1024L
 private const val MAX_OPENPHISH_HOSTS = 250_000
-private const val OPENPHISH_FEED_TTL_MILLIS = 30L * 60L * 1_000L
+private const val OPENPHISH_FEED_TTL_MILLIS = 6L * 60L * 60L * 1_000L
 private const val MAX_REMOTE_TTL_MILLIS = 24L * 60L * 60L * 1_000L
 
 private val urlThreatClient: OkHttpClient =
@@ -33,118 +32,6 @@ private val urlThreatClient: OkHttpClient =
         .readTimeout(8, TimeUnit.SECONDS)
         .build()
 
-internal class UrlhausThreatAdapter(
-    private val authKey: String? = null,
-    private val client: OkHttpClient = urlThreatClient,
-) : UrlThreatAdapter {
-    override val source: UrlThreatSource = UrlThreatSource.URLHAUS
-    override val sourceVersion: String = source.defaultVersion
-    override val isConfigured: Boolean
-        get() = !authKey.isNullOrBlank()
-
-    override suspend fun lookup(
-        canonicalUrl: String,
-        nowMillis: Long,
-    ): UrlThreatResult =
-        withContext(Dispatchers.IO) {
-            if (!isConfigured) {
-                return@withContext UrlThreatResult.unknown(
-                    source,
-                    sourceVersion,
-                    canonicalUrl,
-                    nowMillis,
-                    detail = "auth_key_unconfigured",
-                )
-            }
-            if (!canonicalUrl.isSafeLookupUrl()) {
-                return@withContext UrlThreatResult.unknown(source, sourceVersion, canonicalUrl, nowMillis, "invalid_url")
-            }
-            try {
-                val requestBody = FormBody.Builder().add("url", canonicalUrl).build()
-                val request =
-                    Request
-                        .Builder()
-                        .url(URLHAUS_ENDPOINT)
-                        .post(requestBody)
-                        .header("Auth-Key", authKey.orEmpty())
-                        .header("User-Agent", URL_THREAT_USER_AGENT)
-                        .build()
-                client.newCall(request).execute().use { response ->
-                    if (!response.isSuccessful) {
-                        return@withContext UrlThreatResult.unknown(
-                            source,
-                            sourceVersion,
-                            canonicalUrl,
-                            nowMillis,
-                            detail = "http_${response.code}",
-                        )
-                    }
-                    val body = response.body.readUtf8Bounded(MAX_URL_LOOKUP_RESPONSE_BYTES)
-                    if (body !is BoundedResponseBody.Text) {
-                        return@withContext UrlThreatResult.unknown(source, sourceVersion, canonicalUrl, nowMillis, "empty_body")
-                    }
-                    parseUrlhausResponse(body.value, canonicalUrl, nowMillis)
-                }
-            } catch (_: IOException) {
-                UrlThreatResult.unknown(source, sourceVersion, canonicalUrl, nowMillis, "unavailable")
-            } catch (_: RuntimeException) {
-                UrlThreatResult.unknown(source, sourceVersion, canonicalUrl, nowMillis, "unavailable")
-            }
-        }
-
-    companion object {
-        internal fun parseUrlhausResponse(
-            body: String,
-            canonicalUrl: String,
-            nowMillis: Long,
-        ): UrlThreatResult {
-            val queryStatus = jsonString(body, "query_status").orEmpty()
-            val threat = jsonString(body, "threat").orEmpty()
-            val tags = jsonStringArray(body, "tags")
-            val category =
-                if (threat.contains("phish", ignoreCase = true) ||
-                    queryStatus.contains("phish", ignoreCase = true)
-                ) {
-                    UrlThreatCategory.PHISHING
-                } else {
-                    UrlThreatCategory.MALWARE
-                }
-            val positive =
-                queryStatus in setOf("is_malware", "is_phishing", "is_botnet_cc") ||
-                    (
-                        queryStatus == "ok" &&
-                            (jsonString(body, "url_status") != null || jsonArrayHasEntries(body, "urls"))
-                    )
-            return if (positive) {
-                UrlThreatResult.malicious(
-                    source = UrlThreatSource.URLHAUS,
-                    sourceVersion = UrlThreatSource.URLHAUS.defaultVersion,
-                    canonicalUrl = canonicalUrl,
-                    category = category,
-                    nowMillis = nowMillis,
-                    tags = tags,
-                    detail = threat.ifBlank { queryStatus },
-                )
-            } else if (queryStatus == "no_results") {
-                UrlThreatResult.clean(
-                    UrlThreatSource.URLHAUS,
-                    UrlThreatSource.URLHAUS.defaultVersion,
-                    canonicalUrl,
-                    nowMillis,
-                )
-            } else {
-                UrlThreatResult.unknown(
-                    UrlThreatSource.URLHAUS,
-                    UrlThreatSource.URLHAUS.defaultVersion,
-                    canonicalUrl,
-                    nowMillis,
-                    detail = queryStatus.ifBlank { "unrecognized_response" },
-                )
-            }
-        }
-    }
-}
-
 internal class PhishTankThreatAdapter(
     private val appKey: String? = null,
     private val client: OkHttpClient = urlThreatClient,
@@ -269,6 +156,11 @@ internal class OpenPhishThreatAdapter(
             loadSnapshot(nowMillis)
                 ?: return UrlThreatResult.unknown(source, sourceVersion, canonicalUrl, nowMillis, "feed_unavailable")
         val host = canonicalUrl.toHttpUrlOrNull()?.host?.removePrefix("www.")
+        val resultTtlMillis =
+            minOf(
+                UrlThreatCache.DEFAULT_TTL_MILLIS,
+                (snapshot.expiresAtMillis - nowMillis).coerceAtLeast(1L),
+            )
         return if (host != null && host in snapshot.hosts) {
             UrlThreatResult.malicious(
                 source = source,
@@ -277,9 +169,10 @@ internal class OpenPhishThreatAdapter(
                 category = UrlThreatCategory.PHISHING,
                 nowMillis = nowMillis,
                 detail = "feed_domain_match",
+                ttlMillis = resultTtlMillis,
             )
         } else {
-            UrlThreatResult.clean(source, sourceVersion, canonicalUrl, nowMillis)
+            UrlThreatResult.clean(source, sourceVersion, canonicalUrl, nowMillis, ttlMillis = resultTtlMillis)
         }
     }
 
@@ -293,8 +186,14 @@ internal class OpenPhishThreatAdapter(
                         .Builder()
                         .url(OPENPHISH_FEED_URL)
                         .header("User-Agent", URL_THREAT_USER_AGENT)
+                        .apply { feedSnapshot?.etag?.let { header("If-None-Match", it) } }
                         .build()
                 client.newCall(request).execute().use { response ->
+                    if (response.code == 304) {
+                        return@withContext feedSnapshot?.copy(expiresAtMillis = nowMillis + OPENPHISH_FEED_TTL_MILLIS)?.also {
+                            feedSnapshot = it
+                        }
+                    }
                     if (!response.isSuccessful) return@withContext null
                     val body = response.body.readUtf8Bounded(MAX_OPENPHISH_FEED_BYTES)
                     if (body !is BoundedResponseBody.Text) return@withContext null
@@ -311,7 +210,11 @@ internal class OpenPhishThreatAdapter(
                             }.distinct()
                             .take(MAX_OPENPHISH_HOSTS)
                             .toSet()
-                    FeedSnapshot(hosts = hosts, expiresAtMillis = nowMillis + OPENPHISH_FEED_TTL_MILLIS).also {
+                    FeedSnapshot(
+                        hosts = hosts,
+                        expiresAtMillis = nowMillis + OPENPHISH_FEED_TTL_MILLIS,
+                        etag = response.header("ETag")?.takeIf { it.isNotBlank() && it.length <= 256 },
+                    ).also {
                         feedSnapshot = it
                     }
                 }
@@ -326,6 +229,7 @@ internal class OpenPhishThreatAdapter(
     private data class FeedSnapshot(
         val hosts: Set<String>,
         val expiresAtMillis: Long,
+        val etag: String?,
     )
 }
 
@@ -610,13 +514,3 @@ private fun jsonStringArray(
         .take(32)
         .toList()
 }
-
-private fun jsonArrayHasEntries(
-    body: String,
-    key: String,
-): Boolean =
-    Regex("\\\"${Regex.escape(key)}\\\"\\s*:\\s*\\[([^]]*)]", RegexOption.IGNORE_CASE)
-        .find(body)
-        ?.groupValues
-        ?.getOrNull(1)
-        ?.isNotBlank() == true