feat(blocklists): refresh external subscriptions on a schedule

A subscription was fetched only when it was added or switched back on, so a
list that changes daily went stale on the device indefinitely.

ExternalBlocklistRefreshWorker runs every six hours on a network and fetches
each enabled list that is due. A list is due a day after its last successful
sync, or on the interval it declares in a uBlock-style header
("# Expires: 12 hours", or a top-level "expires" in JSON), clamped to between
six hours and a week. A failing list is retried no sooner than six hours
after the last attempt.

Nobody reads a preview in the background, so a download that comes back
empty or under half the list's rows is not applied. The last good rows stay
and the row says why, the way a collapsed database or hot-feed publish is
refused. A person can still accept the smaller list by switching it off and
on, which goes through the ordinary apply path. The settings row now shows
when the list last updated.

Tests: a WorkManager test (work-testing, test scope only) drives the real
worker over a real repository through a refresh, a failed refresh, retry
pacing and recovery, plus shrink, empty, Expires and disabled cases. Policy
and parser cases are covered separately. Disabling the shrink guard or the
retry pacing fails them. IsolatedRepositoryFixture takes an optional list
source so the worker test can fake the list server, and WorkerScheduleTest
pins the new request. No existing assertion changed.

The three new strings ship with zh-rCN translations, and the zh coverage
floor is ratcheted to 75.7.

Author: SysAdminDoc

CHANGELOG.md

@@ -26,6 +26,16 @@ All notable changes to CallShield will be documented in this file.
 - A short but valid international number (the Faroes, Greenland, Andorra, some
   German landlines) is no longer counted as a malformed one by the heuristics.
 
+- External blocklist subscriptions now stay current. A list used to be
+  fetched only when you added it or switched it back on, so one that changes
+  daily went stale on your phone indefinitely. Each list is now fetched again
+  once a day, or on the interval it declares in an `Expires:` header line,
+  never more often than every six hours. A download that comes back empty or
+  with under half the list's numbers isn't applied in the background, because
+  a broken export looks just like a list that cleaned itself up. The list
+  keeps its last good copy and says why on its row, which also shows when it
+  last updated.
+
 ### Live caller lookup
 
 - Three of the four lookup services are gone, so the app no longer asks them.

README.md

@@ -553,6 +553,7 @@ by locale and message type without shipping personal data:
 ### Data & System
 - Selective backup/restore for rules, non-secret settings, and opt-in logs; CSV log export; auto-cleanup (7/14/30/90 days)
 - Weekly full sync + scheduled hot list refresh, daily digest notification
+- External blocklist subscriptions (Settings > External blocklists) take HTTPS CSV, TXT or JSON number lists of up to 1 MB and 20,000 rows. Each list is fetched again once a day, or on the interval it declares in its header (`# Expires: 12 hours`, or `"expires": "12h"` in JSON), never more often than every six hours and at least weekly. A download that comes back empty or with under half the list's numbers isn't applied in the background. The list keeps its last good copy and says why on its row
 - Quick Settings tile, app shortcuts, home screen widget
 - Protection test validates all layers and permissions, including checker errors and deadline cutoffs
 - Rules surface priority conflicts after sync and edits, with the winning rule and a review path
@@ -808,7 +809,7 @@ language in [issue #7](https://github.com/SysAdminDoc/CallShield/issues/7).
 | URL Safety | Local spam-domain data; optional URLhaus (abuse.ch) |
 | Verification | Local Gradle, lint, and release-artifact checks |
 | Tests | 1100 JVM unit tests (JUnit) |
-| Strings | 1411 string resources and 33 plural groups (translation-ready) |
+| Strings | 1414 string resources and 33 plural groups (translation-ready) |
 | Accessibility | 100+ content descriptions, 48dp touch targets |
 | Min SDK | 29 (Android 10) |
 | Target SDK | 36 |

app/build.gradle.kts

@@ -279,6 +279,7 @@ dependencies {
     testImplementation(libs.kotlinx.serialization.json)
     testImplementation(libs.robolectric)
     testImplementation(libs.androidx.test.core)
+    testImplementation(libs.androidx.work.testing)
 
     // Instrumentation tests (emulator / device)
     androidTestImplementation(platform(libs.androidx.compose.bom))

app/gradle.lockfile

@@ -189,6 +189,7 @@ androidx.window:window-core:1.5.0=debugAndroidTestRuntimeClasspath,debugRuntimeC
 androidx.window:window:1.5.0=debugAndroidTestRuntimeClasspath,debugRuntimeClasspath,debugUnitTestRuntimeClasspath,releaseRuntimeClasspath,releaseUnitTestRuntimeClasspath
 androidx.work:work-runtime-ktx:2.11.2=debugAndroidTestCompileClasspath,debugCompileClasspath,debugRuntimeClasspath,debugUnitTestCompileClasspath,debugUnitTestRuntimeClasspath,releaseCompileClasspath,releaseRuntimeClasspath,releaseUnitTestCompileClasspath,releaseUnitTestRuntimeClasspath
 androidx.work:work-runtime:2.11.2=debugAndroidTestCompileClasspath,debugCompileClasspath,debugRuntimeClasspath,debugUnitTestCompileClasspath,debugUnitTestRuntimeClasspath,releaseCompileClasspath,releaseRuntimeClasspath,releaseUnitTestCompileClasspath,releaseUnitTestRuntimeClasspath
+androidx.work:work-testing:2.11.2=debugUnitTestCompileClasspath,debugUnitTestRuntimeClasspath,releaseUnitTestCompileClasspath,releaseUnitTestRuntimeClasspath
 ch.qos.logback:logback-classic:1.3.16=ktlint
 ch.qos.logback:logback-core:1.3.16=ktlint
 com.almworks.sqlite4java:sqlite4java:1.0.392=debugUnitTestCompileClasspath,debugUnitTestRuntimeClasspath,releaseUnitTestCompileClasspath,releaseUnitTestRuntimeClasspath

app/src/main/java/com/sysadmindoc/callshield/CallShieldApp.kt

@@ -23,6 +23,7 @@ import com.sysadmindoc.callshield.service.AppUpdateWorker
 import com.sysadmindoc.callshield.service.CrashReporter
 import com.sysadmindoc.callshield.service.DigestWorker
 import com.sysadmindoc.callshield.service.DirectBootScreeningStore
+import com.sysadmindoc.callshield.service.ExternalBlocklistRefreshWorker
 import com.sysadmindoc.callshield.service.HotDataSync
 import com.sysadmindoc.callshield.service.HotListSyncWorker
 import com.sysadmindoc.callshield.service.NotificationHelper
@@ -89,6 +90,7 @@ class CallShieldApp :
             SyncWorker.schedule(this)
             HotListSyncWorker.schedule(this)
             DigestWorker.schedule(this)
+            ExternalBlocklistRefreshWorker.schedule(this)
             PendingBlockedCallLogWorker.schedule(this)
             ProtectionHealthWorker.schedule(this)
             ProtectionHealthWorker.checkNow(this)

app/src/main/java/com/sysadmindoc/callshield/data/ExternalBlocklistParser.kt

@@ -28,6 +28,8 @@ internal data class ParsedExternalBlocklist(
     val format: String,
     val numbers: List<SpamNumber>,
     val skippedRows: Int,
+    /** Hours between refreshes the list declares for itself, or 0 for none. */
+    val declaredRefreshHours: Int = 0,
 )
 
 private data class ExternalBlocklistRow(
@@ -53,6 +55,10 @@ internal object ExternalBlocklistParser {
     private val descriptionFields = setOf("description", "comment", "name", "reason")
     private val arrayFields = setOf("numbers", "blocklist", "entries", "data")
 
+    private const val EXPIRES_HEADER_LINES = 50
+    private val expiresLine = Regex("""^expires\s*:\s*(.+)$""", RegexOption.IGNORE_CASE)
+    private val expiresValue = Regex("""^(\d{1,5})(?!\d)\s*([a-z]*)""", RegexOption.IGNORE_CASE)
+
     fun parse(
         rawUrl: String,
         rawLabel: String,
@@ -104,9 +110,63 @@ internal object ExternalBlocklistParser {
             format = format,
             numbers = numbers,
             skippedRows = skippedRows,
+            declaredRefreshHours = declaredRefreshHours(format, body) ?: 0,
         )
     }
 
+    /**
+     * The refresh interval, in hours, that a list declares for itself, or null
+     * when it declares none. This is uBlock Origin's `Expires:` convention:
+     * text and CSV lists put it in a header comment (`# Expires: 12 hours`),
+     * JSON lists in a top-level `"expires"` value. A bare number means days.
+     */
+    fun declaredRefreshHours(
+        format: String,
+        body: String,
+    ): Int? {
+        val declared =
+            if (format == "json") {
+                jsonExpires(body)
+            } else {
+                body
+                    .lineSequence()
+                    .map { it.trim() }
+                    .filter { it.isNotEmpty() }
+                    .take(EXPIRES_HEADER_LINES)
+                    .takeWhile { it.startsWith("#") || it.startsWith("//") }
+                    .map { it.removePrefix("//").removePrefix("#").trim() }
+                    .firstNotNullOfOrNull { expiresLine.find(it)?.groupValues?.get(1) }
+            }
+        return declared?.let(::expiresHours)
+    }
+
+    fun expiresHours(value: String): Int? {
+        val match = expiresValue.find(value.trim()) ?: return null
+        val amount = match.groupValues[1].toInt()
+        if (amount <= 0) return null
+        return when (match.groupValues[2].lowercase()) {
+            "h", "hr", "hrs", "hour", "hours" -> amount
+            "", "d", "day", "days" -> amount * 24
+            else -> null
+        }
+    }
+
+    private fun jsonExpires(body: String): String? =
+        try {
+            val reader = JsonReader.of(Buffer().writeUtf8(body))
+            var value: String? = null
+            if (reader.peek() == JsonReader.Token.BEGIN_OBJECT) {
+                reader.beginObject()
+                while (value == null && reader.hasNext()) {
+                    if (reader.nextName() == "expires") value = reader.nextScalarString() else reader.skipValue()
+                }
+            }
+            value
+        } catch (e: Exception) {
+            // The declaration is optional metadata; never fail a list over it.
+            null
+        }
+
     fun validateHttpUrl(rawUrl: String): String {
         val url =
             rawUrl.trim().toHttpUrlOrNull()

app/src/main/java/com/sysadmindoc/callshield/data/ExternalBlocklistRefreshPolicy.kt

@@ -0,0 +1,52 @@
+package com.sysadmindoc.callshield.data
+
+import com.sysadmindoc.callshield.data.model.ExternalBlocklistSubscription
+import java.util.concurrent.TimeUnit
+
+/**
+ * When a subscribed list is fetched again in the background, and when a fetch
+ * is too suspicious to apply without the preview a person would have seen.
+ */
+internal object ExternalBlocklistRefreshPolicy {
+    /** A list that declares no `Expires:` interval is refreshed daily. */
+    val DEFAULT_INTERVAL_MILLIS = TimeUnit.HOURS.toMillis(24)
+
+    /** Never sooner than this, whatever the list declares or however often it fails. */
+    val MIN_INTERVAL_MILLIS = TimeUnit.HOURS.toMillis(6)
+
+    /** A list that declares a longer interval is still checked weekly. */
+    val MAX_INTERVAL_MILLIS = TimeUnit.DAYS.toMillis(7)
+
+    fun intervalMillis(declaredHours: Int): Long =
+        if (declaredHours <= 0) {
+            DEFAULT_INTERVAL_MILLIS
+        } else {
+            TimeUnit.HOURS.toMillis(declaredHours.toLong()).coerceIn(MIN_INTERVAL_MILLIS, MAX_INTERVAL_MILLIS)
+        }
+
+    fun isDue(
+        subscription: ExternalBlocklistSubscription,
+        now: Long,
+    ): Boolean {
+        if (!subscription.enabled) return false
+        // A stamp in the future means the clock went back. Waiting for it to
+        // come round again would stall the list for as long as the jump.
+        val sinceAttempt = now - subscription.lastAttemptAt
+        if (subscription.lastAttemptAt > 0L && sinceAttempt in 0L until MIN_INTERVAL_MILLIS) return false
+        val sinceSuccess = now - subscription.lastSyncedAt
+        return subscription.lastSyncedAt <= 0L ||
+            sinceSuccess < 0L ||
+            sinceSuccess >= intervalMillis(subscription.declaredRefreshHours)
+    }
+
+    /**
+     * A download holding under half of what the list holds now is kept back,
+     * as an empty one is. A publisher's broken export looks exactly like a
+     * list that cleaned itself up, and only a person looking at the preview
+     * can tell them apart.
+     */
+    fun isSuspiciousShrink(
+        currentRows: Int,
+        nextRows: Int,
+    ): Boolean = currentRows > 0 && nextRows.toLong() * 2 < currentRows
+}

app/src/main/java/com/sysadmindoc/callshield/data/SpamRepository.kt

@@ -20,7 +20,9 @@ import com.sysadmindoc.callshield.data.checker.CheckerDependencies
 import com.sysadmindoc.callshield.data.local.AppDatabase
 import com.sysadmindoc.callshield.data.local.SpamDao
 import com.sysadmindoc.callshield.data.model.*
+import com.sysadmindoc.callshield.data.remote.ExternalBlocklistDataSource
 import com.sysadmindoc.callshield.data.remote.GitHubDataSource
+import com.sysadmindoc.callshield.data.remote.OkHttpExternalBlocklistDataSource
 import com.sysadmindoc.callshield.data.remote.SpamDataSource
 import com.sysadmindoc.callshield.data.repository.BlocklistRepository
 import com.sysadmindoc.callshield.data.repository.SettingsRepository
@@ -79,6 +81,7 @@ class SpamRepository(
     privateSettingsDataStore: DataStore<Preferences>? = null,
     private val phoneIdentityCanonicalizer: PhoneIdentityCanonicalizer =
         PhoneIdentityCanonicalizer.fromContext(context.applicationContext),
+    externalBlocklistDataSource: ExternalBlocklistDataSource = OkHttpExternalBlocklistDataSource(),
 ) {
     private val appContext: Context = context.applicationContext
     private val db: AppDatabase = database
@@ -107,6 +110,7 @@ class SpamRepository(
             settingsRepository = settingsRepository,
             normalizeNumber = phoneIdentityCanonicalizer::canonicalizePhone,
             invalidateAllCaches = spamRepositoryImpl::invalidateAllCaches,
+            externalBlocklistDataSource = externalBlocklistDataSource,
         )
     private val blocklistRepository =
         BlocklistRepository(
@@ -646,6 +650,8 @@ class SpamRepository(
 
     suspend fun removeExternalBlocklistSubscription(id: String) = syncRepository.removeExternalBlocklistSubscription(id)
 
+    suspend fun refreshDueExternalBlocklists(now: Long = System.currentTimeMillis()): List<ExternalBlocklistRefreshOutcome> = syncRepository.refreshDueExternalBlocklists(now)
+
     // ── Blocklist management ───────────────────────────────────────────
     suspend fun blockNumber(
         number: String,

app/src/main/java/com/sysadmindoc/callshield/data/model/ExternalBlocklistSubscription.kt

@@ -10,6 +10,10 @@ data class ExternalBlocklistSubscription(
     val lastAdded: Int = 0,
     val lastRemoved: Int = 0,
     val lastError: String = "",
+    /** Hours between refreshes that the list declares in an `Expires:` line, or 0 for none. */
+    val declaredRefreshHours: Int = 0,
+    /** The last fetch, successful or not. Paces retries of a list that keeps failing. */
+    val lastAttemptAt: Long = 0L,
 ) {
     val source: String get() = sourceFor(id)
 
@@ -34,6 +38,14 @@ data class ExternalBlocklistPreview(
     val blockedByOtherSources: Int,
 )
 
+enum class ExternalBlocklistRefreshOutcome {
+    REFRESHED,
+
+    /** The download was empty or under half the list's size, so the last good rows stay. */
+    HELD,
+    FAILED,
+}
+
 data class ExternalBlocklistImportResult(
     val success: Boolean,
     val message: String,