fix(feeds): retry a refused feed at its commit, and fail over fast when GitHub is unreachable

Three findings from a refuting review of the feed mirror and the signature
retry.

The retry after a signature refusal appended ?cb=<nanoTime>. Live headers
show raw.githubusercontent.com and jsDelivr both leave the query out of the
cache key (X-Cache: HIT, Source-Age continuing), so the retry re-read the same
stale body and signature, up to 16 MB for the database. It now asks GitHub
for the commit the branch points to (one small API call, Accept:
application/vnd.github.sha) and fetches both files at /<sha>/, a path whose
body and signature always come from one publish. A mirror gets no second try.
The test fake served a fresh copy to any URL with a "?", which is how the
no-op passed. It now ignores queries like the real CDN.

A blackholed GitHub cost about 90 s per file (the default-branch lookup, then
main and master at 30 s each) before the mirror was asked, so a 256-shard
sync could never finish. A network-level GitHub failure (timeout, refused or
unreachable connection, a name that won't resolve) now puts the mirror first
for ten minutes, a failed default-branch lookup isn't repeated for ten
minutes, and each call has a five-minute ceiling. An HTTP answer, a 404
included, doesn't count as unreachable.

FeedMirror was filled asynchronously at process start, so a worker in a cold
process could build its sources before the setting arrived and skip the
mirror. The app marks the setting as loading before it reads it, and a fetch
waits up to three seconds for it.

The Settings text, README and data README said GitHub is always tried first
and that a mirror "can't change what you get". Both are now accurate: GitHub
goes first except right after it couldn't be reached, and a mirror can't
alter the data but can hold back updates.

Tests: GitHubDataSourceFetchTest's two past-cache tests become commit-retry
tests, plus a refusal that stands when the commit can't be resolved, a mirror
copy that isn't retried, the outage window opening and closing, a 404 not
counting, the branch lookup not repeating, and a fetch waiting for the mirror
setting. FeedMirrorTest covers the wait, no wait when nothing loads, the
timeout, and a failed read. Eight mutants are each killed by name.

Author: SysAdminDoc

CHANGELOG.md

@@ -25,7 +25,8 @@ All notable changes to CallShield will be documented in this file.
 - A download whose signature arrives damaged or late is handled for what it
   is. Right after a new feed is published, the download network can hand out
   the new file with the old signature for a few minutes, so a mismatch is now
-  fetched once more past its cache before it counts as a failure. A signature
+  fetched again, both files at the exact commit, before it counts as a
+  failure. A signature
   the server fails to send is a network hiccup to retry, not an unsigned feed.
   A trending feed that is refused shows as refused in Protection Test instead
   of "unreachable".
@@ -39,11 +40,13 @@ All notable changes to CallShield will be documented in this file.
   longer hides why. Its "not found" answer used to replace a passing server
   error and stop the app from retrying.
 - If GitHub is blocked where you live, Settings > Feed mirror takes a second
-  address for the protection data. CallShield still asks GitHub first and only
-  then tries the mirror, before falling back to the copy built into the app.
-  One tap fills in jsDelivr, which serves the same files and can lag by up to
-  12 hours. The signature check applies to every mirrored file, so a mirror
-  can't slip in anything the project didn't sign.
+  address for the protection data. CallShield asks GitHub first and then the
+  mirror, before falling back to the copy built into the app. For ten minutes
+  after GitHub couldn't be reached at all, the mirror goes first, so a sync
+  behind a block doesn't wait out a timeout for every file. One tap fills in
+  jsDelivr, which serves the same files and can lag by up to 12 hours. The
+  signature check applies to every mirrored file, so a mirror can't slip in
+  anything the project didn't sign, though it can hold back updates.
 - The ML scorer no longer judges callers outside North America. It treated any
   ten-digit number as a US or Canadian one once the `+` was stripped, so an
   unknown Singapore mobile, or a landline in New Zealand, Belgium, Thailand or

README.md

@@ -558,7 +558,7 @@ by locale and message type without shipping personal data:
 - Selective backup/restore for rules, non-secret settings, and opt-in logs, plus CSV log export and auto-cleanup (7/14/30/90 days)
 - Database sync every 6 hours, a trending-feed check every 30 minutes, and a daily digest notification
 - External blocklist subscriptions (Settings > External blocklists) take HTTPS CSV, TXT or JSON number lists of up to 1 MB and 20,000 rows. Each list is fetched again once a day, or on the interval it declares in its header (`# Expires: 12 hours`, or `"expires": "12h"` in JSON), never more often than every six hours and at least weekly. A download that comes back empty or with under half the list's numbers isn't applied in the background. The list keeps its last good copy and says why on its row. Each row shows the list's name, the host it comes from, how many numbers it holds and when it last updated, and TalkBack reads it as one item. Tap a row to switch its list off or on. Removing a list takes its numbers out straight away, and Undo puts both back without downloading the list again
-- If GitHub is blocked where you live, Settings > Feed mirror takes a second address for the protection data. CallShield asks GitHub first, then the mirror, then falls back to the copy bundled with the app. One tap fills in jsDelivr (`https://cdn.jsdelivr.net/gh/SysAdminDoc/CallShield@master/`), which serves the same files and can run up to 12 hours behind. Mirrored files go through the same signature check, so a mirror can't change what you get. [data/README.md](data/README.md#mirrors-and-recovery) covers running your own
+- If GitHub is blocked where you live, Settings > Feed mirror takes a second address for the protection data. CallShield asks GitHub first, then the mirror, then falls back to the copy bundled with the app. For ten minutes after GitHub couldn't be reached at all, the mirror goes first. One tap fills in jsDelivr (`https://cdn.jsdelivr.net/gh/SysAdminDoc/CallShield@master/`), which serves the same files and can run up to 12 hours behind. Mirrored files go through the same signature check, so a mirror can't alter the data, though it can hold back updates. [data/README.md](data/README.md#mirrors-and-recovery) covers running your own
 - Quick Settings tile, app shortcuts, home screen widget
 - Protection test validates all layers and permissions, including checker errors and deadline cutoffs
 - Rules surface priority conflicts after sync and edits, with the winning rule and a review path

app/src/main/java/com/sysadmindoc/callshield/CallShieldApp.kt

@@ -102,10 +102,13 @@ class CallShieldApp :
 
             registerCacheInvalidationObservers()
 
+            // Before the read starts, so a worker that runs first waits for it.
+            FeedMirror.startLoading()
             appScope.launch {
                 try {
                     SpamRepository.getInstance(this@CallShieldApp).feedMirrorUrl.collect { FeedMirror.set(it) }
                 } catch (e: Exception) {
+                    FeedMirror.loadFailed()
                     Log.w("CallShieldApp", "Failed to follow the feed mirror setting", e)
                 }
             }

app/src/main/java/com/sysadmindoc/callshield/data/remote/FeedMirror.kt

@@ -1,18 +1,21 @@
 package com.sysadmindoc.callshield.data.remote
 
+import kotlinx.coroutines.CompletableDeferred
+import kotlinx.coroutines.withTimeoutOrNull
 import okhttp3.HttpUrl.Companion.toHttpUrlOrNull
 
 /**
  * A second place to download the protection feeds from, for when
  * raw.githubusercontent.com is blocked where someone lives or the repository
  * moves. Off unless the user sets it.
  *
- * The mirror is tried only after every GitHub branch has failed, and before
- * the callers fall back to the bundled snapshot. It needs no certificate pins
- * of its own: every signed feed is checked against [FeedSignature] whichever
- * host serves it, so a mirror can't hand a device anything the maintainer
- * didn't sign. The database's rollback checks run on the parsed manifest, so
- * they apply to a mirrored one the same way.
+ * The mirror is tried after every GitHub branch has failed, or first for a
+ * few minutes after GitHub couldn't be reached at all, and before the callers
+ * fall back to the bundled snapshot. It needs no certificate pins of its own:
+ * every signed feed is checked against [FeedSignature] whichever host serves
+ * it, so a mirror can't hand a device anything the maintainer didn't sign.
+ * The database's rollback checks run on the parsed manifest, so they apply to
+ * a mirrored one the same way. A mirror can still hold back updates.
  */
 internal object FeedMirror {
     /**
@@ -27,11 +30,47 @@ internal object FeedMirror {
     var baseUrl: String? = null
         private set
 
+    @Volatile
+    private var loading = false
+
+    @Volatile
+    private var loaded = CompletableDeferred<Unit>()
+
+    /**
+     * Called just before the app starts reading the stored setting. Until
+     * [set] or [loadFailed] follows, [awaitLoaded] holds a fetch briefly, so a
+     * worker that starts in a fresh process doesn't build its source list
+     * before the setting arrives and skip the mirror.
+     */
+    fun startLoading() {
+        loading = true
+    }
+
+    /** The stored setting couldn't be read, so stop holding fetches for it. */
+    fun loadFailed() {
+        loaded.complete(Unit)
+    }
+
+    /** Waits up to [timeoutMs] for the stored setting while the app is still reading it. */
+    suspend fun awaitLoaded(timeoutMs: Long = LOAD_WAIT_MS) {
+        if (loading) withTimeoutOrNull(timeoutMs) { loaded.await() }
+    }
+
     /** Sets the mirror from a stored value; anything that isn't a usable base URL clears it. */
     fun set(raw: String?) {
         baseUrl = raw?.let(::normalize)
+        loaded.complete(Unit)
     }
 
+    /** Back to the state of a fresh process, for tests. */
+    internal fun resetForTest() {
+        baseUrl = null
+        loading = false
+        loaded = CompletableDeferred()
+    }
+
+    private const val LOAD_WAIT_MS = 3_000L
+
     /** The mirror's URL for a repository path such as `data/hot_numbers.json`, or null with no mirror. */
     fun urlFor(path: String): String? = baseUrl?.let { it + path }
 

app/src/main/java/com/sysadmindoc/callshield/data/remote/GitHubDataSource.kt

@@ -16,6 +16,11 @@ import okhttp3.Interceptor
 import okhttp3.Request
 import okhttp3.Response
 import okio.Buffer
+import java.io.IOException
+import java.io.InterruptedIOException
+import java.net.ConnectException
+import java.net.NoRouteToHostException
+import java.net.UnknownHostException
 import java.util.concurrent.TimeUnit
 
 internal enum class GitHubFeedFailureReason {
@@ -61,18 +66,23 @@ private val SHARD_PATH_REGEX = Regex("data/spam_number_shards/[0-9a-f]{2}\\.json
 class GitHubDataSource internal constructor(
     /** Tests serve canned responses through this instead of the network. */
     testInterceptor: Interceptor?,
+    /** Tests move time on to end a GitHub outage window. */
+    private val clock: () -> Long = System::currentTimeMillis,
 ) : SpamDataSource,
     HotFeedDataSource {
     constructor() : this(null)
 
     // Derived client with longer timeouts for large database downloads;
     // shares the connection pool with other callers via HttpClient.shared.
+    // The call timeout bounds a whole download: connect and read timeouts
+    // alone let a host that trickles bytes hold a sync open indefinitely.
     private val client =
         HttpClient.shared
             .newBuilder()
             .apply { testInterceptor?.let(::addInterceptor) }
             .connectTimeout(30, TimeUnit.SECONDS)
             .readTimeout(30, TimeUnit.SECONDS)
+            .callTimeout(CALL_TIMEOUT_MINUTES, TimeUnit.MINUTES)
             .build()
 
     private val moshi =
@@ -100,6 +110,13 @@ class GitHubDataSource internal constructor(
     // Cache of resolved default branch per "owner/repo" → (branch, resolvedAtMs).
     private val defaultBranchLock = Any()
     private val defaultBranchCache = mutableMapOf<String, Pair<String, Long>>()
+    private val defaultBranchFailedAt = mutableMapOf<String, Long>()
+
+    // When a GitHub download last failed without an answer: a timeout, a
+    // refused or unreachable connection, a name that won't resolve. For
+    // GITHUB_OUTAGE_MS after that, fetches ask the mirror first.
+    @Volatile
+    private var gitHubUnreachableAt: Long? = null
 
     companion object {
         const val DEFAULT_REPO_OWNER = "SysAdminDoc"
@@ -144,6 +161,18 @@ class GitHubDataSource internal constructor(
         /** How long a resolved default branch stays cached before re-querying. */
         private const val DEFAULT_BRANCH_TTL_MS = 6L * 60L * 60L * 1000L // 6 hours
         private val FALLBACK_BRANCHES = listOf("main", "master")
+
+        /** How long a failed default-branch lookup stands before GitHub's API is asked again. */
+        private const val DEFAULT_BRANCH_RETRY_MS = 10L * 60L * 1000L
+
+        /** How long the mirror goes first after GitHub couldn't be reached at all. */
+        internal const val GITHUB_OUTAGE_MS = 10L * 60L * 1000L
+
+        /** Bounds one whole download, the 16 MB legacy database included. */
+        private const val CALL_TIMEOUT_MINUTES = 5L
+
+        private val COMMIT_SHA_REGEX = Regex("[0-9a-f]{40}(?:[0-9a-f]{24})?")
+        private const val MAX_COMMIT_SHA_BYTES = 128L
         private val RAW_FEED_SPECS =
             mapOf(
                 DATA_PATH to RawFeedSpec("spam database", MAX_SPAM_DATABASE_BYTES),
@@ -563,9 +592,13 @@ class GitHubDataSource internal constructor(
         var lastError: Exception? = null
         for (source in feedSources(path, owner, repo)) {
             try {
-                fetchVerified(path, source)?.let { return Result.success(it) }
+                fetchVerified(path, source)?.let {
+                    if (!source.mirror) gitHubUnreachableAt = null
+                    return Result.success(it)
+                }
                 lastError = moreTelling(lastError, Exception("Empty response body"), source.mirror)
             } catch (e: Exception) {
+                if (!source.mirror && isUnreachable(e)) gitHubUnreachableAt = clock()
                 lastError = moreTelling(lastError, e, source.mirror)
             }
         }
@@ -577,26 +610,59 @@ class GitHubDataSource internal constructor(
      * the user's [FeedMirror] when one is set. Every source carries its own
      * signature URL on the same host and branch, so a copy refused on one
      * source can't borrow another source's signature.
+     *
+     * For [GITHUB_OUTAGE_MS] after GitHub couldn't be reached at all, the
+     * mirror goes first and the branch lookup is skipped. Behind a GitHub
+     * block every GitHub request costs a full timeout, three of them per file,
+     * so a sync of 256 shards would never end. GitHub stays in the list, after
+     * the mirror, in case the mirror fails too.
      */
     private suspend fun feedSources(
         path: String,
         owner: String,
         repo: String,
     ): List<FeedSource> {
-        val github =
-            resolveCandidateBranches(owner, repo).map { branch ->
-                FeedSource(buildRawUrl(owner, repo, branch, path), buildRawUrl(owner, repo, branch, "$path.sig"))
-            }
+        FeedMirror.awaitLoaded()
         val mirror = FeedMirror.urlFor(path)?.let { FeedSource(it, "$it.sig", mirror = true) }
-        return github + listOfNotNull(mirror)
+        if (mirror != null && gitHubRecentlyUnreachable()) {
+            return listOf(mirror) + gitHubSources(path, owner, repo, knownBranches(owner, repo))
+        }
+        return gitHubSources(path, owner, repo, resolveCandidateBranches(owner, repo)) + listOfNotNull(mirror)
     }
 
+    private fun gitHubSources(
+        path: String,
+        owner: String,
+        repo: String,
+        branches: List<String>,
+    ): List<FeedSource> =
+        branches.map { branch ->
+            FeedSource(
+                url = buildRawUrl(owner, repo, branch, path),
+                signatureUrl = buildRawUrl(owner, repo, branch, "$path.sig"),
+                pin = GitHubPin(owner, repo, branch),
+            )
+        }
+
+    private fun gitHubRecentlyUnreachable(): Boolean = gitHubUnreachableAt?.let { clock() - it < GITHUB_OUTAGE_MS } == true
+
+    /** A GitHub download that got no answer at all, as opposed to one GitHub refused or failed. */
+    private fun isUnreachable(error: Exception): Boolean =
+        error is InterruptedIOException ||
+            error is ConnectException ||
+            error is NoRouteToHostException ||
+            error is UnknownHostException
+
     /**
      * [path] from [source], size-checked and signature-checked, or null for
-     * an empty body. The CDN caches a file and its signature separately, for
-     * minutes, so right after a publish it can pair the new body with the old
-     * signature. A signature refusal gets one more try past the cache, and
-     * stands if that copy doesn't verify either.
+     * an empty body. GitHub's CDN caches a file and its signature separately,
+     * for minutes, and leaves the query string out of its cache key, so right
+     * after a publish it can pair the new body with the old signature and no
+     * `?` gets past that. A signature refusal from a GitHub branch is tried
+     * once more at the commit the branch points to: that path never changes,
+     * so its body and signature come from one publish. The refusal stands if
+     * that copy fails too or the commit can't be resolved. A mirror gets no
+     * second try, since its cache turns over on its own schedule.
      */
     private fun fetchVerified(
         path: String,
@@ -606,14 +672,41 @@ class GitHubDataSource internal constructor(
             fetchVerifiedOnce(path, source.url, source.signatureUrl)
         } catch (refused: GitHubFeedValidationException) {
             if (refused.reason != GitHubFeedFailureReason.SIGNATURE) throw refused
-            val pastCache = "cb=${System.nanoTime()}"
+            val pin = source.pin ?: throw refused
+            val commit = resolveCommit(pin) ?: throw refused
             try {
-                fetchVerifiedOnce(path, "${source.url}?$pastCache", "${source.signatureUrl}?$pastCache")
+                fetchVerifiedOnce(
+                    path,
+                    buildRawUrl(pin.owner, pin.repo, commit, path),
+                    buildRawUrl(pin.owner, pin.repo, commit, "$path.sig"),
+                )
             } catch (_: Exception) {
                 throw refused
             }
         }
 
+    /** The commit [pin]'s branch points to, or null when GitHub won't say. One small API call, made only after a refusal. */
+    private fun resolveCommit(pin: GitHubPin): String? =
+        try {
+            val request =
+                Request
+                    .Builder()
+                    .url("$GITHUB_API_BASE/${pin.owner}/${pin.repo}/commits/${pin.branch}")
+                    .header("Accept", "application/vnd.github.sha")
+                    .header("User-Agent", USER_AGENT)
+                    .build()
+            client.newCall(request).execute().use { response ->
+                if (!response.isSuccessful) return null
+                readLimitedBody(response, "GitHub commit API", MAX_COMMIT_SHA_BYTES)
+                    ?.trim()
+                    ?.takeIf(COMMIT_SHA_REGEX::matches)
+            }
+        } catch (_: IOException) {
+            null
+        } catch (_: GitHubFeedValidationException) {
+            null
+        }
+
     private fun fetchVerifiedOnce(
         path: String,
         url: String,
@@ -716,20 +809,38 @@ class GitHubDataSource internal constructor(
         repo: String,
     ): String? {
         val key = "$owner/$repo"
-        val now = System.currentTimeMillis()
+        val now = clock()
         synchronized(defaultBranchLock) {
             val cached = defaultBranchCache[key]
             if (cached != null && now - cached.second < DEFAULT_BRANCH_TTL_MS) {
                 return cached.first
             }
+            // A lookup that just failed isn't repeated for every file in a
+            // sync: behind a GitHub block each attempt costs a full timeout.
+            val failedAt = defaultBranchFailedAt[key]
+            if (failedAt != null && now - failedAt < DEFAULT_BRANCH_RETRY_MS) return cached?.first
         }
-        val resolved = fetchDefaultBranch(owner, repo).getOrNull() ?: return null
+        val resolved = fetchDefaultBranch(owner, repo).getOrNull()
         synchronized(defaultBranchLock) {
+            if (resolved == null) {
+                defaultBranchFailedAt[key] = now
+                return defaultBranchCache[key]?.first
+            }
             defaultBranchCache[key] = resolved to now
+            defaultBranchFailedAt.remove(key)
         }
         return resolved
     }
 
+    /** The candidate branches without asking the API: the cached default, then the fallbacks. */
+    private fun knownBranches(
+        owner: String,
+        repo: String,
+    ): List<String> {
+        val cached = synchronized(defaultBranchLock) { defaultBranchCache["$owner/$repo"]?.first }
+        return listOfNotNull(cached).plus(FALLBACK_BRANCHES).distinct()
+    }
+
     private suspend fun fetchDefaultBranch(
         owner: String,
         repo: String,
@@ -889,6 +1000,14 @@ class GitHubDataSource internal constructor(
         val url: String,
         val signatureUrl: String,
         val mirror: Boolean = false,
+        /** The GitHub branch a source reads, for the retry at its commit. Null for the mirror. */
+        val pin: GitHubPin? = null,
+    )
+
+    private data class GitHubPin(
+        val owner: String,
+        val repo: String,
+        val branch: String,
     )
 
     private data class GitHubReleasePayload(