Skip to content

Commit cd42d7b

Browse files
SysAdminDocSysAdminDoc
committed
feat(reports): send each community report once, and queue the ones that can't go out
The same report kept arriving more than once. Issue #10's came through three times minutes apart, one number arrived 12 times on 2026-08-31, and the Worker's five-minute, address-keyed dedup catches neither. Notification actions also threw away the submission result, so a report made offline or while rate-limited was silently lost. CommunityReportSubmitter now claims the report in a day-long ledger (number + vote type, in the no-backup DataStore) before any network call. A repeat within the day returns ALREADY_SUBMITTED without sending, across a double tap or a process death. A report that fails for a reason waiting can fix (offline, 429, 5xx) goes to CommunityReportWorker, a unique-work outbox keyed by number and type with a network constraint. It's delivered once the connection is back and given up after 12 attempts that actually ran. Every entry point (Lookup, number detail, overlay, notification actions) goes through it and now takes a Context. The Worker's "Duplicate report" 429 now counts as delivered: it means an earlier attempt was stored and only its response was lost, and resending it after the Worker's window would store it twice. A 400 is the plausibility gate refusing the number, so it isn't retried either. contributeSmsSpam had no callers and would have bypassed the ledger, so it's gone. Tests (new only): the ledger's window, keying, future stamps and cap; the Worker answer mapping; the submitter with a real DataStore (a repeat never reaches the transport, the record survives a process death, a correction is a separate report, a transient failure queues once, a refused number isn't queued); and the outbox under the WorkManager test driver (queued twice offline, delivered once when constraints are met, plus retry and give-up rules and SMS indicators). Bypassing the claim, skipping the enqueue or ignoring the duplicate marker each fails them. The submitter test runs under Robolectric because DataStore on a bare JVM reads SDK 0 and falls back to File.renameTo, which can't replace a file on Windows. The two new strings ship with zh-rCN translations.
1 parent 9ebf97c commit cd42d7b

17 files changed

Lines changed: 703 additions & 52 deletions

‎CHANGELOG.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,15 @@ All notable changes to CallShield will be documented in this file.
9191

9292
### Community reports
9393

94-
These take effect when the report Worker is next deployed.
94+
- Each report goes out once. Reporting the same number the same way again
95+
within a day now says you've already reported it instead of sending it
96+
again, even after the app restarts. One number had arrived 12 times in a
97+
single day, and a double-counted report looks like extra corroboration.
98+
- A report made offline, or while the report server is busy, is no longer
99+
lost. It waits and goes out once the connection is back. That includes the
100+
Block and Report buttons on notifications, which used to fail silently.
101+
102+
The rest of this section takes effect when the report Worker is next deployed.
95103

96104
- The Worker counts an IPv6 client by its /64 instead of its full address. One
97105
subscriber could otherwise rotate through addresses to slip past the rate

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -546,7 +546,7 @@ by locale and message type without shipping personal data:
546546
- Quick-access to lookup and protection status
547547

548548
### Community
549-
- **One-tap anonymous contribution** via [Cloudflare Worker](https://callshield-reports.snafumatthew.workers.dev)
549+
- **One-tap anonymous contribution** via [Cloudflare Worker](https://callshield-reports.snafumatthew.workers.dev). Each number and vote goes out at most once a day, and a report made offline is sent when the connection returns
550550
- False positive reporting subtracts votes
551551
- Share spam warnings to any app
552552

@@ -811,7 +811,7 @@ language in [issue #7](https://github.com/SysAdminDoc/CallShield/issues/7).
811811
| URL Safety | Local spam-domain data; optional URLhaus (abuse.ch) |
812812
| Verification | Local Gradle, lint, and release-artifact checks |
813813
| Tests | 1100 JVM unit tests (JUnit) |
814-
| Strings | 1414 string resources and 33 plural groups (translation-ready) |
814+
| Strings | 1416 string resources and 33 plural groups (translation-ready) |
815815
| Accessibility | 100+ content descriptions, 48dp touch targets |
816816
| Min SDK | 29 (Android 10) |
817817
| Target SDK | 36 |

‎app/src/main/java/com/sysadmindoc/callshield/data/CommunityContributor.kt‎

Lines changed: 91 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
package com.sysadmindoc.callshield.data
22

3+
import android.content.Context
34
import com.sysadmindoc.callshield.data.remote.HttpClient
5+
import com.sysadmindoc.callshield.service.CommunityReportWorker
6+
import kotlinx.coroutines.CancellationException
47
import kotlinx.coroutines.Dispatchers
58
import kotlinx.coroutines.withContext
69
import okhttp3.MediaType.Companion.toMediaType
@@ -15,6 +18,11 @@ import java.util.concurrent.TimeUnit
1518
* No user account or API key needed from the app side.
1619
*
1720
* Supports both spam reports AND false positive reports ("not_spam").
21+
*
22+
* A number and vote type is sent at most once a day ([CommunityReportLedger]).
23+
* A report that can't go out now (offline, rate-limited, server error) is
24+
* handed to [CommunityReportWorker], which delivers it once the network is
25+
* back instead of dropping it.
1826
*/
1927
object CommunityContributor {
2028
private const val WORKER_URL = "https://callshield-reports.snafumatthew.workers.dev"
@@ -23,8 +31,13 @@ object CommunityContributor {
2331
private const val MIN_SMS_REPORT_DOMAIN_LENGTH = 5
2432
private const val MAX_SMS_REPORT_DOMAIN_LENGTH = 253
2533
private const val MAX_SMS_REPORT_DOMAIN_LABEL_LENGTH = 63
34+
private const val HTTP_BAD_REQUEST = 400
2635
private const val HTTP_TOO_MANY_REQUESTS = 429
2736
private const val DEFAULT_RETRY_AFTER_SECONDS = 60
37+
private const val ERROR_BODY_PEEK_BYTES = 1024L
38+
39+
/** The Worker's answer to a report it already stored, which is not a failure. */
40+
private const val WORKER_DUPLICATE_MARKER = "Duplicate report"
2841

2942
private val reportDomainPattern = Regex("^[a-z0-9](?:[a-z0-9.-]{0,251}[a-z0-9])?$")
3043
private val urlIndicatorPattern = Regex("^[a-z_]{3,40}$")
@@ -48,6 +61,16 @@ object CommunityContributor {
4861
RATE_LIMITED,
4962
SERVER_ERROR,
5063
NETWORK_ERROR,
64+
65+
/** The same number and vote type went out in the last day; nothing was sent. */
66+
ALREADY_SUBMITTED,
67+
68+
/** Couldn't be sent now; the outbox delivers it when the network allows. */
69+
QUEUED,
70+
;
71+
72+
/** A failure that trying again later can fix. */
73+
val isTransient: Boolean get() = this == RATE_LIMITED || this == SERVER_ERROR || this == NETWORK_ERROR
5174
}
5275

5376
data class ContributeResult(
@@ -62,39 +85,38 @@ object CommunityContributor {
6285
* Report a number as spam.
6386
*/
6487
suspend fun contribute(
88+
context: Context,
6589
number: String,
6690
type: String = "spam",
6791
smsIndicators: SmsContentAnalyzer.SmsReportIndicators? = null,
68-
): ContributeResult = post(number, type, smsIndicators)
69-
70-
/**
71-
* Report SMS spam with body-free URL/domain indicators only.
72-
*/
73-
suspend fun contributeSmsSpam(
74-
number: String,
75-
smsBody: String,
76-
): ContributeResult = post(number, "sms_spam", SmsContentAnalyzer.extractReportableIndicators(smsBody))
92+
): ContributeResult = submitter(context).submit(number, type, smsIndicators)
7793

7894
/**
7995
* Report a false positive — this number is NOT spam.
8096
* The merge script will subtract votes from this number.
8197
*/
82-
suspend fun reportNotSpam(number: String): ContributeResult = post(number, "not_spam")
83-
84-
private suspend fun post(
98+
suspend fun reportNotSpam(
99+
context: Context,
85100
number: String,
101+
): ContributeResult = submitter(context).submit(number, "not_spam", null)
102+
103+
private fun submitter(context: Context): CommunityReportSubmitter {
104+
val appContext = context.applicationContext
105+
return CommunityReportSubmitter(
106+
claim = { number, type, now -> SpamRepository.getInstance(appContext).claimCommunityReport(number, type, now) },
107+
transport = ::send,
108+
enqueue = { number, type, indicators -> CommunityReportWorker.enqueue(appContext, number, type, indicators) },
109+
)
110+
}
111+
112+
/** One attempt to deliver an already-normalized report. Used by the outbox too. */
113+
internal suspend fun send(
114+
normalized: String,
86115
type: String,
87116
smsIndicators: SmsContentAnalyzer.SmsReportIndicators? = null,
88117
): ContributeResult =
89118
withContext(Dispatchers.IO) {
90119
try {
91-
val normalized =
92-
normalizeForReport(number)
93-
?: return@withContext ContributeResult(
94-
success = false,
95-
message = "Invalid number",
96-
outcome = ContributeOutcome.INVALID_NUMBER,
97-
)
98120
val json = buildReportJson(normalized, type, smsIndicators)
99121
val body = json.toRequestBody("application/json".toMediaType())
100122

@@ -106,28 +128,15 @@ object CommunityContributor {
106128
.build()
107129

108130
client.newCall(request).execute().use { response ->
109-
if (response.isSuccessful) {
110-
if (type == "not_spam") {
111-
ContributeResult(true, "Reported as not spam", ContributeOutcome.REPORTED_NOT_SPAM)
112-
} else {
113-
ContributeResult(true, "Contributed anonymously", ContributeOutcome.REPORTED_SPAM)
114-
}
115-
} else if (response.code == HTTP_TOO_MANY_REQUESTS) {
116-
val retryAfter = response.header("Retry-After")?.toIntOrNull() ?: DEFAULT_RETRY_AFTER_SECONDS
117-
ContributeResult(
118-
success = false,
119-
message = "Too many reports. Please wait ${retryAfter}s and try again.",
120-
outcome = ContributeOutcome.RATE_LIMITED,
121-
retryAfterSeconds = retryAfter,
122-
)
123-
} else {
124-
ContributeResult(
125-
success = false,
126-
message = "Server error (${response.code})",
127-
outcome = ContributeOutcome.SERVER_ERROR,
128-
)
129-
}
131+
resultFor(
132+
code = response.code,
133+
type = type,
134+
body = if (response.isSuccessful) "" else response.peekBody(ERROR_BODY_PEEK_BYTES).string(),
135+
retryAfterHeader = response.header("Retry-After"),
136+
)
130137
}
138+
} catch (e: CancellationException) {
139+
throw e
131140
} catch (e: Exception) {
132141
ContributeResult(
133142
success = false,
@@ -137,6 +146,48 @@ object CommunityContributor {
137146
}
138147
}
139148

149+
/** How the app reads the Worker's answer to one report. */
150+
internal fun resultFor(
151+
code: Int,
152+
type: String,
153+
body: String,
154+
retryAfterHeader: String?,
155+
): ContributeResult {
156+
// A duplicate answer means an earlier attempt was stored and only its
157+
// response was lost. Sending it again after the Worker's window would
158+
// store it twice.
159+
val duplicate = code == HTTP_TOO_MANY_REQUESTS && WORKER_DUPLICATE_MARKER in body
160+
return when {
161+
code in 200..299 || duplicate -> {
162+
if (type == "not_spam") {
163+
ContributeResult(true, "Reported as not spam", ContributeOutcome.REPORTED_NOT_SPAM)
164+
} else {
165+
ContributeResult(true, "Contributed anonymously", ContributeOutcome.REPORTED_SPAM)
166+
}
167+
}
168+
169+
code == HTTP_TOO_MANY_REQUESTS -> {
170+
val retryAfter = retryAfterHeader?.toIntOrNull() ?: DEFAULT_RETRY_AFTER_SECONDS
171+
ContributeResult(
172+
success = false,
173+
message = "Too many reports. Please wait ${retryAfter}s and try again.",
174+
outcome = ContributeOutcome.RATE_LIMITED,
175+
retryAfterSeconds = retryAfter,
176+
)
177+
}
178+
179+
// The Worker's plausibility gate refused the number, and sending it
180+
// again later gets the same answer.
181+
code == HTTP_BAD_REQUEST -> {
182+
ContributeResult(false, "Rejected as invalid", ContributeOutcome.INVALID_NUMBER)
183+
}
184+
185+
else -> {
186+
ContributeResult(false, "Server error ($code)", ContributeOutcome.SERVER_ERROR)
187+
}
188+
}
189+
}
190+
140191
private fun escapeJson(value: String): String =
141192
value
142193
.replace("\\", "\\\\")

‎app/src/main/java/com/sysadmindoc/callshield/data/CommunityReportLedger.kt‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
package com.sysadmindoc.callshield.data
2+
3+
import java.util.concurrent.TimeUnit
4+
5+
/**
6+
* The reports this device sent or queued in the last day, one entry per
7+
* number and vote type, kept as `number|type|epochMillis` strings.
8+
*
9+
* Without it the same report went out again whenever someone tapped report
10+
* twice or retried after a slow response: issue #10's report arrived three
11+
* times minutes apart, and one number arrived 12 times on 2026-08-31. The
12+
* Worker's own dedup lasts five minutes and keys on the network address, so it
13+
* catches neither.
14+
*/
15+
internal object CommunityReportLedger {
16+
val WINDOW_MILLIS = TimeUnit.HOURS.toMillis(24)
17+
const val MAX_ENTRIES = 256
18+
private const val SEPARATOR = "|"
19+
20+
/**
21+
* Entries still inside the window. A stamp from the future is dropped: it
22+
* was written under a wrong clock, and keeping it would block that report
23+
* for as long as the clock was off.
24+
*/
25+
fun prune(
26+
entries: Set<String>,
27+
now: Long,
28+
): Set<String> =
29+
entries.filterTo(LinkedHashSet()) { entry ->
30+
val stamp = entry.substringAfterLast(SEPARATOR).toLongOrNull() ?: return@filterTo false
31+
now - stamp in 0L until WINDOW_MILLIS
32+
}
33+
34+
fun contains(
35+
entries: Set<String>,
36+
number: String,
37+
type: String,
38+
): Boolean = entries.any { it.startsWith(keyOf(number, type) + SEPARATOR) }
39+
40+
/** Adds the report, dropping the oldest entries past [MAX_ENTRIES]. */
41+
fun add(
42+
entries: Set<String>,
43+
number: String,
44+
type: String,
45+
now: Long,
46+
): Set<String> {
47+
val next = entries + "${keyOf(number, type)}$SEPARATOR$now"
48+
if (next.size <= MAX_ENTRIES) return next
49+
return next
50+
.sortedBy { it.substringAfterLast(SEPARATOR).toLongOrNull() ?: Long.MIN_VALUE }
51+
.takeLast(MAX_ENTRIES)
52+
.toSet()
53+
}
54+
55+
private fun keyOf(
56+
number: String,
57+
type: String,
58+
) = "$number$SEPARATOR$type"
59+
}

‎app/src/main/java/com/sysadmindoc/callshield/data/CommunityReportSubmitter.kt‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
package com.sysadmindoc.callshield.data
2+
3+
import com.sysadmindoc.callshield.data.CommunityContributor.ContributeOutcome
4+
import com.sysadmindoc.callshield.data.CommunityContributor.ContributeResult
5+
import com.sysadmindoc.callshield.data.SmsContentAnalyzer.SmsReportIndicators
6+
7+
/**
8+
* Sends a community report at most once a day per number and vote type, and
9+
* hands one it can't send now to the outbox instead of dropping it.
10+
*
11+
* The claim comes before the network, so a second tap while the first is
12+
* still in flight is suppressed too. A report that was claimed and then
13+
* queued stays claimed: the outbox will deliver it.
14+
*/
15+
internal class CommunityReportSubmitter(
16+
private val claim: suspend (number: String, type: String, now: Long) -> Boolean,
17+
private val transport: suspend (number: String, type: String, indicators: SmsReportIndicators?) -> ContributeResult,
18+
private val enqueue: (number: String, type: String, indicators: SmsReportIndicators?) -> Unit,
19+
private val clock: () -> Long = System::currentTimeMillis,
20+
) {
21+
suspend fun submit(
22+
number: String,
23+
type: String,
24+
indicators: SmsReportIndicators?,
25+
): ContributeResult {
26+
val normalized =
27+
CommunityContributor.normalizeForReport(number)
28+
?: return ContributeResult(false, "Invalid number", ContributeOutcome.INVALID_NUMBER)
29+
if (!claim(normalized, type, clock())) {
30+
// The community already has this report from this device, so it
31+
// counts as done rather than as a failure.
32+
return ContributeResult(true, "Already submitted", ContributeOutcome.ALREADY_SUBMITTED)
33+
}
34+
val result = transport(normalized, type, indicators)
35+
if (!result.outcome.isTransient) return result
36+
enqueue(normalized, type, indicators)
37+
return ContributeResult(true, "Queued after: ${result.message}", ContributeOutcome.QUEUED, result.retryAfterSeconds)
38+
}
39+
}

‎app/src/main/java/com/sysadmindoc/callshield/data/SpamRepository.kt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,7 @@ class SpamRepository(
206206
internal val KEY_AUTO_CLEANUP = booleanPreferencesKey("auto_cleanup_enabled")
207207
internal val KEY_CLEANUP_DAYS = intPreferencesKey("cleanup_retention_days")
208208
internal val KEY_ABSTRACT_API_KEY = stringPreferencesKey("abstract_api_key")
209+
internal val KEY_COMMUNITY_REPORT_LEDGER = stringSetPreferencesKey("community_report_ledger")
209210
internal val KEY_EXTERNAL_BLOCKLIST_SUBSCRIPTIONS =
210211
stringPreferencesKey("external_blocklist_subscriptions")
211212
val KEY_ML_SCORER = booleanPreferencesKey("ml_scorer_enabled")
@@ -652,6 +653,13 @@ class SpamRepository(
652653

653654
suspend fun refreshDueExternalBlocklists(now: Long = System.currentTimeMillis()): List<ExternalBlocklistRefreshOutcome> = syncRepository.refreshDueExternalBlocklists(now)
654655

656+
/** False when this number and vote type was already reported in the last day. */
657+
suspend fun claimCommunityReport(
658+
number: String,
659+
type: String,
660+
now: Long = System.currentTimeMillis(),
661+
): Boolean = settingsRepository.claimCommunityReport(number, type, now)
662+
655663
// ── Blocklist management ───────────────────────────────────────────
656664
suspend fun blockNumber(
657665
number: String,

‎app/src/main/java/com/sysadmindoc/callshield/data/repository/SettingsRepository.kt‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import com.sysadmindoc.callshield.data.CallCategory
1515
import com.sysadmindoc.callshield.data.CallbackDetector
1616
import com.sysadmindoc.callshield.data.CategoryCallAction
1717
import com.sysadmindoc.callshield.data.CategoryCallPolicy
18+
import com.sysadmindoc.callshield.data.CommunityReportLedger
1819
import com.sysadmindoc.callshield.data.ContactGroupCatalog
1920
import com.sysadmindoc.callshield.data.MessageCapabilitySource
2021
import com.sysadmindoc.callshield.data.MessageCapabilityStatus
@@ -620,6 +621,31 @@ class SettingsRepository(
620621
preferences[SpamRepository.KEY_FEED_TRUST_NOTICE_VERSION] = version
621622
}
622623

624+
/**
625+
* Records a community report unless the same number and vote type was
626+
* already reported in the last day. One edit does both, so two taps that
627+
* race each other can't both claim it. Kept in the no-backup store: it is
628+
* a day of this device's activity, not a setting.
629+
*/
630+
suspend fun claimCommunityReport(
631+
number: String,
632+
type: String,
633+
now: Long,
634+
): Boolean {
635+
var claimed = false
636+
privateDataStore.edit { preferences ->
637+
val live =
638+
CommunityReportLedger.prune(
639+
preferences[SpamRepository.KEY_COMMUNITY_REPORT_LEDGER].orEmpty(),
640+
now,
641+
)
642+
claimed = !CommunityReportLedger.contains(live, number, type)
643+
preferences[SpamRepository.KEY_COMMUNITY_REPORT_LEDGER] =
644+
if (claimed) CommunityReportLedger.add(live, number, type, now) else live
645+
}
646+
return claimed
647+
}
648+
623649
suspend fun readExternalBlocklistSubscriptions(): List<ExternalBlocklistSubscription> =
624650
decodeExternalBlocklistSubscriptions(
625651
dataStore.data.first()[SpamRepository.KEY_EXTERNAL_BLOCKLIST_SUBSCRIPTIONS],

0 commit comments

Comments
 (0)