Skip to content

Commit e3214be

Browse files
committed
fix(pipeline): set aside reports the old Worker misread as NANP
Reports stored before the bare-digit fix can't show whether "+1..." was sent that way, so bare_digit_misreading judges them by two signs: the number's +86 reading is already a known number, or its area code isn't in service and no app sent it (no report id). It only judges reports dated before BARE_DIGIT_FIX_DAY (2026-10-31 UTC, the day after the Worker went out), so a spoofed caller ID in a dead area code reported from then on is kept. The merge drops such reports at intake and sweeps unpublished pending rows that already hold them; a row in the database, an approved number, or a row whose events have all aged out is left alone. The hot list skips the same reports. The watch-list assertion block in test_report_pipeline is a new test; no existing test was wrong. Its runner now sets CALLSHIELD_BARE_DIGIT_FIX_DAY to the fixture's TODAY so the dated rule holds whatever day the suite runs.
1 parent 8f49f4f commit e3214be

5 files changed

Lines changed: 191 additions & 5 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ All notable changes to CallShield will be documented in this file.
8181
- On a phone with two SIMs, the caller ID popup for a call on the second SIM closed the moment it opened. It went by the main SIM's call state, and that reads idle while the other SIM rings. The popup now watches each SIM and closes once every one of them is idle, the same way an outgoing call's warning does.
8282
- On a phone with two SIMs, Answer and hang up now rejects a spam call when either SIM is roaming. Android doesn't tell call screening apps which SIM a call came in on, so CallShield only checked the main one, and a call on a roaming second SIM could get answered abroad, where answering can cost you.
8383
- The report service now takes a number sent without a + only when it has ten digits, read as a US or Canadian number. Any other bare length is turned away, since nothing says which country it's from. A Chinese mobile sent as 15017944158 was being stored as +1 501-794-4158, a line in Arkansas, and could end up on the watch list. The import scripts for North American feeds still read a bare 1 plus ten digits as +1.
84+
- The community report merge and the watch list now set aside reports from before that fix that were most likely misread that way. That's a +1 number whose +86 reading is already a known number, or one in an area code no phone line uses that didn't come from the app. A number already in the database or approved by hand is never set aside, and neither is a report from after the fix.
8485

8586
## v1.12.0 (2026-10-09)
8687

‎scripts/generate_hot_list.py‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
import sys
1818
from datetime import datetime, timezone, timedelta
1919
from pathlib import Path
20-
from phone_normalization import validated_report_number
20+
from phone_normalization import bare_digit_misreading, validated_report_number
2121

2222
from pipeline_io import (
2323
FeedCollapseError,
@@ -152,6 +152,26 @@ def main(argv: list[str] | None = None) -> int:
152152
except Exception as e:
153153
print(f" Skipping {report_file.name}: {e}")
154154

155+
# Reports stored before 2026-10-30 may carry a bare foreign number read as
156+
# +1 (see bare_digit_misreading); the merge drops the same ones.
157+
db_numbers: set[str] = set()
158+
if DB_FILE.exists():
159+
with open(DB_FILE) as f:
160+
db_numbers = {str(entry.get("number", "")) for entry in json.load(f).get("numbers", [])}
161+
known_numbers = db_numbers | {entry[0] for entry in pending}
162+
files_with_id = {token for report_id, _, token in report_ids if report_id}
163+
misread = {
164+
entry[4]
165+
for entry in pending
166+
if entry[0] not in db_numbers
167+
and bare_digit_misreading(
168+
entry[0], known_numbers, entry[4] in files_with_id, entry[1].astimezone(timezone.utc).date().isoformat()
169+
)
170+
}
171+
if misread:
172+
print(f" Dropped {len(misread)} report(s) for +1 numbers that read as bare foreign numbers")
173+
pending = [entry for entry in pending if entry[4] not in misread]
174+
155175
# A report the app sent twice under one id is one reporter, even when the
156176
# resend came from another network and so from another reporter bucket.
157177
resent_reports = find_resent_reports(report_ids)

‎scripts/merge_community_reports.py‎

Lines changed: 51 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,12 @@
1515
from datetime import date, datetime, timedelta, timezone
1616
from pathlib import Path
1717

18-
from phone_normalization import is_plausible_number, split_country_code, validated_report_number
18+
from phone_normalization import (
19+
bare_digit_misreading,
20+
is_plausible_number,
21+
split_country_code,
22+
validated_report_number,
23+
)
1924
from pipeline_io import (
2025
atomic_write_json,
2126
report_queue_digest,
@@ -1036,6 +1041,37 @@ def main(argv: list[str] | None = None):
10361041
if revoked_released or revoked_removed:
10371042
print(f"Revoked approvals: {revoked_removed} rows unpublished, {revoked_released} left to their other sources")
10381043

1044+
# Reports the Worker stored before 2026-10-30 may carry a bare foreign
1045+
# number read as +1. A number the database lists or a maintainer approved
1046+
# stands on its own evidence; an unpublished ledger row doesn't. A row
1047+
# judged here still holds its reports, so a row whose app report aged out
1048+
# isn't mistaken for one no app sent.
1049+
approved_numbers = {approval["number"] for approval in approvals}
1050+
known_numbers = set(existing) | set(pending) | {number for _, number, _ in peeked}
1051+
1052+
def misread(number: str, has_report_id: bool, reported_day: str) -> bool:
1053+
return (
1054+
number not in existing
1055+
and number not in approved_numbers
1056+
and bare_digit_misreading(number, known_numbers, has_report_id, reported_day)
1057+
)
1058+
1059+
misread_rows = sorted(
1060+
number
1061+
for number, state in pending.items()
1062+
if not state["published"]
1063+
and state["events"]
1064+
and misread(
1065+
number,
1066+
any(str(event.get("key", "")).startswith("id:") for event in state["events"]),
1067+
max(event["day"] for event in [*state["events"], *state.get("watch_events", [])]),
1068+
)
1069+
)
1070+
for number in misread_rows:
1071+
del pending[number]
1072+
if misread_rows:
1073+
print(f"Dropped {len(misread_rows)} pending +1 rows that read as bare foreign numbers: {', '.join(misread_rows)}")
1074+
10391075
demoted = 0
10401076
for number, entry in list(existing.items()):
10411077
if canonical_source_ids(entry) != {"community_reports"}:
@@ -1063,6 +1099,7 @@ def main(argv: list[str] | None = None):
10631099
# Counting them by raw value keeps the run auditable without reprinting one
10641100
# line per file for a burst of the same fictional number.
10651101
implausible: Counter[str] = Counter()
1102+
misread_reports: Counter[str] = Counter()
10661103
processed_files = []
10671104
counted_ids: set[str] = set()
10681105
already_merged = 0
@@ -1090,6 +1127,15 @@ def main(argv: list[str] | None = None):
10901127
skipped += 1
10911128
continue
10921129

1130+
parsed_at = parse_reported_at(report.get("reported_at"))
1131+
reported_at = parsed_at.astimezone(timezone.utc).date().isoformat() if parsed_at else today
1132+
reported_at = min(reported_at, today)
1133+
1134+
if misread(number, bool(validated_report_id(report.get("report_id"))), reported_at):
1135+
misread_reports[number] += 1
1136+
processed_files.append(report_file)
1137+
continue
1138+
10931139
if report_file.name in previously_merged:
10941140
processed_files.append(report_file)
10951141
already_merged += 1
@@ -1109,9 +1155,6 @@ def main(argv: list[str] | None = None):
11091155
report_id = validated_report_id(report.get("report_id"))
11101156

11111157
spam_type = report.get("type", "unknown")
1112-
parsed_at = parse_reported_at(report.get("reported_at"))
1113-
reported_at = parsed_at.astimezone(timezone.utc).date().isoformat() if parsed_at else today
1114-
reported_at = min(reported_at, today)
11151158

11161159
# Handle false-positive reports: collect one vote per reporter for review.
11171160
# SECURITY: anonymous not_spam votes may only weaken COMMUNITY rows.
@@ -1395,6 +1438,10 @@ def main(argv: list[str] | None = None):
13951438
print(f"\nRejected as implausible ({skipped} report(s), {len(implausible)} distinct):")
13961439
for raw, count in sorted(implausible.items(), key=lambda item: (-item[1], item[0])):
13971440
print(f" {raw} x{count}")
1441+
if misread_reports:
1442+
print(f"\nDropped as bare foreign numbers read as +1 ({sum(misread_reports.values())} report(s)):")
1443+
for number, count in sorted(misread_reports.items(), key=lambda item: (-item[1], item[0])):
1444+
print(f" {number} x{count}")
13981445

13991446
print(
14001447
f"\nMerged: {added} new, {updated} updated, {skipped} skipped (implausible), "

‎scripts/phone_normalization.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
"""Shared phone normalization helpers for report and import scripts."""
22

3+
import csv
4+
import os
5+
from collections.abc import Container
6+
from functools import lru_cache
7+
from pathlib import Path
8+
9+
NANP_AREA_CODES_FILE = Path(__file__).resolve().parent.parent / "data" / "nanp-area-codes.csv"
10+
# The Worker stopped reading bare digits as NANP on 2026-10-30, so a report
11+
# dated from the next UTC day on went through the fixed Worker. Tests move the
12+
# day with CALLSHIELD_BARE_DIGIT_FIX_DAY.
13+
BARE_DIGIT_FIX_DAY = os.environ.get("CALLSHIELD_BARE_DIGIT_FIX_DAY", "2026-10-31")
14+
315
FORMAT_CONTROL_CODES = {0x200B, 0x200C, 0x200D, 0x200E, 0x200F, 0xFEFF}
416

517
# Every assigned two-digit country calling code (ITU-T E.164). Calling codes are
@@ -148,6 +160,34 @@ def is_plausible_number(e164: str | None) -> bool:
148160
return True
149161

150162

163+
@lru_cache(maxsize=1)
164+
def nanp_area_codes_in_service() -> frozenset[str]:
165+
"""Area codes the pinned NANPA snapshot (data/nanp-area-codes.csv) lists as in service."""
166+
with NANP_AREA_CODES_FILE.open(encoding="utf-8-sig", newline="") as source:
167+
rows = csv.DictReader(line for line in source if not line.startswith("#"))
168+
return frozenset(row["npa"] for row in rows if row["status"] == "in_service")
169+
170+
171+
def bare_digit_misreading(e164: str, known: Container[str], has_report_id: bool, reported_day: str) -> bool:
172+
"""Whether a stored +1 number is most likely a bare foreign number read as NANP.
173+
174+
Until 2026-10-30 the report paths read any bare 11 digits as "+digits", so
175+
a Chinese mobile sent as 15017944158 was stored as +1 501-794-4158, an
176+
Arkansas line. The stored form can't show that, so two signs decide: its
177+
+86 reading is already a known number, or no subscriber can hold it (its
178+
area code isn't in service) and no app sent it (the app has attached a
179+
report id since v1.10.0). A spoofed caller ID can sit in a dead area code
180+
too, which is why the second sign needs both halves, and why a report
181+
dated BARE_DIGIT_FIX_DAY or later is never judged: the fixed Worker stored
182+
it as sent.
183+
"""
184+
if reported_day >= BARE_DIGIT_FIX_DAY or len(e164) != 12 or not e164.startswith("+1"):
185+
return False
186+
if f"+86{e164[1:]}" in known:
187+
return True
188+
return not has_report_id and e164[2:5] not in nanp_area_codes_in_service()
189+
190+
151191
def validated_report_number(raw: str | None) -> str | None:
152192
"""Normalize and plausibility-check a community report number.
153193

‎scripts/test_report_pipeline.py‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,9 @@ def run_script_result(
3535
# The merge asks FCC about pending numbers. Here it reads a saved lookup
3636
# instead, and a test that has no such file gets no complaints.
3737
env["CALLSHIELD_FCC_COMPLAINTS_FILE"] = str(fcc_complaints_file(data_dir))
38+
# Reports dated BASE_DAY came in before the Worker's bare-digit fix and
39+
# reports dated TODAY after it, whatever day the suite runs.
40+
env["CALLSHIELD_BARE_DIGIT_FIX_DAY"] = TODAY
3841
result = subprocess.run(
3942
[sys.executable, str(SCRIPTS_DIR / name), *(args or [])],
4043
cwd=ROOT,
@@ -1164,6 +1167,78 @@ def assert_a_published_number_leaves_the_watch_list(data_dir: Path) -> None:
11641167
assert feed["numbers"] == [] and feed["cleared"] is True, feed
11651168

11661169

1170+
def assert_bare_digit_misreadings_stay_off_the_watch_list(data_dir: Path) -> None:
1171+
"""Reports stored before 2026-10-30 turned bare Chinese mobiles into +1
1172+
numbers (15017944158 into +1 501-794-4158, an Arkansas line). One whose +86
1173+
reading is already listed, or one in a dead area code that no app sent,
1174+
never reaches the watch list, and a ledger row it left behind is dropped.
1175+
Reports the fixed Worker stored, and numbers the database lists or a
1176+
maintainer approved, are taken as sent."""
1177+
def fcc_row(number: str) -> dict:
1178+
return {"number": number, "type": "spam", "reports": 1, "first_seen": BASE_DAY, "last_seen": BASE_DAY,
1179+
"description": "FCC complaint", "sources": ["fcc_complaints"]}
1180+
1181+
twin_listed = "+15017944158"
1182+
dead_area_code = "+18957465363"
1183+
sent_by_the_app = "+18957465364"
1184+
after_the_fix = "+18957465365"
1185+
listed = "+18957465366"
1186+
approved = "+18957465367"
1187+
aged_out = "+18957465368"
1188+
ordinary = "+12122340801"
1189+
twin_listed_later = "+18638657795"
1190+
write_json(
1191+
data_dir / "spam_numbers.json",
1192+
{"version": 1, "updated": TODAY, "sources": ["fcc_complaints"], "prefixes": [],
1193+
"numbers": [fcc_row("+8615017944158"), fcc_row(listed)]},
1194+
)
1195+
write_json(
1196+
data_dir / "spam_numbers_approved.json",
1197+
{"approved": [{"number": approved, "type": "scam", "reviewed_at": BASE_DAY, "reference": "https://example.org/review"}]},
1198+
)
1199+
reported = (datetime.now(timezone.utc) - timedelta(days=1)).replace(hour=10, minute=0, second=0, microsecond=0)
1200+
for number in (twin_listed, dead_area_code, sent_by_the_app, listed, approved, ordinary, twin_listed_later):
1201+
for index, bucket in enumerate(BUCKETS[:2], start=1):
1202+
report_id = f"00000000-0000-4000-8000-{index:012x}" if number == sent_by_the_app else None
1203+
write_report(
1204+
data_dir, f"{number[1:]}-{index}.json", number, bucket, (reported + timedelta(hours=index)).isoformat(),
1205+
device=BUCKETS[index + 1], report_id=report_id,
1206+
)
1207+
for index, bucket in enumerate(BUCKETS[:2], start=1):
1208+
write_report(
1209+
data_dir, f"{after_the_fix[1:]}-{index}.json", after_the_fix, bucket, f"{TODAY}T00:0{index}:00+00:00",
1210+
device=BUCKETS[index + 1],
1211+
)
1212+
# Too old to count as a report, the app's report still leaves a watch row.
1213+
write_report(
1214+
data_dir, f"{aged_out[1:]}.json", aged_out, BUCKETS[0], (reported - timedelta(days=40)).isoformat(),
1215+
report_id="00000000-0000-4000-8000-0000000000ff",
1216+
)
1217+
run_drain(data_dir)
1218+
feed = json.loads((data_dir / "community_watch.json").read_text(encoding="utf-8"))
1219+
watched = sorted(row["number"] for row in feed["numbers"])
1220+
assert watched == [ordinary, twin_listed_later, sent_by_the_app, after_the_fix], feed
1221+
pending = json.loads((data_dir / "community_pending.json").read_text(encoding="utf-8"))["numbers"]
1222+
assert twin_listed not in pending and dead_area_code not in pending, sorted(pending)
1223+
assert not list((data_dir / "reports").glob("*.json")), "the merge left misread reports in the queue"
1224+
assert row_for(data_dir, listed)["reports"] == 3, row_for(data_dir, listed)
1225+
assert row_for(data_dir, approved) is not None and pending_reports_for(data_dir, approved) == 2
1226+
1227+
# Its +86 reading listed later, the ledger row an earlier drain kept goes.
1228+
database = json.loads((data_dir / "spam_numbers.json").read_text(encoding="utf-8"))
1229+
database["numbers"].append(fcc_row("+8618638657795"))
1230+
write_json(data_dir / "spam_numbers.json", database)
1231+
run_script("merge_community_reports.py", data_dir)
1232+
feed = json.loads((data_dir / "community_watch.json").read_text(encoding="utf-8"))
1233+
watched = sorted(row["number"] for row in feed["numbers"])
1234+
assert watched == [ordinary, sent_by_the_app, after_the_fix], feed
1235+
pending = json.loads((data_dir / "community_pending.json").read_text(encoding="utf-8"))["numbers"]
1236+
assert twin_listed_later not in pending, sorted(pending)
1237+
assert pending_reports_for(data_dir, approved) == 2
1238+
# Its id went with its report, which doesn't make it a report no app sent.
1239+
assert pending[aged_out]["watch_events"] and not pending[aged_out]["events"], pending.get(aged_out)
1240+
1241+
11671242
def assert_maintainer_approval_publishes_a_reviewed_number(data_dir: Path) -> None:
11681243
"""Issue #27: one report of a Munich number waits for corroboration no
11691244
imported source can give. The maintainer's review publishes it, and later
@@ -1933,6 +2008,9 @@ def main() -> None:
19332008
with tempfile.TemporaryDirectory() as tmp:
19342009
assert_a_published_number_leaves_the_watch_list(Path(tmp) / "data")
19352010

2011+
with tempfile.TemporaryDirectory() as tmp:
2012+
assert_bare_digit_misreadings_stay_off_the_watch_list(Path(tmp) / "data")
2013+
19362014
with tempfile.TemporaryDirectory() as tmp:
19372015
assert_spam_issues_listed_with_pool_state(Path(tmp) / "data")
19382016

0 commit comments

Comments
 (0)