Skip to content

Commit 3893913

Browse files
committed
feat: scripts/device_smoke.py runs the release's S22 pass over adb
The v1.12.0 pass was driven by hand from uiautomator dumps and caught the number screen's Undo under the navigation bar, which the JVM tests missed. The script resets to the tab bar before each step, so one failure doesn't cascade, and Report runs only behind --report since it sends real data. test_device_smoke.py covers the dump parsing and repo lookups and joins the pipeline test list. 9 of 9 steps passed on the S22.
1 parent dbc5cee commit 3893913

5 files changed

Lines changed: 404 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,11 @@ All notable changes to CallShield will be documented in this file.
1616
- Lookup's result card now offers to block the number's whole area code,
1717
with Undo, the same way a number's own screen does. Before, you had to
1818
scroll down and open the full detail to find it.
19+
- New `scripts/device_smoke.py` runs the release's phone checks over adb
20+
and prints PASS or FAIL for each: sync, What's new, the recommended
21+
lists, both area code offers with Undo, a number with no offer, and Not
22+
spam's Undo. `--report` adds Report's block and Undo, and it sends a
23+
real report.
1924

2025
## v1.12.0 (2026-10-09)
2126

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -542,6 +542,14 @@ After `gh release create` has published the tag, write and sign the release noti
542542
python scripts/write_app_release.py
543543
```
544544

545+
With the new APK installed on a phone, run the device pass. It prints PASS or FAIL for each step (sync, What's new, the recommended lists, the area code offers and their Undo, Not spam's Undo) and exits 1 if any failed:
546+
547+
```powershell
548+
python scripts/device_smoke.py --serial <adb serial>
549+
```
550+
551+
`--report <number>` also checks that Report blocks the number on the phone. It sends a real anonymous report, so only use it for a number you mean to report.
552+
545553
`verifyReleaseSbom` also writes `<release-apk-stem>.cdx.json`,
546554
`<release-apk-stem>.provenance.json`, and `<release-apk-stem>.sha256` beside the
547555
APK. The SBOM contains the exact `releaseRuntimeClasspath` coordinates from

‎scripts/device_smoke.py‎

Lines changed: 344 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,344 @@
1+
"""Release smoke pass on a real phone: the checks a person used to tap through.
2+
3+
Run it after installing the release APK, with the app set up and the phone
4+
unlocked. It drives the English UI through adb and uiautomator, prints PASS or
5+
FAIL for each step, and exits non-zero if any step failed.
6+
7+
py -3 scripts/device_smoke.py --serial R5CT139QJ5F
8+
py -3 scripts/device_smoke.py --serial R5CT139QJ5F --report +13152328257
9+
10+
Every block it makes is undone, and the script checks that it was. Not spam
11+
goes out five seconds after its tap unless Undo is tapped first, so that step
12+
fails loudly if Undo isn't found. --report sends a real anonymous report for
13+
the number given; pick one already in the database, or one this phone
14+
reported in the last day, which the app won't send twice.
15+
"""
16+
17+
from __future__ import annotations
18+
19+
import argparse
20+
import html
21+
import json
22+
import re
23+
import subprocess
24+
import sys
25+
import time
26+
from dataclasses import dataclass
27+
from pathlib import Path
28+
29+
REPO = Path(__file__).resolve().parent.parent
30+
PACKAGE = "com.sysadmindoc.callshield"
31+
ACTIVITY = f"{PACKAGE}/.ui.MainActivity"
32+
DUMP = "/sdcard/callshield-smoke.xml"
33+
34+
# A Paris number has the area code +33 1; a Japanese one under +81 42 has none
35+
# CallShield can block safely (GeographicAreaCodes drops 4, and 42 has gaps).
36+
AREA_NUMBER, AREA_LABEL = "+33145230001", "+33 1"
37+
NO_AREA_NUMBER = "+81427003737"
38+
39+
40+
@dataclass(frozen=True)
41+
class Node:
42+
label: str
43+
x: int
44+
y: int
45+
46+
47+
def parse_nodes(xml: str) -> list[Node]:
48+
"""Every node with text or a content description, at the center of its bounds."""
49+
nodes = []
50+
for match in re.finditer(r"<node [^>]*>", xml):
51+
attrs = dict(re.findall(r'([\w-]+)="([^"]*)"', match.group(0)))
52+
label = attrs.get("text") or attrs.get("content-desc") or ""
53+
bounds = [int(n) for n in re.findall(r"\d+", attrs.get("bounds", ""))]
54+
if label and len(bounds) == 4:
55+
# Numbers come wrapped in Unicode isolates (⁦...⁩) for right-to-left layouts.
56+
label = html.unescape(label).replace("⁦", "").replace("⁩", "")
57+
nodes.append(Node(label, (bounds[0] + bounds[2]) // 2, (bounds[1] + bounds[3]) // 2))
58+
return nodes
59+
60+
61+
def bottom_most(nodes: list[Node], label: str) -> Node | None:
62+
"""The tab bar's entry: a tab's name also heads its screen, higher up."""
63+
exact = [n for n in nodes if n.label == label]
64+
return max(exact, key=lambda n: n.y, default=None)
65+
66+
67+
def app_version(gradle: Path = REPO / "app" / "build.gradle.kts") -> str:
68+
match = re.search(r'versionName = "([^"]+)"', gradle.read_text(encoding="utf-8"))
69+
if not match:
70+
raise SystemExit(f"no versionName in {gradle}")
71+
return match.group(1)
72+
73+
74+
def catalog_names(catalog: Path = REPO / "data" / "list_catalog.json") -> list[str]:
75+
return [entry["name"] for entry in json.loads(catalog.read_text(encoding="utf-8"))["lists"]]
76+
77+
78+
class Phone:
79+
def __init__(self, adb: str, serial: str):
80+
self.adb, self.serial = adb, serial
81+
size = re.search(r"(\d+)x(\d+)", self.shell("wm", "size"))
82+
self.width, self.height = (int(size.group(1)), int(size.group(2))) if size else (1080, 2340)
83+
84+
def shell(self, *args: str) -> str:
85+
out = subprocess.run([self.adb, "-s", self.serial, "shell", *args], capture_output=True, check=False)
86+
return out.stdout.decode("utf-8", "replace")
87+
88+
def nodes(self) -> list[Node]:
89+
self.shell("uiautomator", "dump", DUMP)
90+
return parse_nodes(self.shell("cat", DUMP))
91+
92+
def tap(self, node: Node) -> None:
93+
self.shell("input", "tap", str(node.x), str(node.y))
94+
time.sleep(0.8)
95+
96+
def scroll(self, down: bool = True) -> None:
97+
# The left third, clear of a picture-in-picture window in the corner.
98+
x = str(self.width // 4)
99+
low, high = str(self.height * 3 // 4), str(self.height * 3 // 10)
100+
self.shell("input", "swipe", x, low if down else high, x, high if down else low, "300")
101+
time.sleep(0.6)
102+
103+
def find(self, label: str, contains: bool = False, swipes: int = 0, down: bool = True) -> Node | None:
104+
for attempt in range(swipes + 1):
105+
for node in self.nodes():
106+
if node.label == label or (contains and label in node.label):
107+
return node
108+
if attempt < swipes:
109+
self.scroll(down)
110+
return None
111+
112+
def tap_label(self, label: str, contains: bool = False, swipes: int = 0) -> None:
113+
node = self.find(label, contains, swipes)
114+
if node is None:
115+
raise StepFailed(f"no {label!r} on screen")
116+
self.tap(node)
117+
118+
def tab(self, name: str) -> None:
119+
node = bottom_most(self.nodes(), name)
120+
if node is None:
121+
raise StepFailed(f"no {name} tab")
122+
self.tap(node)
123+
124+
def back(self) -> None:
125+
# The system Back key: a screen's own back arrow scrolls away with it.
126+
self.shell("input", "keyevent", "4")
127+
time.sleep(0.8)
128+
129+
def reset(self) -> None:
130+
"""Back out of a number's screen, or relaunch, until the tab bar shows."""
131+
for attempt in range(5):
132+
nodes = self.nodes()
133+
if bottom_most(nodes, "Home") and bottom_most(nodes, "More"):
134+
return
135+
if attempt < 3:
136+
self.back()
137+
else:
138+
self.shell("am", "start", "-n", ACTIVITY)
139+
time.sleep(2)
140+
raise StepFailed("couldn't get back to the tab bar")
141+
142+
def to_top(self) -> None:
143+
# Until the screen stops changing, which also waits out a fling.
144+
before: list[str] = []
145+
for _ in range(15):
146+
self.scroll(down=False)
147+
labels = [n.label for n in self.nodes()]
148+
if labels == before:
149+
return
150+
before = labels
151+
152+
153+
class StepFailed(Exception):
154+
pass
155+
156+
157+
def lookup(phone: Phone, number: str) -> None:
158+
phone.tab("Lookup")
159+
phone.to_top()
160+
clear = phone.find("Close")
161+
if clear:
162+
phone.tap(clear)
163+
phone.tap_label("Phone number")
164+
phone.shell("input", "text", number)
165+
phone.shell("input", "keyevent", "111") # Escape closes the keyboard.
166+
phone.tap_label("Check number")
167+
time.sleep(1.5)
168+
169+
170+
def undo_snackbar(phone: Phone, what: str) -> None:
171+
undo = phone.find("Undo")
172+
if undo is None:
173+
raise StepFailed(f"no Undo after {what}")
174+
phone.tap(undo)
175+
176+
177+
def wildcards_empty(phone: Phone) -> None:
178+
phone.tab("Rules")
179+
phone.tap_label("Wildcards")
180+
if phone.find("No wildcard rules") is None:
181+
raise StepFailed("a wildcard rule is still there")
182+
183+
184+
def step_version(phone: Phone, version: str) -> None:
185+
installed = re.search(r"versionName=(\S+)", phone.shell("dumpsys", "package", PACKAGE))
186+
if not installed or installed.group(1) != version:
187+
raise StepFailed(f"installed {installed.group(1) if installed else 'nothing'}, expected {version}")
188+
189+
190+
def step_sync(phone: Phone) -> None:
191+
phone.tab("Home")
192+
phone.to_top()
193+
# "Synced just now" shows for about a minute after a sync ends.
194+
for _ in range(2):
195+
phone.tap_label("Sync database")
196+
for _ in range(12):
197+
if phone.find("Synced just now"):
198+
return
199+
time.sleep(3)
200+
raise StepFailed("never showed Synced just now")
201+
202+
203+
def step_whats_new(phone: Phone, version: str) -> None:
204+
phone.tab("More")
205+
phone.tap_label("What's new", swipes=8)
206+
labels = [n.label for n in phone.nodes()]
207+
phone.back()
208+
if f"v{version}" not in labels or "LATEST" not in labels:
209+
raise StepFailed(f"latest release isn't v{version}")
210+
211+
212+
def step_catalog(phone: Phone, names: list[str]) -> None:
213+
phone.tab("More")
214+
phone.to_top()
215+
phone.tap_label("Settings")
216+
phone.to_top()
217+
phone.tap_label("Advanced")
218+
seen: set[str] = set()
219+
try:
220+
for _ in range(40):
221+
labels = [n.label for n in phone.nodes()]
222+
seen.update(label for label in labels if label in names)
223+
if len(seen) == len(names) or "FEED MIRROR" in labels:
224+
break
225+
phone.scroll()
226+
finally:
227+
phone.to_top()
228+
phone.tap_label("Basic")
229+
phone.back()
230+
missing = [n for n in names if n not in seen]
231+
if missing:
232+
raise StepFailed(f"Recommended lists is missing {missing}")
233+
234+
235+
def step_lookup_area_code(phone: Phone) -> None:
236+
lookup(phone, AREA_NUMBER)
237+
offer = phone.find(f"Block all {AREA_LABEL} numbers", swipes=2)
238+
if offer is None:
239+
raise StepFailed(f"Lookup didn't offer {AREA_LABEL}")
240+
phone.tap(offer)
241+
undo_snackbar(phone, "the area code block")
242+
wildcards_empty(phone)
243+
244+
245+
def step_no_area_code(phone: Phone) -> None:
246+
lookup(phone, NO_AREA_NUMBER)
247+
if phone.find("Block all", contains=True, swipes=2):
248+
raise StepFailed(f"Lookup offered an area code for {NO_AREA_NUMBER}")
249+
phone.tap_label("Open full detail", swipes=2)
250+
found = phone.find("Block all", contains=True, swipes=2)
251+
phone.back()
252+
if found:
253+
raise StepFailed(f"the number's screen offered an area code for {NO_AREA_NUMBER}")
254+
255+
256+
def step_detail_area_code(phone: Phone) -> None:
257+
lookup(phone, AREA_NUMBER)
258+
phone.tap_label("Open full detail", swipes=2)
259+
phone.tap_label(f"Block all {AREA_LABEL} numbers", swipes=3)
260+
undo_snackbar(phone, "the area code block on the number's screen")
261+
phone.back()
262+
wildcards_empty(phone)
263+
264+
265+
def my_reports(phone: Phone) -> list[str]:
266+
phone.tab("More")
267+
phone.to_top()
268+
phone.tap_label("My reports", swipes=6)
269+
labels = [n.label for n in phone.nodes()]
270+
phone.back()
271+
return labels
272+
273+
274+
def step_not_spam_undo(phone: Phone, number: str) -> None:
275+
before = my_reports(phone)
276+
lookup(phone, number)
277+
phone.tap_label("Open full detail", swipes=3)
278+
phone.tap_label("Not spam", swipes=6)
279+
undo_snackbar(phone, "Not spam")
280+
phone.back()
281+
time.sleep(6) # Past the five seconds a missed Undo would have taken.
282+
if my_reports(phone) != before:
283+
raise StepFailed("My reports changed after Not spam's Undo")
284+
285+
286+
def step_report_blocks(phone: Phone, number: str) -> None:
287+
lookup(phone, number)
288+
phone.tap_label("Open full detail", swipes=3)
289+
phone.tap_label("Report", swipes=3)
290+
time.sleep(2)
291+
message = phone.find("Blocked on this phone.", contains=True)
292+
if message is None:
293+
raise StepFailed("Report's message doesn't say it blocked the number")
294+
undo_snackbar(phone, "Report's block")
295+
time.sleep(1)
296+
blocked = phone.find("Unblock", swipes=3, down=False)
297+
phone.back()
298+
if blocked:
299+
raise StepFailed("the number is still blocked after Undo")
300+
301+
302+
def main() -> int:
303+
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
304+
parser.add_argument("--serial", required=True, help="adb serial of the phone")
305+
parser.add_argument("--adb", default="adb", help="path to adb")
306+
parser.add_argument("--report", metavar="NUMBER", help="also check that Report blocks NUMBER here (sends a real report)")
307+
parser.add_argument("--database-number", default="+13152328257", help="a number in the shared database, for Not spam")
308+
args = parser.parse_args()
309+
310+
phone = Phone(args.adb, args.serial)
311+
version = app_version()
312+
phone.shell("input", "keyevent", "KEYCODE_WAKEUP")
313+
phone.shell("am", "start", "-n", ACTIVITY)
314+
time.sleep(2)
315+
316+
steps = [
317+
("installed version", lambda: step_version(phone, version)),
318+
("database sync", lambda: step_sync(phone)),
319+
("What's new shows this version", lambda: step_whats_new(phone, version)),
320+
("Recommended lists match the catalog", lambda: step_catalog(phone, catalog_names())),
321+
("Lookup offers +33 1, and Undo removes it", lambda: step_lookup_area_code(phone)),
322+
("no area code offer for +81 42", lambda: step_no_area_code(phone)),
323+
("number's screen offers +33 1, and Undo removes it", lambda: step_detail_area_code(phone)),
324+
("Not spam then Undo sends nothing", lambda: step_not_spam_undo(phone, args.database_number)),
325+
]
326+
if args.report:
327+
steps.append(("Report blocks the number, and Undo unblocks it", lambda: step_report_blocks(phone, args.report)))
328+
329+
failed = 0
330+
for name, step in steps:
331+
try:
332+
phone.reset()
333+
step()
334+
print(f"PASS {name}")
335+
except StepFailed as error:
336+
failed += 1
337+
print(f"FAIL {name}: {error}")
338+
phone.shell("rm", "-f", DUMP)
339+
print(f"{len(steps) - failed} of {len(steps)} passed")
340+
return 1 if failed else 0
341+
342+
343+
if __name__ == "__main__":
344+
sys.exit(main())

‎scripts/run-pipeline-tests.ps1‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ if ($node) {
8181
# from a non-interactive shell with no console attached (CI-style invocation).
8282
$python = Get-Tool @('python3', 'python')
8383
if ($python) {
84-
foreach ($test in @('test_phone_normalization.py', 'test_report_dedup.py', 'test_report_pipeline.py', 'test_pipeline_liveness.py', 'test_model_calibration.py', 'test_ml_feature_contract.py', 'test_release_sbom.py', 'test_check_translations.py', 'test_source_registry.py', 'test_spam_shards.py', 'test_incremental_sources.py', 'test_regional_prefixes.py', 'test_release_drift.py', 'test_check_live_pins.py', 'test_probe_live_sources.py', 'test_feed_signing.py', 'test_write_app_release.py', 'test_list_catalog.py')) {
84+
foreach ($test in @('test_phone_normalization.py', 'test_report_dedup.py', 'test_report_pipeline.py', 'test_pipeline_liveness.py', 'test_model_calibration.py', 'test_ml_feature_contract.py', 'test_release_sbom.py', 'test_check_translations.py', 'test_source_registry.py', 'test_spam_shards.py', 'test_incremental_sources.py', 'test_regional_prefixes.py', 'test_release_drift.py', 'test_check_live_pins.py', 'test_probe_live_sources.py', 'test_feed_signing.py', 'test_write_app_release.py', 'test_list_catalog.py', 'test_device_smoke.py')) {
8585
$path = Join-Path $PSScriptRoot $test
8686
if (-not (Test-Path $path)) { continue }
8787
Write-Host "Running $test..."

0 commit comments

Comments
 (0)