Skip to content

Commit 6bda0b9

Browse files
committed
feat(pipeline): warn a week before a feed source goes stale
Imports are manual and the liveness run is weekly, so a source only showed up once it was past stale_after_days: FTC's newest record lapsed 2026-10-07 and nothing would have said so before the 2026-10-12 run. warn_source_staleness reports a regular-cadence source within seven days of its limit (newest record date for FCC and FTC, last import otherwise). main() exits 3 for warnings only, and the workflow opens or updates the pipeline-stalled issue with a warning title without failing the job. A stale source still fails. Tests cover 6, 7, 13, 14 and 15 days, a 30-day source and the exit code; a mutant window of 6 days fails them.
1 parent b5a463f commit 6bda0b9

4 files changed

Lines changed: 145 additions & 14 deletions

File tree

‎.github/workflows/pipeline-liveness.yml‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,10 @@ name: Pipeline liveness
33
# Weekly check that community reports are being consumed and that upstream
44
# sources with a timetable are still being imported. A failure opens one issue
55
# labelled pipeline-stalled, later failures comment on it, and the first passing
6-
# run closes it. The queue is expected to go stale between drains, so this lives
7-
# apart from the push workflow instead of turning it permanently red.
6+
# run closes it. A source within a week of going stale (exit code 3) opens or
7+
# updates the same issue as a warning without failing the job. The queue is
8+
# expected to go stale between drains, so this lives apart from the push
9+
# workflow instead of turning it permanently red.
810

911
on:
1012
schedule:
@@ -38,7 +40,7 @@ jobs:
3840
cat liveness.txt
3941
4042
- name: Open or update the tracking issue
41-
if: steps.check.outputs.status != '0'
43+
if: steps.check.outputs.status != '0' && steps.check.outputs.status != '3'
4244
env:
4345
GH_TOKEN: ${{ github.token }}
4446
run: |
@@ -52,6 +54,20 @@ jobs:
5254
fi
5355
exit 1
5456
57+
- name: Warn about a source close to stale
58+
if: steps.check.outputs.status == '3'
59+
env:
60+
GH_TOKEN: ${{ github.token }}
61+
run: |
62+
body="$(printf 'The weekly pipeline check passed on %s, with a warning: a source goes stale within a week.\n\n```\n%s\n```\n' "$(date -u +%Y-%m-%d)" "$(cat liveness.txt)")"
63+
number="$(gh issue list --state open --label pipeline-stalled --json number --jq '.[0].number // empty')"
64+
if [ -n "$number" ]; then
65+
gh issue comment "$number" --body "$body"
66+
else
67+
gh label create pipeline-stalled --color B60205 --description "The weekly pipeline check is failing" 2>/dev/null || true
68+
gh issue create --title "Report pipeline warning: a source goes stale within a week" --label pipeline-stalled --body "$body"
69+
fi
70+
5571
- name: Close the tracking issue once healthy
5672
if: steps.check.outputs.status == '0'
5773
env:

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,10 @@ All notable changes to CallShield will be documented in this file.
137137
Singapore and Thailand. You can turn on "Allow apps from unverified
138138
developers" in Developer options and wait a day, or install over USB with
139139
`adb install`.
140+
- The weekly pipeline check warns a week before an upstream source goes
141+
stale, without failing, so an overdue FCC or FTC import shows up while
142+
there is still time to run it. FTC went stale on 2026-10-07 and nothing
143+
said so until the next weekly run would have failed.
140144

141145
## v1.11.0 (2026-09-30)
142146

‎scripts/pipeline_liveness.py‎

Lines changed: 43 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,8 @@
2929
With `--scheduled` (the weekly workflow) the age check measures against the
3030
current time instead of the database date, upstream sources with a regular
3131
cadence are checked against `stale_after_days`, and published rows whose
32-
evidence expires within a month are reported. Both are left out of
32+
evidence expires within a month are reported. A source within a week of its
33+
limit is a warning, exit code 3, which the workflow reports without failing. Both are left out of
3334
`verifyPipelineTests`: that task runs inside `check`, where a clock would fail
3435
every later build of an old tag. Measured against the database date, the age
3536
check cannot fire in the weeks after a drain, because every report queued since
@@ -76,6 +77,12 @@
7677
REGULAR_CADENCES = frozenset({"daily", "weekly"})
7778
# How far ahead the weekly run looks for downloaded rows whose evidence runs out.
7879
EVIDENCE_EXPIRY_WARNING_DAYS = 30
80+
# A source this many days from its stale_after_days limit gets a warning. The
81+
# run is weekly, so a source that only failed once it lapsed could sit stale
82+
# for up to a week first (FTC lapsed 2026-10-07 and the next run was 10-12).
83+
SOURCE_STALE_WARNING_DAYS = 7
84+
# main()'s exit code when nothing failed but a source is close to its limit.
85+
EXIT_WARNING = 3
7986

8087

8188
def _parse_updated(value: object) -> datetime | None:
@@ -190,14 +197,29 @@ def evaluate_source_freshness(manifest: object, freshness: object, now: datetime
190197
A manifest this can't read is a failure rather than a pass: a broken
191198
manifest would otherwise switch the whole check off.
192199
"""
200+
return _source_freshness(manifest, freshness, now)[0]
201+
202+
203+
def warn_source_staleness(manifest: object, freshness: object, now: datetime) -> list[str]:
204+
"""One warning per regular-cadence source within SOURCE_STALE_WARNING_DAYS of its limit.
205+
206+
The source is still fresh, so the weekly run reports it without failing
207+
(EXIT_WARNING), and a source past its limit is evaluate_source_freshness's.
208+
"""
209+
return _source_freshness(manifest, freshness, now)[1]
210+
211+
212+
def _source_freshness(manifest: object, freshness: object, now: datetime) -> tuple[list[str], list[str]]:
213+
"""The failures and the warnings for every regular-cadence source."""
193214
sources = manifest.get("sources") if isinstance(manifest, dict) else None
194215
if not isinstance(sources, list):
195-
return [f"{MANIFEST_FILE.name} is missing or unreadable, so upstream freshness can't be checked"]
216+
return [f"{MANIFEST_FILE.name} is missing or unreadable, so upstream freshness can't be checked"], []
196217
recorded = freshness.get("last_success") if isinstance(freshness, dict) else None
197218
last_success = recorded if isinstance(recorded, dict) else {}
198219
recorded_dates = freshness.get("newest_record_date") if isinstance(freshness, dict) else None
199220
newest_record_date = recorded_dates if isinstance(recorded_dates, dict) else {}
200221
problems: list[str] = []
222+
warnings: list[str] = []
201223
for source in sources:
202224
if not isinstance(source, dict):
203225
problems.append(f"{MANIFEST_FILE.name} has a source entry that isn't an object")
@@ -222,13 +244,15 @@ def evaluate_source_freshness(manifest: object, freshness: object, now: datetime
222244
f"{source_id} ({cadence} source) has no {'record date' if record_dated else 'successful import'} recorded in "
223245
f"{FRESHNESS_FILE.name} - {refresh}"
224246
)
225-
elif now - stamp > timedelta(days=limit):
226-
age_description = "newest record is from" if record_dated else "was last imported"
227-
problems.append(
228-
f"{source_id} {age_description} {stamp.date().isoformat()}, {(now - stamp).days} days ago, "
229-
f"past its {limit:g}-day limit - {refresh}"
230-
)
231-
return problems
247+
continue
248+
age_description = "newest record is from" if record_dated else "was last imported"
249+
age = f"{source_id} {age_description} {stamp.date().isoformat()}, {(now - stamp).days} days ago"
250+
if now - stamp > timedelta(days=limit):
251+
problems.append(f"{age}, past its {limit:g}-day limit - {refresh}")
252+
elif now - stamp >= timedelta(days=limit - SOURCE_STALE_WARNING_DAYS):
253+
stale_on = (stamp + timedelta(days=limit)).date().isoformat()
254+
warnings.append(f"{age}, and passes its {limit:g}-day limit after {stale_on} - {refresh}")
255+
return problems, warnings
232256

233257

234258
def row_expiry_epoch_ms(row: dict) -> int | None:
@@ -359,14 +383,22 @@ def main(argv: list[str] | None = None) -> int:
359383
reports, unreadable = load_queue(REPORTS_DIR)
360384
total = len(reports) + unreadable
361385
problems = evaluate_queue_health(reports, load_database_updated(DB_FILE), unreadable, now=now)
386+
warnings: list[str] = []
362387
if now is not None:
363-
problems += evaluate_source_freshness(_load_json(MANIFEST_FILE), _load_json(FRESHNESS_FILE), now)
388+
manifest, freshness = _load_json(MANIFEST_FILE), _load_json(FRESHNESS_FILE)
389+
problems += evaluate_source_freshness(manifest, freshness, now)
364390
problems += evaluate_evidence_expiry(_load_json(DB_FILE), now)
391+
warnings = warn_source_staleness(manifest, freshness, now)
365392
if problems:
366393
print(f"Report pipeline is not healthy ({total} queued file(s) in {REPORTS_DIR}):", file=sys.stderr)
367-
for problem in problems:
394+
for problem in problems + [f"warning: {warning}" for warning in warnings]:
368395
print(f" - {problem}", file=sys.stderr)
369396
return 1
397+
if warnings:
398+
print(f"Report pipeline warning, a source is close to stale ({total} queued file(s) in {REPORTS_DIR}):")
399+
for warning in warnings:
400+
print(f" - {warning}")
401+
return EXIT_WARNING
370402
suffix = f", {unreadable} unreadable" if unreadable else ""
371403
print(f"Report queue liveness OK ({total} file(s) pending{suffix}).")
372404
return 0

‎scripts/test_pipeline_liveness.py‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,13 @@
1313
MIN_BUCKET_SAMPLE,
1414
evaluate_evidence_expiry,
1515
evaluate_queue_health,
16+
EXIT_WARNING,
17+
SOURCE_STALE_WARNING_DAYS,
1618
evaluate_source_freshness,
1719
load_database_updated,
1820
load_queue,
1921
row_expiry_epoch_ms,
22+
warn_source_staleness,
2023
)
2124

2225
BUCKET = "0123456789abcdef"
@@ -234,8 +237,84 @@ def scheduled_checks() -> None:
234237
problems = evaluate_source_freshness(real_manifest, default_import, BASE)
235238
assert len(problems) == 1 and problems[0].startswith("fcc_complaints newest record is from 2026-08-01"), problems
236239

240+
source_warning_checks()
237241
evidence_expiry_checks()
238242
check_scheduled_switch()
243+
check_scheduled_warning()
244+
245+
246+
def source_warning_checks() -> None:
247+
"""A source a week from its limit warns without failing; past the limit it fails instead."""
248+
assert SOURCE_STALE_WARNING_DAYS == 7
249+
manifest = {
250+
"sources": [
251+
{"id": "ftc_complaints", "cadence": "daily", "stale_after_days": 14},
252+
{"id": "toastedspam", "cadence": "weekly", "stale_after_days": 30, "import_flag": "--allow-insecure-sources"},
253+
]
254+
}
255+
256+
def ftc_aged(days: float) -> dict:
257+
return {"newest_record_date": {"ftc_complaints": (BASE - timedelta(days=days)).isoformat()}}
258+
259+
assert warn_source_staleness(manifest, ftc_aged(6), BASE) == []
260+
for days in (7, 13, 14):
261+
warnings = warn_source_staleness(manifest, ftc_aged(days), BASE)
262+
assert len(warnings) == 1, (days, warnings)
263+
assert warnings[0].startswith(f"ftc_complaints newest record is from {(BASE - timedelta(days=days)).date().isoformat()}"), warnings
264+
if days == 14:
265+
assert warnings[0].endswith("limit after 2026-09-04 - run scripts/import_all_sources.py"), warnings
266+
assert evaluate_source_freshness(manifest, ftc_aged(days), BASE) == [], days
267+
stale = ftc_aged(15)
268+
assert warn_source_staleness(manifest, stale, BASE) == []
269+
assert len(evaluate_source_freshness(manifest, stale, BASE)) == 1
270+
271+
# A 30-day source warns from day 23, by its last import, naming its flag.
272+
imported = ftc_aged(0)
273+
imported["last_success"] = {"toastedspam": (BASE - timedelta(days=22)).isoformat()}
274+
assert warn_source_staleness(manifest, imported, BASE) == []
275+
imported["last_success"]["toastedspam"] = (BASE - timedelta(days=23)).isoformat()
276+
warnings = warn_source_staleness(manifest, imported, BASE)
277+
assert len(warnings) == 1 and warnings[0].startswith("toastedspam was last imported"), warnings
278+
assert warnings[0].endswith("--allow-insecure-sources"), warnings
279+
280+
# A broken manifest stays a failure and adds no warnings.
281+
assert warn_source_staleness(None, ftc_aged(10), BASE) == []
282+
283+
284+
def check_scheduled_warning() -> None:
285+
"""A source close to its limit exits with EXIT_WARNING and says so; nothing else fails."""
286+
import contextlib
287+
import io
288+
289+
import pipeline_liveness as liveness
290+
291+
saved = (liveness.REPORTS_DIR, liveness.DB_FILE, liveness.MANIFEST_FILE, liveness.FRESHNESS_FILE)
292+
with tempfile.TemporaryDirectory() as directory:
293+
data = Path(directory)
294+
(data / "reports").mkdir()
295+
today = datetime.now(timezone.utc)
296+
(data / "spam_numbers.json").write_text(json.dumps({"updated": today.date().isoformat(), "numbers": []}), encoding="utf-8")
297+
(data / "source-manifest.json").write_text(
298+
json.dumps({"sources": [{"id": "ftc_complaints", "cadence": "daily", "stale_after_days": 14}]}),
299+
encoding="utf-8",
300+
)
301+
aged = (today - timedelta(days=9)).date().isoformat()
302+
(data / "source-freshness.json").write_text(json.dumps({"newest_record_date": {"ftc_complaints": aged}}), encoding="utf-8")
303+
liveness.REPORTS_DIR = data / "reports"
304+
liveness.DB_FILE = data / "spam_numbers.json"
305+
liveness.MANIFEST_FILE = data / "source-manifest.json"
306+
liveness.FRESHNESS_FILE = data / "source-freshness.json"
307+
try:
308+
output = io.StringIO()
309+
with contextlib.redirect_stdout(output), contextlib.redirect_stderr(output):
310+
assert liveness.main(["--scheduled"]) == EXIT_WARNING, output.getvalue()
311+
assert "close to stale" in output.getvalue() and f"newest record is from {aged}" in output.getvalue(), output.getvalue()
312+
313+
# Without --scheduled there's no clock, so nothing to warn about.
314+
with contextlib.redirect_stdout(io.StringIO()):
315+
assert liveness.main([]) == 0
316+
finally:
317+
liveness.REPORTS_DIR, liveness.DB_FILE, liveness.MANIFEST_FILE, liveness.FRESHNESS_FILE = saved
239318

240319

241320
def evidence_row(expires_in_days: float | None, *, number: str = "+15125550100") -> dict:

0 commit comments

Comments
 (0)