fix(telemetry): fail loudly when collection stops, and detect a drained feed

.github/workflows/collect-stats.yml

@@ -80,11 +80,24 @@ jobs:
         run: npm run collect-chrome
 
       - name: Collect first-party CLI telemetry
+        id: telemetry
+        # continue-on-error keeps a telemetry outage from dropping the other
+        # collectors' data: this step runs before generate-md/generate-summary
+        # and the commit step, so a hard failure here would silently cost a full
+        # day of npm/pypi/docker/hf/chrome numbers.
+        #
+        # But continue-on-error alone would ALSO swallow the failure — the job
+        # would go green and a scheduled run that nobody watches would notify
+        # nobody, which is the exact silence this is meant to end. The
+        # "Surface telemetry outage" step at the end re-raises it once the data
+        # is safely committed.
+        continue-on-error: true
         env:
           # Registry base URL exposing the coarse public adoption feed
           # (GET /api/v1/telemetry/v1/adoption/public). Only coarse, anonymous
           # active-user aggregates are collected here; fine-grained detail is
-          # never persisted. Unset -> the collector skips gracefully.
+          # never persisted. Unset -> warns, and errors only if telemetry was
+          # previously being collected (i.e. the variable was removed).
           REGISTRY_URL: ${{ vars.REGISTRY_URL }}
         run: npm run collect-telemetry
 
@@ -100,4 +113,24 @@ jobs:
           git config user.email "actions@github.com"
           git add data/analytics.db data/badge-*.json data/stats-*.json data/npm-stats-*.json data/pypi-stats-*.json data/docker-badge-*.json data/hf-badge-*.json data/chrome-badge-*.json data/chrome-stats-*.json data/telemetry-badge.json data/summary.json ANALYTICS.md ANALYTICS_DETAILED.md
           git diff --staged --quiet || git commit -m "chore: update analytics data [skip ci]"
-          git push || echo "No changes to push"
+          # `git push || echo "No changes to push"` used to swallow EVERY push
+          # failure — a non-fast-forward, a rejected hook, an auth error — and
+          # report the run green with the day's data never pushed. Distinguish
+          # "nothing to push" (fine) from "the push failed" (not fine): the
+          # Surface step below relies on the data actually being committed.
+          if git diff --quiet HEAD@{upstream} HEAD 2>/dev/null; then
+            echo "No changes to push."
+          else
+            git push
+          fi
+
+      # Re-raise a telemetry outage AFTER the day's data is safely committed, so
+      # the run goes red and the scheduled-workflow failure notification fires.
+      # Without this the telemetry step's continue-on-error would leave the job
+      # green and nobody would ever learn that collection stopped — which is the
+      # failure mode this whole thing exists to prevent.
+      - name: Surface telemetry outage
+        if: steps.telemetry.outcome == 'failure'
+        run: |
+          echo "::error::First-party telemetry collection failed — see the 'Collect first-party CLI telemetry' step above. Other collectors' data for today was committed successfully."
+          exit 1

.github/workflows/test.yml

@@ -0,0 +1,51 @@
+name: Test
+
+# Nothing ran the test suite before this workflow existed: the only workflow was
+# the daily collector, so every test in test/ was decoration — it passed on
+# whoever's laptop last remembered to run it, and a change that broke it merged
+# green. That is the same shape as the silent-collector bug this suite exists to
+# guard against, so it gets the same treatment: run it, and let it fail loudly.
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+# A newer run of the same branch makes older ones irrelevant.
+concurrency:
+  group: test-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          # Matches collect-stats.yml. better-sqlite3 is a native module, so the
+          # runner needs a toolchain — ubuntu-latest has one.
+          node-version: '24'
+          cache: 'npm'
+
+      - run: npm ci
+
+      - name: Test
+        run: npm test
+
+      - name: Build
+        run: npm run build
+
+      # The suite drives the real collector against fixture databases and writes
+      # a badge to a temp path. If either escapes into the repo, a test could
+      # publish a fabricated number that the daily workflow then commits — which
+      # has already happened once and was caught by hand.
+      - name: Assert the test suite left no artifacts behind
+        run: |
+          if [ -n "$(git status --porcelain)" ]; then
+            echo "::error::The test suite modified tracked files. Tests must not write into the repo."
+            git status --porcelain
+            git diff
+            exit 1
+          fi
+          echo "Working tree clean."

README.md

@@ -84,9 +84,10 @@ HF_AUTHOR=opena2a                                             # auto-discovers a
 HF_MODELS=org/model,org/other                                 # optional extra models
 HF_TOKEN=hf_...                                               # optional, raises HF rate limits / private repos
 GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcp-key.json          # optional for BigQuery country stats
-REGISTRY_URL=https://api.oa2a.org                     # optional; enables first-party CLI telemetry
+REGISTRY_URL=https://api.oa2a.org                             # optional; enables first-party CLI telemetry
 REGISTRY_TELEMETRY_TOKEN=...                                  # optional; SA token for the gated detail view (server-side only)
 ANALYTICS_TRACKER_PASSWORD=...                                # optional; gates the fine-grained telemetry view on THIS dashboard
+TELEMETRY_STALE_AFTER_DAYS=2                                  # optional; consecutive failure days tolerated before the collector exits 1
 ```
 
 The collector persists only the coarse, anonymous adoption feed. The fine-grained
@@ -101,7 +102,55 @@ The included workflow (`.github/workflows/collect-stats.yml`) runs daily at `6:0
 1. Settings → Secrets and variables → Actions → New secret.
 2. Add `GH_STATS_TOKEN` (a Personal Access Token with `repo` or `public_repo` scope).
 3. Optionally set `GOOGLE_APPLICATION_CREDENTIALS_JSON` for BigQuery country stats.
-4. Optionally set the `REGISTRY_URL` Actions **variable** to enable first-party CLI telemetry collection (the collector skips gracefully when unset).
+4. Optionally set the `REGISTRY_URL` Actions **variable** to enable first-party CLI telemetry collection. Unset, the collector warns and skips.
+
+### When telemetry collection stops
+
+A collector that fails quietly is worse than one that fails outright. Two different
+failures are checked, because they look nothing alike.
+
+**The feed can't be retrieved.** A blip is tolerated; an outage is not:
+
+| Situation | Result |
+|---|---|
+| `REGISTRY_URL` unset, nothing ever collected | warning, exit 0 — legitimately not provisioned |
+| `REGISTRY_URL` unset, but telemetry *was* collected before | **error, exit 1** — the variable was removed |
+| `REGISTRY_URL` set, nothing ever collected | **error, exit 1** — broken feed or wrong URL |
+| Failing for ≤ `TELEMETRY_STALE_AFTER_DAYS` (default 2) | warning, exit 0 — transient |
+| Failing for longer | **error, exit 1** — the dashboard is serving stale numbers |
+
+**The feed responds fine but the numbers drained.** This is the shape a broken
+ingest path actually takes: the adoption feed keeps answering `200` with a
+structurally valid payload while the active-user counts empty out. A retrieval check
+sees a healthy response and happily persists the zeros. So the collector also errors
+when the feed reports `mau=0` while the last snapshot with any activity had users —
+real users do not all vanish overnight.
+
+Two details that matter, both learned the hard way:
+
+- It keys on **MAU alone**, not on "MAU and installs are both zero". `total_installs`
+  is a 90-day window and `mau` is 30-day, so a broken ingest empties MAU at T+30
+  while installs coasts to T+90. Requiring both would wave through ~60 days of a dead
+  pipeline. Installs still reporting while actives are zero is corroboration, not a
+  reason to stay quiet.
+- It compares against the **last snapshot that had users**, not yesterday. Comparing
+  to yesterday makes the alert self-silencing: day one fires, its zeros become the
+  baseline, and every later day sees zero-following-zero and reports healthy. Keying
+  on the last live snapshot keeps the alarm ringing and states the growing gap.
+
+The zeros are still recorded as reported (we don't suppress what the feed said); the
+point is to raise the alarm. A real decline is not an outage — only reaching zero is.
+A fleet that has never reported an active user is not an outage either.
+
+Failures emit GitHub Actions annotations so they surface in the run summary, and a
+final step re-raises the failure **after** the day's data is committed — so the run
+goes red and the scheduled-run notification fires, without a telemetry outage costing
+the other collectors their data. A failed run never writes: zeros must not overwrite
+a good snapshot.
+
+Untrusted feed text (HTTP error bodies) is sanitized before it reaches any log line —
+the Actions runner parses `::command::` lines on both stdout and stderr, so an
+unsanitized response body could otherwise forge or suppress these very annotations.
 
 The workflow auto-discovers public repos in the orgs listed in `GITHUB_ORG`. Add a new repo to the org, the next run picks it up. No manual list maintenance.