Improve worker version reporting, navigation, and contributor tooling

.github/workflows/ci.yml

@@ -0,0 +1,76 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches: [master]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  unit-tests:
+    name: Go, Python, and JavaScript tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
+        with:
+          persist-credentials: false
+      - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6
+        with:
+          go-version: stable
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+        with:
+          python-version: '3.13'
+      - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
+        with:
+          node-version: '24'
+      - name: Go tests
+        run: go test ./...
+      - name: Python tests
+        run: python3 -m unittest test_fetch_and_generate.py
+      - name: JavaScript tests
+        working-directory: docs
+        run: npm test
+
+  browser-tests:
+    name: Firefox layout tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
+        with:
+          persist-credentials: false
+      - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6
+        with:
+          go-version: stable
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+        with:
+          python-version: '3.13'
+      - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
+        with:
+          node-version: '24'
+          cache: npm
+          cache-dependency-path: docs/package-lock.json
+      - name: Install test dependencies
+        working-directory: docs
+        run: npm ci
+      - name: Install Firefox and system dependencies
+        working-directory: docs
+        run: npx playwright install --with-deps firefox
+      - name: Run layout tests
+        working-directory: docs
+        run: npm run test:browser
+      - name: Upload failure screenshots and traces
+        if: failure()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: firefox-layout-results
+          path: docs/test-results/
+          if-no-files-found: ignore
+          retention-days: 7

.gitignore

@@ -1,3 +1,9 @@
-mozilla-history
-audit-worker-versions
-!audit-worker-versions/main.go
+# Locally built executables.
+/mozilla-history
+/audit-worker-versions/audit-worker-versions
+
+# Browser-test dependencies and artifacts.
+/docs/node_modules/
+/docs/playwright-report/
+/docs/test-results/
+__pycache__/

README.md

@@ -46,23 +46,96 @@ mozilla-history
 This will populate subdirectories `Clients`, `Hooks`, `Roles` and `WorkerPools`
 of the current directory.
 
-## Automating the Process
+## Refreshing All Local Data
 
-You can automate this reporting process by setting up a cron job to execute `run-report.sh` at regular intervals.
+For a complete local refresh, run:
+
+```sh
+./fetch_and_generate.py
+```
+
+When `~/.tc_token` exists, the script loads its JSON `clientId` and
+`accessToken` fields. Otherwise, it uses `TASKCLUSTER_CLIENT_ID` and
+`TASKCLUSTER_ACCESS_TOKEN` from the environment. The token file takes
+precedence so stale exported credentials cannot silently override it.
+
+The script builds temporary copies of the Go tools, refreshes `Clients`,
+`Hooks`, `Roles`, and `WorkerPools`, schedules worker-version probes, polls their
+sealed task group until every task reaches a terminal state, writes
+`WorkerVersions`, and stops. It does not pull, commit, push, or deploy anything.
+Review the generated changes before committing them.
+
+The local refresh intentionally does not rebuild `docs/history.json`. History is
+derived exclusively from worker snapshots already committed to Git, so it cannot
+include the snapshot generated by the current run. The production publishing
+workflow rebuilds it after the snapshot is committed.
+
+Use `--poll-interval SECONDS` to change the 60-second polling interval. If the
+script is interrupted after scheduling probes, resume without creating another
+group using `--task-group-id TASK_GROUP_ID`. `TASKCLUSTER_ROOT_URL`,
+`REPORT_SCHEDULER_ID`, and `REPORT_PREFIX` can be overridden in the environment.
+
+Worker versions require two phases because Worker Manager configuration does
+not expose the implementation and version actually running in each pool. The
+first phase schedules an intentionally malformed task on every pool so the
+worker identifies itself in its task log. Once all tasks are resolved, the
+second phase reads those logs and generates the snapshot. Polling replaces the
+older fixed-delay assumption while preserving that probe-and-collect design.
+
+## Rendering an Existing Worker Snapshot
+
+The worker-version report can be regenerated from a saved snapshot without
+Taskcluster credentials or probe tasks:
+
+```sh
+go run ./audit-worker-versions render \
+  WorkerVersions/workers.json \
+  WorkerVersions/README.md
+```
+
+Omit the output path to print the generated Markdown to standard output.
+
+To preview the generated report in the website, serve the repository root:
+
+```sh
+./run_local.sh
+```
+
+Then open <http://localhost:8000/docs/index-local.html>. The local preview uses
+`WorkerVersions/README.md` and `docs/history.json` from the checkout. The
+published page continues to load the current report from GitHub. Pass a port as
+the first argument to override the default, for example `./run_local.sh 8080`.
+
+## Testing
+
+Pull requests and pushes to `master` run Go, Python, JavaScript, and Firefox
+layout tests through `.github/workflows/ci.yml`. These checks do not create
+Taskcluster probes or publish reports.
+
+## Optional Quick previews
+
+See [scripts/quick/README.md](scripts/quick/README.md) for personal Quick previews
+of the checkout's report. These helpers are separate from production publishing.
+
+## Production Automation
+
+The existing `run-reports.sh` and `audit.sh` scripts implement the repository's
+production publishing workflow. They include git and production-site behavior;
+use `fetch_and_generate.py` for local refreshes.
 
 ### Prerequisites
 - Valid Taskcluster credentials must be set in the environment variables:
   - `TASKCLUSTER_CLIENT_ID`
   - `TASKCLUSTER_ACCESS_TOKEN`
 
 ### How it works
-1. `run-report.sh` executes `audit.sh`
+1. `run-reports.sh` executes `audit.sh`
 2. `audit.sh` schedules tasks for each worker pool to extract worker implementation details from logs
-3. Results are stored in the `WorkerPools` directory
+3. Results are stored in the `WorkerVersions` directory
 4. `mozilla-history` stores Taskcluster configurations in their respective directories:
    - Hooks definitions in `Hooks/`
    - Roles definitions in `Roles/`
-   - WorkerPool definitions in `WorkerPool/`
+   - Worker Pool definitions in `WorkerPools/`
 5. `build-docs-history.sh` collects every past revision of `WorkerVersions/workers.json` into
    `docs/history.json`, which feeds the two GitHub Pages views:
    - `docs/index.html` &mdash; the current report plus the full history tables and graphs