mozilla-history

Taskcluster history from https://firefox-ci-tc.services.mozilla.com deployment for:

• Clients
• Hooks
• Roles
• Worker Pools

For https://community-tc.services.mozilla.com history, please see community-history.

Please note, until 9 November 2019 this repository stored the history for the https://taskcluster.net deployment. On 9 November 2019 the https://taskcluster.net deployment was decommissioned and the https://firefox-ci-tc.services.mozilla.com was instated.

Entity filenames

The files are named after the entities themselves, except for the following character conversions:

1. * -> ★

This conversion avoids illegal filenames.

2. / -> ⁄

Rather than creating nested subdirectories, this conversion avoids directory names colliding with entity filenames.

Installing

go get github.com/taskcluster/mozilla-history@v1.0.0

Running

unset TASKCLUSTER_CLIENT_ID TASKCLUSTER_ACCESS_TOKEN TASKCLUSTER_CERTIFICATE
export TASKCLUSTER_ROOT_URL='https://firefox-ci-tc.services.mozilla.com'
mozilla-history

This will populate subdirectories Clients, Hooks, Roles and WorkerPools of the current directory.

Refreshing All Local Data

For a complete local refresh, run:

./fetch_and_generate.py

When ~/.tc_token exists, the script loads its JSON clientId and accessToken fields. Otherwise, it uses TASKCLUSTER_CLIENT_ID and TASKCLUSTER_ACCESS_TOKEN from the environment. The token file takes precedence so stale exported credentials cannot silently override it.

The script builds temporary copies of the Go tools, refreshes Clients, Hooks, Roles, and WorkerPools, schedules worker-version probes, polls their sealed task group until every task reaches a terminal state, writes WorkerVersions, and stops. It does not pull, commit, push, or deploy anything. Review the generated changes before committing them.

The local refresh intentionally does not rebuild docs/history.json. History is derived exclusively from worker snapshots already committed to Git, so it cannot include the snapshot generated by the current run. The production publishing workflow rebuilds it after the snapshot is committed.

Use --poll-interval SECONDS to change the 60-second polling interval. If the script is interrupted after scheduling probes, resume without creating another group using --task-group-id TASK_GROUP_ID. TASKCLUSTER_ROOT_URL, REPORT_SCHEDULER_ID, and REPORT_PREFIX can be overridden in the environment.

Worker versions require two phases because Worker Manager configuration does not expose the implementation and version actually running in each pool. The first phase schedules an intentionally malformed task on every pool so the worker identifies itself in its task log. Once all tasks are resolved, the second phase reads those logs and generates the snapshot. Polling replaces the older fixed-delay assumption while preserving that probe-and-collect design.

Rendering an Existing Worker Snapshot

The worker-version report can be regenerated from a saved snapshot without Taskcluster credentials or probe tasks:

go run ./audit-worker-versions render \
  WorkerVersions/workers.json \
  WorkerVersions/README.md

Omit the output path to print the generated Markdown to standard output.

To preview the generated report in the website, serve the repository root:

./run_local.sh

Then open http://localhost:8000/docs/index-local.html. The local preview uses WorkerVersions/README.md and docs/history.json from the checkout. The published page continues to load the current report from GitHub. Pass a port as the first argument to override the default, for example ./run_local.sh 8080.

Testing

Pull requests and pushes to master run Go, Python, JavaScript, and Firefox layout tests through .github/workflows/ci.yml. These checks do not create Taskcluster probes or publish reports.

Optional Quick previews

See scripts/quick/README.md for personal Quick previews of the checkout's report. These helpers are separate from production publishing.

Production Automation

GitHub Actions (NAS replacement)

.github/workflows/reports.yml runs on Mondays at 07:23 UTC when enabled, or manually through Actions → Mozilla history reports → Run workflow. Install it on the default branch to enable dispatch and scheduling. Both production and forks skip scheduled report jobs unless the repository variable ENABLE_SCHEDULED_REPORTS is set to true. Add it under Settings → Secrets and variables → Actions → Variables → New repository variable to enable weekly runs. Set it to false or remove it to disable scheduled report jobs. Manual runs remain available regardless of this setting, including publishing when dry_run is disabled.

Configure repository Actions secrets TASKCLUSTER_CLIENT_ID and TASKCLUSTER_ACCESS_TOKEN with a dedicated Taskcluster client approved by the team. It needs permission to create lowest-priority probe tasks on the audited pools using scheduler smoketest, seal their task groups, and read the required configuration and artifacts. Have the Taskcluster owners scope this client to those operations; do not copy an unrestricted personal token into CI.

The job tests and builds the Go tools, snapshots configuration, creates the existing intentionally malformed probes, waits 90 minutes, and collects reports. It commits only Clients, Hooks, Roles, WorkerPools, WorkerVersions, and docs/history.json. History is rebuilt after committing the worker snapshot. An unchanged snapshot is successful. One report job runs at a time; a concurrent remote push causes publication to fail safely rather than overwrite changes. Publishing requires the default branch and branch rules permitting the Actions bot's normal push using GITHUB_TOKEN (contents: write).

Manual runs default to dry_run=true: they create real probes and generate a reviewable report-changes patch artifact, but do not push or deploy. Review the patch and task group in the run summary before running with dry_run disabled. Failures appear in Actions logs and run status; maintainers should enable Actions failure notifications. The job has a 150-minute timeout. If a run fails after creating probes, inspect its task group before retrying (a retry creates new probes). Keep the NAS job disabled during tests to avoid duplicate probes, and retire it after a successful publishing run in the team repository.

Testing GitHub Pages on a fork

Set Settings → Pages → Build and deployment → Source to GitHub Actions. Run Publish report to GitHub Pages on the desired branch to deploy its docs/ site; this does not require Taskcluster secrets or create probes. The deployment URL appears on the Actions environment. docs/index.html is the landing page, and migration.html remains alongside it at the site root. There is no redirect through /docs/. The published report retains its existing behavior of loading current worker data from upstream GitHub; history comes from the deployed checkout's docs/history.json. Use the local or Quick preview to view the checkout's own worker snapshot.

Set repository variable PUBLISH_REPORT_PAGES=true to deploy Pages automatically after a successful non-dry report run. The report workflow calls the Pages workflow directly because pushes made with GITHUB_TOKEN do not trigger other push workflows. Leave this unset if the team uses a different publishing setup. Quick deployment remains a separate operation.

GitHub schedules run on the default branch and can be delayed; public-repository schedules are disabled after 60 days without repository activity. See GitHub schedule documentation and GITHUB_TOKEN event behavior.

Legacy NAS scripts

The existing run-reports.sh and audit.sh scripts implement the repository's production publishing workflow. They include git and production-site behavior; use fetch_and_generate.py for local refreshes.

Prerequisites

• Valid Taskcluster credentials must be set in the environment variables:
  • TASKCLUSTER_CLIENT_ID
  • TASKCLUSTER_ACCESS_TOKEN

How it works

1. run-reports.sh executes audit.sh
2. audit.sh schedules tasks for each worker pool to extract worker implementation details from logs
3. Results are stored in the WorkerVersions directory
4. mozilla-history stores Taskcluster configurations in their respective directories:
   • Hooks definitions in Hooks/
   • Roles definitions in Roles/
   • Worker Pool definitions in WorkerPools/
5. build-docs-history.sh collects every past revision of WorkerVersions/workers.json into docs/history.json, which feeds the two GitHub Pages views:
   • docs/index.html — the current report plus the full history tables and graphs
   • docs/migration.html — an animated month-by-month timeline of the docker-worker to generic-worker migration and of the generic-worker version rollout