From 719e85c16e37bb76bf76aafca34b59f78f535723 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:04:59 +0000 Subject: [PATCH 01/22] concepts/pipeline/artifact_deployment.md: Correct UDF deploy trigger from nightly to CircleCI/Jenkins Finding-Id: df663f5a96cc49298045ee91bf335174 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/telemetry-airflow/blob/main/dags/bqetl_artifact_deployment.py --- src/concepts/pipeline/artifact_deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/concepts/pipeline/artifact_deployment.md b/src/concepts/pipeline/artifact_deployment.md index e8bb8cb73..85b2ea5aa 100644 --- a/src/concepts/pipeline/artifact_deployment.md +++ b/src/concepts/pipeline/artifact_deployment.md @@ -18,7 +18,7 @@ There are two categories of user-defined functions: - UDFs for internal use only: These UDFs are published to the `udf` and `udf_js` datasets in the `moz-fx-data-shared-prod` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/moz-fx-data-shared-prod/udf) - public UDFs that can be used even outside of Mozilla projects: These UDFs are published to the `mozfun` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/mozfun) -The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG which run nightly, but can be triggered manually by clearing the tasks. +The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG. The DAG no longer runs on a nightly schedule; it is triggered by CircleCI on merges to `main` and by Jenkins after schema deploys, and tasks can still be triggered manually by clearing them. ## Table Deployment From 2d9091a381d0520d1f20416655670b4bb554a495 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:05:00 +0000 Subject: [PATCH 02/22] concepts/pipeline/artifact_deployment.md: Replace retired publish_new_tables task with publish_tables_and_views Finding-Id: dab451274d604f7dbd99787209d652c0 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/telemetry-airflow/blob/main/dags/bqetl_artifact_deployment.py --- src/concepts/pipeline/artifact_deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/concepts/pipeline/artifact_deployment.md b/src/concepts/pipeline/artifact_deployment.md index 85b2ea5aa..44fb13dfc 100644 --- a/src/concepts/pipeline/artifact_deployment.md +++ b/src/concepts/pipeline/artifact_deployment.md @@ -22,7 +22,7 @@ The UDF deploy is triggered through the `publish_public_udfs` and `publish_persi ## Table Deployment -Tables get deployed through the [`publish_new_tables` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) which runs nightly. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied. +Tables get deployed through the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_new_tables` task. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied. ## View Deployment From 8162523105a30bd02364c942930d434e856109c1 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:05:02 +0000 Subject: [PATCH 03/22] concepts/pipeline/artifact_deployment.md: Replace retired publish_views task with publish_tables_and_views Finding-Id: 55db0524531f4950ba5cd453bb9fc45a Reviewed: no (this PR is the review) Source: https://github.com/mozilla/telemetry-airflow/blob/main/dags/bqetl_artifact_deployment.py --- src/concepts/pipeline/artifact_deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/concepts/pipeline/artifact_deployment.md b/src/concepts/pipeline/artifact_deployment.md index 44fb13dfc..6a4555280 100644 --- a/src/concepts/pipeline/artifact_deployment.md +++ b/src/concepts/pipeline/artifact_deployment.md @@ -26,7 +26,7 @@ Tables get deployed through the [`publish_tables_and_views` Airflow task](https: ## View Deployment -View deployment runs after new tables have been published through the [`publish_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment). This task will run all SQL generators, which ensures that SQL for generated views will be available. Only views with either changes to their SQL definition, schema, metadata or newly created views will be deployed. +View deployment happens together with table deployment in the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_views` task. This task will run all SQL generators, which ensures that SQL for generated views will be available. Only views with either changes to their SQL definition, schema, metadata or newly created views will be deployed. Views that have been defined in bigquery-etl will be tagged with a `managed` label. This label is used to automatically remove views from BigQuery that have been deleted in bigquery-etl. Having this label ensures that manually created views or views created through other tooling won't get deleted as part of this process. From 88e720feac5a8ff06940bf6bfe3103b2a3911c83 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:05:22 +0000 Subject: [PATCH 04/22] concepts/pipeline/event_pipeline.md: Note Glean API has replaced legacy Telemetry event APIs on Desktop Finding-Id: 5cef813209a94aab80d6465fc1cc7502 Reviewed: no (this PR is the review) Source: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/collection/events.html Source: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/collection/webextension-api.html Source: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/start/adding-a-new-probe.html --- src/concepts/pipeline/event_pipeline.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/concepts/pipeline/event_pipeline.md b/src/concepts/pipeline/event_pipeline.md index e0f20efd7..8d2b2a649 100644 --- a/src/concepts/pipeline/event_pipeline.md +++ b/src/concepts/pipeline/event_pipeline.md @@ -103,6 +103,8 @@ use cases: - The _[Telemetry WebExtension API](https://searchfox.org/mozilla-central/rev/55da592d85c2baf8d8818010c41d9738c97013d2/toolkit/components/extensions/schemas/telemetry.json#87)_ ([introduced here](https://bugzilla.mozilla.org/show_bug.cgi?id=1280234)) which allows Mozilla extensions to record new events into Telemetry. +As of Firefox 132, these legacy Telemetry event APIs are deprecated; Firefox Desktop now records events primarily through the [Glean API](glean_data.md), with the WebExtension event functions (`recordEvent`, `registerEvents`) acting as no-ops. See the [Telemetry event collection documentation](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/collection/events.html) for details. + For all these APIs, events will get sent to the pipeline through the [event ping](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/event-ping.html), which gets sent hourly, if any pings were recorded, or up to every 10 minutes whenever 1000 events were recorded. Before Firefox 62, events were sent through the [main ping](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/main-ping.html) instead, with a hard limit of 500 events per ping. From cdeb58ce89caa76ad3e24a29d3fa7e9d0648c8d4 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:06:00 +0000 Subject: [PATCH 05/22] cookbooks/data_modeling.md: Mention shredder mitigation cookbook on index page Finding-Id: b787147c8f4e4d89a2191601c9a7bf78 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/data-docs/blob/main/src/cookbooks/data_modeling/shredder_mitigation.md --- src/cookbooks/data_modeling/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/cookbooks/data_modeling/index.md b/src/cookbooks/data_modeling/index.md index 7e8214217..07a7f45d5 100644 --- a/src/cookbooks/data_modeling/index.md +++ b/src/cookbooks/data_modeling/index.md @@ -2,4 +2,4 @@ This section contains guidelines to consider when implementing new data models. -It's also a collection of best practices related to different parts of the Data Modeling process, including the review of Pull Requests and techniques for performance improvements and cost reduction. +It's also a collection of best practices related to different parts of the Data Modeling process, including the review of Pull Requests, techniques for performance improvements and cost reduction, and shredder mitigation for safely backfilling aggregate tables affected by GDPR deletions. From 1818575f0f0a283f0d40ecfb20d5c98c4a254b55 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:06:18 +0000 Subject: [PATCH 06/22] cookbooks/data_monitoring/intro.md: Add note on code-based (bigconfig.yml/bqetl) monitoring setup Finding-Id: 96558af2ce754788ba150115ac159796 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/cli/monitoring.py --- src/cookbooks/data_monitoring/intro.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/cookbooks/data_monitoring/intro.md b/src/cookbooks/data_monitoring/intro.md index e38c6e000..1fc579776 100644 --- a/src/cookbooks/data_monitoring/intro.md +++ b/src/cookbooks/data_monitoring/intro.md @@ -12,6 +12,10 @@ If you do not have the necessary access or permissions, please [submit a Jira ti Watch the [Bigeye tutorial](https://www.youtube.com/watch?v=8DWyZuU-w1c&t=9s) to get an overview of the platform. Please refer to additional helpful videos available on the Bigeye channel. +## Code-Based Monitoring Configuration + +In addition to configuring monitors through the Bigeye UI, monitoring can be defined as code in [mozilla/bigquery-etl](https://github.com/mozilla/bigquery-etl). Tables can include a `bigconfig.yml` alongside their `metadata.yaml`, with monitoring enabled via a `monitoring.enabled` field. The `bqetl monitoring` CLI (`deploy`, `update`, `validate`, `deploy_custom_rules`) pushes these configs to Bigeye. + ## Stay Updated on What's Happening [Jira Epic Link](https://mozilla-hub.atlassian.net/browse/DENG-4563) From 4ea3915e8e721c7b59daed2cbfdc928080c4f833 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:08:28 +0000 Subject: [PATCH 07/22] cookbooks/scheduling_queries.md: Note Looker is now preferred over STMO for new dashboards Finding-Id: 0957007de7b74bd894e4b69fd857285c Reviewed: no (this PR is the review) Source: https://docs.telemetry.mozilla.org/introduction/tools Source: https://docs.telemetry.mozilla.org/tools/stmo --- src/cookbooks/scheduling_queries.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/cookbooks/scheduling_queries.md b/src/cookbooks/scheduling_queries.md index 4f50f9200..712ce9ab7 100644 --- a/src/cookbooks/scheduling_queries.md +++ b/src/cookbooks/scheduling_queries.md @@ -8,6 +8,8 @@ Want to refresh a dashboard or other query against our data store automatically? This is by far the easiest option. Any query on [STMO](../tools/stmo.md) can be made to refresh itself automatically. This can be used to power both dashboards inside sql.telemetry.mozilla.org as well as web sites like the [Tab Spinner Dashboard](https://mikeconley.github.io/bug1310250/) and [Numbers that Matter](https://github.com/mozilla/funnel) via Redash's CSV and JSON APIs. +Note that as of August 2021, Looker is the preferred tool for building new dashboards and data visualizations at Mozilla where possible; STMO remains available for SQL-based exploration and the scheduled-query use case described here. + However, there are disadvantages to this approach: - Every time it is refreshed, STMO re-runs the query from scratch. This is fine for smaller or less frequently updated dashboards but when querying a large amount of data, [this can get expensive](./bigquery/optimization.md). From 6497e2bc3f540bc75c3681dad691b00a6de01fd5 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:08:29 +0000 Subject: [PATCH 08/22] cookbooks/scheduling_queries.md: Document manual unpausing of new DAGs in Airflow UI Finding-Id: 1741aaa69d3b47e9b1451824efa3a385 Reviewed: no (this PR is the review) Source: https://docs.telemetry.mozilla.org/concepts/airflow_gotchas.html --- src/cookbooks/scheduling_queries.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/cookbooks/scheduling_queries.md b/src/cookbooks/scheduling_queries.md index 712ce9ab7..c9557e088 100644 --- a/src/cookbooks/scheduling_queries.md +++ b/src/cookbooks/scheduling_queries.md @@ -24,6 +24,8 @@ Also, since the procedure for scheduling a query in this way is submitting a pul For more information on how to use this approach, see [A quick guide to creating a derived dataset with bigquery-etl](https://mozilla.github.io/bigquery-etl/cookbooks/creating_a_derived_dataset/). +Note that new DAGs are deployed in a paused state by default; you'll need to manually unpause the DAG via the Airflow UI on workflow.telemetry.mozilla.org before it will run on its schedule. See [Airflow Gotcha's](https://docs.telemetry.mozilla.org/concepts/airflow_gotchas.html) for details. + ### Testing changes before deploying bigquery-etl can run a query against real data in a private development environment before a pull request is opened, so a new or changed query can be validated — confirming it runs without errors and returns sensible numbers — without affecting the production tables that power live dashboards and metrics. See [Development workflows](https://mozilla.github.io/bigquery-etl/cookbooks/development_workflows/). From 73e15e2035abae91355aaa58e921127dd09791cd Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:08:47 +0000 Subject: [PATCH 09/22] datasets/active_profiles.md: Flag pipeline as likely retired/archived Finding-Id: b07dc1982a424228b91ab3fabe4f5268 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/bigquery-etl/tree/HEAD/sql/moz-fx-data-shared-prod/telemetry_derived Source: https://github.com/mozilla/telemetry-airflow/pull/437 Source: https://github.com/mozilla/telemetry-airflow/tree/HEAD/dags --- src/datasets/active_profiles.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/datasets/active_profiles.md b/src/datasets/active_profiles.md index 8ec731204..663cf5df7 100644 --- a/src/datasets/active_profiles.md +++ b/src/datasets/active_profiles.md @@ -140,3 +140,5 @@ or on [PyPI](https://pypi.org/project/bgbb/). The airflow job is defined in the [`bgbb_airflow` repo](https://github.com/wcbeard/bgbb_airflow). The model to fit the parameters is run weekly, and the table is updated daily. + +**Note:** This pipeline (and its DAG) can no longer be found in Mozilla's current `bigquery-etl` or `telemetry-airflow` repositories, suggesting it has been retired (archived); this page may describe a legacy dataset. From 7c0bea27aef090021b01cf9c39c2f79f6a74c53d Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:09:24 +0000 Subject: [PATCH 10/22] datasets/batch_view/telemetry_aggregates/reference.md: Add decommission notice for telemetry_aggregates dataset Finding-Id: 8b6ff4e7d00c4035b9a4710314a27f3a Reviewed: no (this PR is the review) Source: https://mozilla.github.io/telemetry-dashboard/new-pipeline/evo.html --- src/datasets/batch_view/telemetry_aggregates/reference.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/datasets/batch_view/telemetry_aggregates/reference.md b/src/datasets/batch_view/telemetry_aggregates/reference.md index ed529c1b0..716be826d 100644 --- a/src/datasets/batch_view/telemetry_aggregates/reference.md +++ b/src/datasets/batch_view/telemetry_aggregates/reference.md @@ -2,6 +2,8 @@ +**Note:** The `telemetry_aggregates` dataset and the dashboards it powered (the old Telemetry Measurement Dashboard / aggregates.telemetry.mozilla.org) were decommissioned on December 5, 2024. This page is kept for historical reference; see [GLAM](https://docs.telemetry.mozilla.org/datasets/glam.html) for the current replacement. + # Introduction {{#include ./intro.md}} From b2e4a15b4cc51f32d12c712ab7b0cdc20116454d Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:10:33 +0000 Subject: [PATCH 11/22] datasets/jetstream.md: Point to metric-hub for config-push trigger, note archive Finding-Id: 7e83e5e45d524f339e50c97b54e71838 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/jetstream-config/blob/main/README.md Source: https://github.com/mozilla/metric-hub/blob/main/jetstream/README.md --- src/datasets/jetstream.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/datasets/jetstream.md b/src/datasets/jetstream.md index c3b269dc1..c061a38b6 100644 --- a/src/datasets/jetstream.md +++ b/src/datasets/jetstream.md @@ -8,7 +8,7 @@ BigQuery and serve both as the substrate for the result visualization platform and as a resource for data scientists. Jetstream runs as part of the nightly ETL job (see [Scheduling](#scheduling) below). -Jetstream is also run after pushes to the [`jetstream-config`] repository. +Jetstream is also run after pushes to the [`metric-hub`] repository's `jetstream` directory (the `jetstream-config` repository is archived). Jetstream publishes tables to the dataset `moz-fx-data-experiments.mozanalysis`. Experiments are analyzed using the concept of analysis windows. Analysis @@ -41,6 +41,7 @@ on configuring Jetstream can be found at the [`jetstream-config`] repository. [jetstream]: https://github.com/mozilla/jetstream [`jetstream-config`]: https://github.com/mozilla/jetstream-config +[`metric-hub`]: https://github.com/mozilla/metric-hub/tree/main/jetstream ## Statistics tables From c94abfd058bc6904863b786f4df810f0584f26bf Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:10:34 +0000 Subject: [PATCH 12/22] datasets/jetstream.md: Document days28 and preenrollment analysis windows Finding-Id: 11797fdcd4e146749b71626c9926a921 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/experimenter-docs/blob/main/docs/data-analysis/jetstream/operations.md Source: https://github.com/mozilla/jetstream/blob/main/jetstream/analysis.py Source: https://github.com/mozilla/metric-hub/blob/main/lib/metric-config-parser/metric_config_parser/metric.py --- src/datasets/jetstream.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/datasets/jetstream.md b/src/datasets/jetstream.md index c061a38b6..7bc6b2572 100644 --- a/src/datasets/jetstream.md +++ b/src/datasets/jetstream.md @@ -23,6 +23,11 @@ The week 0 analysis window aggregates data from each client's days 0 through 6, the week 1 window aggregates data from days 7 through 13, and so on. +Jetstream also computes a 28-day window (`days28`) and two +pre-enrollment windows (`preenrollment_week` and `preenrollment_days28`) +that aggregate data from immediately before each client's enrollment, +used for example to support covariate-adjustment statistics. + Clients are given a fixed amount of time, specified in Experimenter and often a week long, to enroll. Final day 0 results are available for reporting at the end of the enrollment period, after the last eligible From e1d22f40d96f3f33fdaf739eda42818d30ce3acc Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:12:38 +0000 Subject: [PATCH 13/22] datasets/new_data.md: Name Glean as the required path for new instrumentation Finding-Id: 3429e401d9374bc6a80982850d0905d0 Reviewed: no (this PR is the review) Source: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/start/adding-a-new-probe.html Source: https://mozilla.github.io/glean/book/user/adding-glean-to-your-project.html Source: https://mozilla.github.io/glean/book/user/integrating-glean-for-product-managers.html --- src/datasets/new_data.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/datasets/new_data.md b/src/datasets/new_data.md index 2b2165aa5..0b3ac9eb3 100644 --- a/src/datasets/new_data.md +++ b/src/datasets/new_data.md @@ -8,8 +8,13 @@ please read the [Data Collection Guidelines.][data_collection] ## Mechanics -The mechanics of how to instrument new data collection in Firefox are covered in -[Adding a new Telemetry probe](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/start/adding-a-new-probe.html). +Glean is the standard telemetry platform required for all new Mozilla products. See +[Adding Glean to your project](https://mozilla.github.io/glean/book/user/adding-glean-to-your-project.html) +for the mechanics of instrumenting new data collection with Glean. + +Legacy Telemetry APIs are deprecated for new instrumentation; for background on the +legacy mechanism, see +[Adding a new Telemetry probe](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/start/adding-a-new-probe.html) (deprecated). For non-Telemetry data collection, we have a mechanism for streamlining ingestion of structured (JSON) data that utilizes the same underlying From 1c7ee02284b61d8f36852b3871207009605de3a4 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:12:55 +0000 Subject: [PATCH 14/22] datasets/non_desktop/day_2_7_activation/reference.md: Note table is deprecated and deleted Finding-Id: b24c0662b36349c59adca73d9c9585a1 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/bigquery-etl/pull/9957 --- src/datasets/non_desktop/day_2_7_activation/reference.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/datasets/non_desktop/day_2_7_activation/reference.md b/src/datasets/non_desktop/day_2_7_activation/reference.md index 05f5b4d01..0c1e56dae 100644 --- a/src/datasets/non_desktop/day_2_7_activation/reference.md +++ b/src/datasets/non_desktop/day_2_7_activation/reference.md @@ -6,6 +6,8 @@ `firefox_nondesktop_day_2_7_activation` is designed for use in calculating the [Day 2-7 Activation metric](../../../metrics/metrics.md#day-2-7-activation), a key result in 2020 for non-desktop products. +**Note:** This table has since been deprecated and deleted from production ([bigquery-etl#9957](https://github.com/mozilla/bigquery-etl/pull/9957)); the information below is retained for historical reference. + # Contents `firefox_nondesktop_day_2_7_activation` is a table with two key metrics: `new_profiles` and `day_2_7_activated`, aggregated over `submission_date`. It is derived from the [non-desktop clients last seen table](https://docs.telemetry.mozilla.org/cookbooks/clients_last_seen_bits.html). From 524654950de084749d5ede45765ee8d22bbc9df6 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:12:56 +0000 Subject: [PATCH 15/22] datasets/non_desktop/day_2_7_activation/reference.md: Mark code reference as removed from bigquery-etl Finding-Id: 453ddd2fecd14f44a457eb5682a3da79 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/bigquery-etl/blob/main/GRAVEYARD.md Source: https://github.com/mozilla/bigquery-etl/pull/9957 --- src/datasets/non_desktop/day_2_7_activation/reference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/datasets/non_desktop/day_2_7_activation/reference.md b/src/datasets/non_desktop/day_2_7_activation/reference.md index 0c1e56dae..523d0e819 100644 --- a/src/datasets/non_desktop/day_2_7_activation/reference.md +++ b/src/datasets/non_desktop/day_2_7_activation/reference.md @@ -68,7 +68,7 @@ root # Code Reference -The `firefox_nondesktop_day_2_7_activation job is` [defined in `bigquery-etl`](https://github.com/mozilla/bigquery-etl/blob/master/sql/moz-fx-data-shared-prod/telemetry_derived/firefox_nondesktop_day_2_7_activation_v1/query.sql). +The `firefox_nondesktop_day_2_7_activation` job has been removed from `bigquery-etl` (archived); it was deleted via [PR #9957](https://github.com/mozilla/bigquery-etl/pull/9957). # Background and Caveats From 6dcda41199b33ae045ecdcfcf2025f5ffd586df0 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:13:27 +0000 Subject: [PATCH 16/22] datasets/search.md: Note new parallel Glean-based search pipeline Finding-Id: f757cb0bd49e42738b9a2dda3a5aa5cd Reviewed: no (this PR is the review) Source: https://github.com/mozilla/bigquery-etl/blob/131aa993034a53540fca26627fc920d82a18f96b/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_glean_v1/README.md --- src/datasets/search.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/datasets/search.md b/src/datasets/search.md index 67e97fbb0..f0f4d3697 100644 --- a/src/datasets/search.md +++ b/src/datasets/search.md @@ -63,6 +63,8 @@ Our partners (search engines) attribute queries to Mozilla using **partner codes We report nine types of searches in our search datasets: `sap`, `tagged-sap`, `tagged-follow-on`, `search_with_ads`, `search_with_ads_organic`, `ad_click`, `ad_click_organic`, `organic`, and `unknown` (although the earliest available date for each varies). These aggregates show up as columns in the`search_aggregates` and `search_clients_engines_sources_daily` datasets. Our search datasets are all derived from `main_summary`. The aggregate columns are derived from the `SEARCH_COUNTS` histogram. +A newer, still-experimental Glean-based pipeline, `search_clients_daily_glean` (built from `firefox_desktop.serp_events`, `events_stream`, and `metrics` ping data), is being developed in parallel with the legacy `main_summary`-derived datasets described in this article. It introduces new dimensions such as `partner_code`; see the [`search_clients_daily_glean_v1` README](https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_glean_v1/README.md) for details. + The **`sap` column counts all SAP (or direct) searches**. `sap` search counts are collected via [probes](https://firefox-source-docs.mozilla.org/browser/browser/BrowserUsageTelemetry.html#search-telemetry) within the Firefox UI These counts are **very reliable, but do not count follow-on queries**. In 2017-06 we deployed the [`followonsearch` addon], which adds probes for `tagged-sap` and `tagged-follow-on` searches. These columns **attempt to count all tagged searches** by looking for Mozilla partner codes in the URL of requests to partner search engines. These search counts are critical to understanding revenue since they exclude untagged searches and include follow-on searches. However, these search counts have **important caveats affecting their reliability**. See [In Content Telemetry Issues](#in-content-telemetry-issues) for more information. From 964fb5c4286697bbda1e1a875dcb03b2154b9926 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:16:23 +0000 Subject: [PATCH 17/22] metrics/dimensions.md: Note that metric-hub is now the canonical dimension source Finding-Id: 1c789bff80a74a869bb3dc401440f2fc Reviewed: no (this PR is the review) Source: https://github.com/mozilla/metric-hub/blob/main/definitions/README.md --- src/metrics/dimensions.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/metrics/dimensions.md b/src/metrics/dimensions.md index 506538628..f81f11e25 100644 --- a/src/metrics/dimensions.md +++ b/src/metrics/dimensions.md @@ -2,6 +2,8 @@ Here we provide definitions and information about our standard slicing dimensions. +Note: the canonical, maintained dimension definitions now live as `[dimensions.*]` TOML blocks in the [metric-hub](https://github.com/mozilla/metric-hub) repository (see `definitions/*.toml` and `opmon/*.toml`), consumed programmatically by OpMon, Jetstream, and Looker. This page remains a stub template and does not reflect current practice. + Table of contents with links to dimensions. --- From f597a4f886cb5c6e65544e73bba904e4c7d9ec59 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:16:24 +0000 Subject: [PATCH 18/22] metrics/dimensions.md: Replace STMO query link bullet with metric-hub reference Finding-Id: 82896ce017684d97a5d30cce6521db7e Reviewed: no (this PR is the review) Source: https://github.com/mozilla/metric-hub/blob/main/definitions/README.md --- src/metrics/dimensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/metrics/dimensions.md b/src/metrics/dimensions.md index f81f11e25..e22c8b6df 100644 --- a/src/metrics/dimensions.md +++ b/src/metrics/dimensions.md @@ -18,7 +18,7 @@ TL;DR: two sentence max. - Definitions of what user has to do to qualify - Definitions of what telemetry must be sent to qualify. - At least one working definition -- Link to a scheduled STMO query (link with `stmocli`?) +- Link to the dimension's `[dimensions.*]` TOML block (data source and `select_expression`) in [metric-hub](https://github.com/mozilla/metric-hub) - Possibly an API-linked graph from STMO - Common Issues: Failures and Gotchas - Resources From abbbb50fdabc39a43ece17994ce71cba5c41ef5a Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:17:05 +0000 Subject: [PATCH 19/22] metrics/metrics.md: Document active_users_aggregates, mobile_engagement, mobile_retention tables Finding-Id: 41a969f34b17419db9138198d4551839 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/telemetry/active_users_aggregates/metadata.yaml Source: https://github.com/mozilla/metric-hub/blob/9fb9ab647f1850ddd3d1c113f393cb8e5eefeedb/definitions/multi_product.toml --- src/metrics/metrics.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/metrics/metrics.md b/src/metrics/metrics.md index 8815fbc08..110788a6b 100644 --- a/src/metrics/metrics.md +++ b/src/metrics/metrics.md @@ -26,6 +26,8 @@ DAU counts unique profiles. Keep in mind that a [profile is not necessarily a us The standard concept of active varies by product, but generally, active users are defined as unique profiles that have sent a `main` ping (on desktop) or a `baseline` ping (on mobile). The precise criteria are defined in the [`usage criterion`](./usage.md) section of this documentation. +DAU, WAU, and MAU values are pre-calculated and available as the single source of truth in the `active_users_aggregates` BigQuery table, which covers both desktop and mobile products. Related mobile engagement and retention metrics are computed from the `mobile_engagement` and `mobile_retention` tables. + We can also restrict the metric to alternative usage criteria. It's **critical to clearly state any non-standard usage criteria on the metric**. The metrics team suggest the following format: `DAU(usage criterion)`. For example, we might be interested in the count of profiles that have viewed more than 5 URIs in a day. We'd denote that metric as `DAU(URI > 5)`. Some common alternative usage criteria documented in the [`usage criterion` section](./usage.md). From 3a234fe0f3ca96ddb7f9d1af9d1fc655bb335fb7 Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:19:29 +0000 Subject: [PATCH 20/22] metrics/policy.md: Note Metric Hub as canonical metric metadata source Finding-Id: 2cf1b290ce664f1ab57aaa2202a8eedb Reviewed: no (this PR is the review) Source: https://github.com/mozilla/data-docs/blob/main/src/concepts/metric_hub.md Source: https://github.com/mozilla/metric-hub --- src/metrics/policy.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/metrics/policy.md b/src/metrics/policy.md index 8d95354a8..9612583d2 100644 --- a/src/metrics/policy.md +++ b/src/metrics/policy.md @@ -4,6 +4,8 @@ Metric standards are maintained by the Metrics Standardization Group (MSG) (link ## How to get more information +The canonical source of truth for metric definitions and metadata is [Metric Hub](https://github.com/mozilla/metric-hub), a repository of metric, data source, and segment definitions that are reviewed before merging. See the [Metric Hub concepts page](https://docs.telemetry.mozilla.org/concepts/metric_hub.html) for more. + ## How to propose a new standard metric, criteria, or dimensions ## How to propose a change to a standards From 117b082f8435bb1a4408fb75e3a153afec319cca Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:19:30 +0000 Subject: [PATCH 21/22] metrics/policy.md: Mention Confluence Metrics Inventory as supporting source Finding-Id: 1c349cb8540c48138fcc41e44a7ad83a Reviewed: no (this PR is the review) Source: https://github.com/mozilla/data-docs/blob/main/src/metrics/metrics.md --- src/metrics/policy.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/metrics/policy.md b/src/metrics/policy.md index 9612583d2..eb765bfa1 100644 --- a/src/metrics/policy.md +++ b/src/metrics/policy.md @@ -6,6 +6,8 @@ Metric standards are maintained by the Metrics Standardization Group (MSG) (link The canonical source of truth for metric definitions and metadata is [Metric Hub](https://github.com/mozilla/metric-hub), a repository of metric, data source, and segment definitions that are reviewed before merging. See the [Metric Hub concepts page](https://docs.telemetry.mozilla.org/concepts/metric_hub.html) for more. +Supporting metric information can also be found on the [Metrics Inventory](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/620494911/Metrics+Inventory) page in Confluence (Mozilla LDAP required). + ## How to propose a new standard metric, criteria, or dimensions ## How to propose a change to a standards From ed8b62e08e33df9175414b72eff02916141407ed Mon Sep 17 00:00:00 2001 From: doc-correctness Date: Thu, 15 Oct 2026 13:19:31 +0000 Subject: [PATCH 22/22] metrics/policy.md: Add deprecate-not-delete metric lifecycle policy Finding-Id: e68e192b39624b899a5857aa38f00132 Reviewed: no (this PR is the review) Source: https://github.com/mozilla/data-docs/blob/main/src/concepts/metric_hub.md --- src/metrics/policy.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/metrics/policy.md b/src/metrics/policy.md index eb765bfa1..a663efc83 100644 --- a/src/metrics/policy.md +++ b/src/metrics/policy.md @@ -12,6 +12,8 @@ Supporting metric information can also be found on the [Metrics Inventory](https ## How to propose a change to a standards +Metric definitions are not deleted once added to [Metric Hub](https://github.com/mozilla/metric-hub); other tools and configurations may still reference them. To retire a metric, set `deprecated = true` on its definition instead of removing it. See the [Metric Hub concepts page](https://docs.telemetry.mozilla.org/concepts/metric_hub.html) for details. + ## Recommendations around citing standards for presentations to senior audiences Something about how all metrics presented should have references to definitions here or explanations as to why a nonstandard metric was used.