diff --git a/src/concepts/pipeline/artifact_deployment.md b/src/concepts/pipeline/artifact_deployment.md index e8bb8cb73..6a4555280 100644 --- a/src/concepts/pipeline/artifact_deployment.md +++ b/src/concepts/pipeline/artifact_deployment.md @@ -18,15 +18,15 @@ There are two categories of user-defined functions: - UDFs for internal use only: These UDFs are published to the `udf` and `udf_js` datasets in the `moz-fx-data-shared-prod` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/moz-fx-data-shared-prod/udf) - public UDFs that can be used even outside of Mozilla projects: These UDFs are published to the `mozfun` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/mozfun) -The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG which run nightly, but can be triggered manually by clearing the tasks. +The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG. The DAG no longer runs on a nightly schedule; it is triggered by CircleCI on merges to `main` and by Jenkins after schema deploys, and tasks can still be triggered manually by clearing them. ## Table Deployment -Tables get deployed through the [`publish_new_tables` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) which runs nightly. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied. +Tables get deployed through the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_new_tables` task. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied. ## View Deployment -View deployment runs after new tables have been published through the [`publish_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment). This task will run all SQL generators, which ensures that SQL for generated views will be available. Only views with either changes to their SQL definition, schema, metadata or newly created views will be deployed. +View deployment happens together with table deployment in the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_views` task. This task will run all SQL generators, which ensures that SQL for generated views will be available. Only views with either changes to their SQL definition, schema, metadata or newly created views will be deployed. Views that have been defined in bigquery-etl will be tagged with a `managed` label. This label is used to automatically remove views from BigQuery that have been deleted in bigquery-etl. Having this label ensures that manually created views or views created through other tooling won't get deleted as part of this process. diff --git a/src/concepts/pipeline/event_pipeline.md b/src/concepts/pipeline/event_pipeline.md index e0f20efd7..8d2b2a649 100644 --- a/src/concepts/pipeline/event_pipeline.md +++ b/src/concepts/pipeline/event_pipeline.md @@ -103,6 +103,8 @@ use cases: - The _[Telemetry WebExtension API](https://searchfox.org/mozilla-central/rev/55da592d85c2baf8d8818010c41d9738c97013d2/toolkit/components/extensions/schemas/telemetry.json#87)_ ([introduced here](https://bugzilla.mozilla.org/show_bug.cgi?id=1280234)) which allows Mozilla extensions to record new events into Telemetry. +As of Firefox 132, these legacy Telemetry event APIs are deprecated; Firefox Desktop now records events primarily through the [Glean API](glean_data.md), with the WebExtension event functions (`recordEvent`, `registerEvents`) acting as no-ops. See the [Telemetry event collection documentation](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/collection/events.html) for details. + For all these APIs, events will get sent to the pipeline through the [event ping](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/event-ping.html), which gets sent hourly, if any pings were recorded, or up to every 10 minutes whenever 1000 events were recorded. Before Firefox 62, events were sent through the [main ping](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/main-ping.html) instead, with a hard limit of 500 events per ping. diff --git a/src/cookbooks/data_modeling/index.md b/src/cookbooks/data_modeling/index.md index 7e8214217..07a7f45d5 100644 --- a/src/cookbooks/data_modeling/index.md +++ b/src/cookbooks/data_modeling/index.md @@ -2,4 +2,4 @@ This section contains guidelines to consider when implementing new data models. -It's also a collection of best practices related to different parts of the Data Modeling process, including the review of Pull Requests and techniques for performance improvements and cost reduction. +It's also a collection of best practices related to different parts of the Data Modeling process, including the review of Pull Requests, techniques for performance improvements and cost reduction, and shredder mitigation for safely backfilling aggregate tables affected by GDPR deletions. diff --git a/src/cookbooks/data_monitoring/intro.md b/src/cookbooks/data_monitoring/intro.md index e38c6e000..1fc579776 100644 --- a/src/cookbooks/data_monitoring/intro.md +++ b/src/cookbooks/data_monitoring/intro.md @@ -12,6 +12,10 @@ If you do not have the necessary access or permissions, please [submit a Jira ti Watch the [Bigeye tutorial](https://www.youtube.com/watch?v=8DWyZuU-w1c&t=9s) to get an overview of the platform. Please refer to additional helpful videos available on the Bigeye channel. +## Code-Based Monitoring Configuration + +In addition to configuring monitors through the Bigeye UI, monitoring can be defined as code in [mozilla/bigquery-etl](https://github.com/mozilla/bigquery-etl). Tables can include a `bigconfig.yml` alongside their `metadata.yaml`, with monitoring enabled via a `monitoring.enabled` field. The `bqetl monitoring` CLI (`deploy`, `update`, `validate`, `deploy_custom_rules`) pushes these configs to Bigeye. + ## Stay Updated on What's Happening [Jira Epic Link](https://mozilla-hub.atlassian.net/browse/DENG-4563) diff --git a/src/cookbooks/scheduling_queries.md b/src/cookbooks/scheduling_queries.md index 4f50f9200..c9557e088 100644 --- a/src/cookbooks/scheduling_queries.md +++ b/src/cookbooks/scheduling_queries.md @@ -8,6 +8,8 @@ Want to refresh a dashboard or other query against our data store automatically? This is by far the easiest option. Any query on [STMO](../tools/stmo.md) can be made to refresh itself automatically. This can be used to power both dashboards inside sql.telemetry.mozilla.org as well as web sites like the [Tab Spinner Dashboard](https://mikeconley.github.io/bug1310250/) and [Numbers that Matter](https://github.com/mozilla/funnel) via Redash's CSV and JSON APIs. +Note that as of August 2021, Looker is the preferred tool for building new dashboards and data visualizations at Mozilla where possible; STMO remains available for SQL-based exploration and the scheduled-query use case described here. + However, there are disadvantages to this approach: - Every time it is refreshed, STMO re-runs the query from scratch. This is fine for smaller or less frequently updated dashboards but when querying a large amount of data, [this can get expensive](./bigquery/optimization.md). @@ -22,6 +24,8 @@ Also, since the procedure for scheduling a query in this way is submitting a pul For more information on how to use this approach, see [A quick guide to creating a derived dataset with bigquery-etl](https://mozilla.github.io/bigquery-etl/cookbooks/creating_a_derived_dataset/). +Note that new DAGs are deployed in a paused state by default; you'll need to manually unpause the DAG via the Airflow UI on workflow.telemetry.mozilla.org before it will run on its schedule. See [Airflow Gotcha's](https://docs.telemetry.mozilla.org/concepts/airflow_gotchas.html) for details. + ### Testing changes before deploying bigquery-etl can run a query against real data in a private development environment before a pull request is opened, so a new or changed query can be validated — confirming it runs without errors and returns sensible numbers — without affecting the production tables that power live dashboards and metrics. See [Development workflows](https://mozilla.github.io/bigquery-etl/cookbooks/development_workflows/). diff --git a/src/datasets/active_profiles.md b/src/datasets/active_profiles.md index 8ec731204..663cf5df7 100644 --- a/src/datasets/active_profiles.md +++ b/src/datasets/active_profiles.md @@ -140,3 +140,5 @@ or on [PyPI](https://pypi.org/project/bgbb/). The airflow job is defined in the [`bgbb_airflow` repo](https://github.com/wcbeard/bgbb_airflow). The model to fit the parameters is run weekly, and the table is updated daily. + +**Note:** This pipeline (and its DAG) can no longer be found in Mozilla's current `bigquery-etl` or `telemetry-airflow` repositories, suggesting it has been retired (archived); this page may describe a legacy dataset. diff --git a/src/datasets/batch_view/telemetry_aggregates/reference.md b/src/datasets/batch_view/telemetry_aggregates/reference.md index ed529c1b0..716be826d 100644 --- a/src/datasets/batch_view/telemetry_aggregates/reference.md +++ b/src/datasets/batch_view/telemetry_aggregates/reference.md @@ -2,6 +2,8 @@ +**Note:** The `telemetry_aggregates` dataset and the dashboards it powered (the old Telemetry Measurement Dashboard / aggregates.telemetry.mozilla.org) were decommissioned on December 5, 2024. This page is kept for historical reference; see [GLAM](https://docs.telemetry.mozilla.org/datasets/glam.html) for the current replacement. + # Introduction {{#include ./intro.md}} diff --git a/src/datasets/jetstream.md b/src/datasets/jetstream.md index c3b269dc1..7bc6b2572 100644 --- a/src/datasets/jetstream.md +++ b/src/datasets/jetstream.md @@ -8,7 +8,7 @@ BigQuery and serve both as the substrate for the result visualization platform and as a resource for data scientists. Jetstream runs as part of the nightly ETL job (see [Scheduling](#scheduling) below). -Jetstream is also run after pushes to the [`jetstream-config`] repository. +Jetstream is also run after pushes to the [`metric-hub`] repository's `jetstream` directory (the `jetstream-config` repository is archived). Jetstream publishes tables to the dataset `moz-fx-data-experiments.mozanalysis`. Experiments are analyzed using the concept of analysis windows. Analysis @@ -23,6 +23,11 @@ The week 0 analysis window aggregates data from each client's days 0 through 6, the week 1 window aggregates data from days 7 through 13, and so on. +Jetstream also computes a 28-day window (`days28`) and two +pre-enrollment windows (`preenrollment_week` and `preenrollment_days28`) +that aggregate data from immediately before each client's enrollment, +used for example to support covariate-adjustment statistics. + Clients are given a fixed amount of time, specified in Experimenter and often a week long, to enroll. Final day 0 results are available for reporting at the end of the enrollment period, after the last eligible @@ -41,6 +46,7 @@ on configuring Jetstream can be found at the [`jetstream-config`] repository. [jetstream]: https://github.com/mozilla/jetstream [`jetstream-config`]: https://github.com/mozilla/jetstream-config +[`metric-hub`]: https://github.com/mozilla/metric-hub/tree/main/jetstream ## Statistics tables diff --git a/src/datasets/new_data.md b/src/datasets/new_data.md index 2b2165aa5..0b3ac9eb3 100644 --- a/src/datasets/new_data.md +++ b/src/datasets/new_data.md @@ -8,8 +8,13 @@ please read the [Data Collection Guidelines.][data_collection] ## Mechanics -The mechanics of how to instrument new data collection in Firefox are covered in -[Adding a new Telemetry probe](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/start/adding-a-new-probe.html). +Glean is the standard telemetry platform required for all new Mozilla products. See +[Adding Glean to your project](https://mozilla.github.io/glean/book/user/adding-glean-to-your-project.html) +for the mechanics of instrumenting new data collection with Glean. + +Legacy Telemetry APIs are deprecated for new instrumentation; for background on the +legacy mechanism, see +[Adding a new Telemetry probe](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/start/adding-a-new-probe.html) (deprecated). For non-Telemetry data collection, we have a mechanism for streamlining ingestion of structured (JSON) data that utilizes the same underlying diff --git a/src/datasets/non_desktop/day_2_7_activation/reference.md b/src/datasets/non_desktop/day_2_7_activation/reference.md index 05f5b4d01..523d0e819 100644 --- a/src/datasets/non_desktop/day_2_7_activation/reference.md +++ b/src/datasets/non_desktop/day_2_7_activation/reference.md @@ -6,6 +6,8 @@ `firefox_nondesktop_day_2_7_activation` is designed for use in calculating the [Day 2-7 Activation metric](../../../metrics/metrics.md#day-2-7-activation), a key result in 2020 for non-desktop products. +**Note:** This table has since been deprecated and deleted from production ([bigquery-etl#9957](https://github.com/mozilla/bigquery-etl/pull/9957)); the information below is retained for historical reference. + # Contents `firefox_nondesktop_day_2_7_activation` is a table with two key metrics: `new_profiles` and `day_2_7_activated`, aggregated over `submission_date`. It is derived from the [non-desktop clients last seen table](https://docs.telemetry.mozilla.org/cookbooks/clients_last_seen_bits.html). @@ -66,7 +68,7 @@ root # Code Reference -The `firefox_nondesktop_day_2_7_activation job is` [defined in `bigquery-etl`](https://github.com/mozilla/bigquery-etl/blob/master/sql/moz-fx-data-shared-prod/telemetry_derived/firefox_nondesktop_day_2_7_activation_v1/query.sql). +The `firefox_nondesktop_day_2_7_activation` job has been removed from `bigquery-etl` (archived); it was deleted via [PR #9957](https://github.com/mozilla/bigquery-etl/pull/9957). # Background and Caveats diff --git a/src/datasets/search.md b/src/datasets/search.md index 67e97fbb0..f0f4d3697 100644 --- a/src/datasets/search.md +++ b/src/datasets/search.md @@ -63,6 +63,8 @@ Our partners (search engines) attribute queries to Mozilla using **partner codes We report nine types of searches in our search datasets: `sap`, `tagged-sap`, `tagged-follow-on`, `search_with_ads`, `search_with_ads_organic`, `ad_click`, `ad_click_organic`, `organic`, and `unknown` (although the earliest available date for each varies). These aggregates show up as columns in the`search_aggregates` and `search_clients_engines_sources_daily` datasets. Our search datasets are all derived from `main_summary`. The aggregate columns are derived from the `SEARCH_COUNTS` histogram. +A newer, still-experimental Glean-based pipeline, `search_clients_daily_glean` (built from `firefox_desktop.serp_events`, `events_stream`, and `metrics` ping data), is being developed in parallel with the legacy `main_summary`-derived datasets described in this article. It introduces new dimensions such as `partner_code`; see the [`search_clients_daily_glean_v1` README](https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_glean_v1/README.md) for details. + The **`sap` column counts all SAP (or direct) searches**. `sap` search counts are collected via [probes](https://firefox-source-docs.mozilla.org/browser/browser/BrowserUsageTelemetry.html#search-telemetry) within the Firefox UI These counts are **very reliable, but do not count follow-on queries**. In 2017-06 we deployed the [`followonsearch` addon], which adds probes for `tagged-sap` and `tagged-follow-on` searches. These columns **attempt to count all tagged searches** by looking for Mozilla partner codes in the URL of requests to partner search engines. These search counts are critical to understanding revenue since they exclude untagged searches and include follow-on searches. However, these search counts have **important caveats affecting their reliability**. See [In Content Telemetry Issues](#in-content-telemetry-issues) for more information. diff --git a/src/metrics/dimensions.md b/src/metrics/dimensions.md index 506538628..e22c8b6df 100644 --- a/src/metrics/dimensions.md +++ b/src/metrics/dimensions.md @@ -2,6 +2,8 @@ Here we provide definitions and information about our standard slicing dimensions. +Note: the canonical, maintained dimension definitions now live as `[dimensions.*]` TOML blocks in the [metric-hub](https://github.com/mozilla/metric-hub) repository (see `definitions/*.toml` and `opmon/*.toml`), consumed programmatically by OpMon, Jetstream, and Looker. This page remains a stub template and does not reflect current practice. + Table of contents with links to dimensions. --- @@ -16,7 +18,7 @@ TL;DR: two sentence max. - Definitions of what user has to do to qualify - Definitions of what telemetry must be sent to qualify. - At least one working definition -- Link to a scheduled STMO query (link with `stmocli`?) +- Link to the dimension's `[dimensions.*]` TOML block (data source and `select_expression`) in [metric-hub](https://github.com/mozilla/metric-hub) - Possibly an API-linked graph from STMO - Common Issues: Failures and Gotchas - Resources diff --git a/src/metrics/metrics.md b/src/metrics/metrics.md index 8815fbc08..110788a6b 100644 --- a/src/metrics/metrics.md +++ b/src/metrics/metrics.md @@ -26,6 +26,8 @@ DAU counts unique profiles. Keep in mind that a [profile is not necessarily a us The standard concept of active varies by product, but generally, active users are defined as unique profiles that have sent a `main` ping (on desktop) or a `baseline` ping (on mobile). The precise criteria are defined in the [`usage criterion`](./usage.md) section of this documentation. +DAU, WAU, and MAU values are pre-calculated and available as the single source of truth in the `active_users_aggregates` BigQuery table, which covers both desktop and mobile products. Related mobile engagement and retention metrics are computed from the `mobile_engagement` and `mobile_retention` tables. + We can also restrict the metric to alternative usage criteria. It's **critical to clearly state any non-standard usage criteria on the metric**. The metrics team suggest the following format: `DAU(usage criterion)`. For example, we might be interested in the count of profiles that have viewed more than 5 URIs in a day. We'd denote that metric as `DAU(URI > 5)`. Some common alternative usage criteria documented in the [`usage criterion` section](./usage.md). diff --git a/src/metrics/policy.md b/src/metrics/policy.md index 8d95354a8..a663efc83 100644 --- a/src/metrics/policy.md +++ b/src/metrics/policy.md @@ -4,10 +4,16 @@ Metric standards are maintained by the Metrics Standardization Group (MSG) (link ## How to get more information +The canonical source of truth for metric definitions and metadata is [Metric Hub](https://github.com/mozilla/metric-hub), a repository of metric, data source, and segment definitions that are reviewed before merging. See the [Metric Hub concepts page](https://docs.telemetry.mozilla.org/concepts/metric_hub.html) for more. + +Supporting metric information can also be found on the [Metrics Inventory](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/620494911/Metrics+Inventory) page in Confluence (Mozilla LDAP required). + ## How to propose a new standard metric, criteria, or dimensions ## How to propose a change to a standards +Metric definitions are not deleted once added to [Metric Hub](https://github.com/mozilla/metric-hub); other tools and configurations may still reference them. To retire a metric, set `deprecated = true` on its definition instead of removing it. See the [Metric Hub concepts page](https://docs.telemetry.mozilla.org/concepts/metric_hub.html) for details. + ## Recommendations around citing standards for presentations to senior audiences Something about how all metrics presented should have references to definitions here or explanations as to why a nonstandard metric was used.