From 60ca004319b8d371509c2a71923db14336fbcef4 Mon Sep 17 00:00:00 2001 From: Elad Barkay Date: Thu, 8 Oct 2026 22:32:38 +0300 Subject: [PATCH 1/3] docs: document BigQuery package settings added in 0.26.1 Co-Authored-By: Claude Opus 5.5 --- docs/data-tests/dbt/dbt-package.mdx | 82 +++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/docs/data-tests/dbt/dbt-package.mdx b/docs/data-tests/dbt/dbt-package.mdx index dd868994b..009f32fd7 100644 --- a/docs/data-tests/dbt/dbt-package.mdx +++ b/docs/data-tests/dbt/dbt-package.mdx @@ -56,3 +56,85 @@ the following var: vars: elementary_full_refresh: true ``` + +## BigQuery settings + +The following settings only affect BigQuery. Other warehouses ignore them. + +### Test table expiration + +Elementary stores test results in temporary tables during a dbt invocation, and the `on-run-end` hooks read them when +the invocation ends. On BigQuery, these tables expire after 1 hour. If your dbt invocation runs longer than that, the +tables can expire before the hooks read them, and the hooks fail. + +To keep the tables longer, set their expiration in hours: + +```yaml +vars: + temp_table_expiration_hours: 6 +``` + +This applies to all temporary tables Elementary creates on BigQuery, not only test tables. + +### Nanosecond timestamps + +dbt Fusion and dbt Core 1.11 record run timings (such as `execute_started_at`) with nanosecond precision, which +BigQuery timestamps don't support. Starting with version 0.26.1, Elementary truncates these values to microseconds +when it uploads them. Rows uploaded by earlier versions may still contain nanosecond values, so by default Elementary +also truncates timestamps whenever it casts them on BigQuery. + +The truncation converts each timestamp to a string and back, so BigQuery can't prune partitions on it. As a result, +anomaly tests on partitioned tables scan the whole table, and tables with `require_partition_filter` reject the query. + +To use plain timestamp casts again: + +1. Fix the nanosecond values that earlier versions already uploaded: + + ```shell + dbt run-operation elementary.fix_nanosecond_timing_values + ``` + + This updates the timing columns of `dbt_run_results` and `dbt_source_freshness_results` in place, truncating them + to microseconds. It scans both tables, and only needs to run once. + +2. Disable the truncation: + + ```yaml + vars: + bigquery_truncate_nanosecond_timestamps: false + ``` + + + Run `fix_nanosecond_timing_values` before you set `bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, + queries that read rows with nanosecond values fail. + + +### Partitioning of test result tables + +Starting with version 0.26.1, `elementary_test_results` and `test_result_rows` are partitioned by day on +`created_at`, like `dbt_run_results` and `dbt_invocations`. This reduces the data scanned when Elementary reads +recent test results. + +dbt only applies partitioning when it creates a table, so tables created by earlier versions keep their current +layout and keep working as before. To partition an existing table and keep its rows, rebuild it while no dbt +invocation is running: + +```sql +create table `..elementary_test_results__partitioned` +partition by timestamp_trunc(created_at, day) +as select * from `..elementary_test_results`; +``` + +Then check that the row counts match, drop the original table, and rename the new table to `elementary_test_results`. +Repeat for `test_result_rows`. + + + Don't use a full refresh to partition these tables. They hold your test history, and a full refresh deletes it. + + +To create Elementary tables without partitioning, set: + +```yaml +vars: + bigquery_disable_partitioning: true +``` From 27fcca1b4afe1d3e97b18fd0e673b60b3bef0098 Mon Sep 17 00:00:00 2001 From: Elad Barkay Date: Sun, 11 Oct 2026 11:48:49 +0300 Subject: [PATCH 2/3] docs: move BigQuery settings to the pages where they're needed Co-Authored-By: Claude Opus 5.5 --- .../Anomaly-troubleshooting-guide.mdx | 35 ++++++++ docs/data-tests/dbt/dbt-package.mdx | 82 ------------------- docs/data-tests/dbt/on-run-end_hooks.mdx | 15 ++++ docs/data-tests/dbt/package-models.mdx | 25 ++++++ 4 files changed, 75 insertions(+), 82 deletions(-) diff --git a/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx b/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx index 200b5f884..4c7382cbf 100644 --- a/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx +++ b/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx @@ -164,3 +164,38 @@ If you change it after executing elementary tests, you will need to run a full r If you want the Elementary UI to show data for a longer period of time, use the days-back option of the CLI: **`edr report --days-back 45`** + + + +dbt Fusion and dbt Core 1.11 record run timings (such as `execute_started_at`) with nanosecond precision, which +BigQuery timestamps don't support. Starting with version 0.26.1, Elementary truncates these values to microseconds +when it uploads them. Rows uploaded by earlier versions may still contain nanosecond values, so by default Elementary +also truncates timestamps whenever it casts them on BigQuery. + +The truncation converts each timestamp to a string and back, so BigQuery can't prune partitions on it. As a result, +anomaly tests on partitioned tables scan the whole table, and tables with `require_partition_filter` reject the query. + +To use plain timestamp casts again: + +1. Fix the nanosecond values that earlier versions already uploaded: + + ```shell + dbt run-operation elementary.fix_nanosecond_timing_values + ``` + + This updates the timing columns of `dbt_run_results` and `dbt_source_freshness_results` in place, truncating them + to microseconds. It scans both tables, and only needs to run once. + +2. Disable the truncation: + + ```yaml dbt_project.yml + vars: + bigquery_truncate_nanosecond_timestamps: false + ``` + + + Run `fix_nanosecond_timing_values` before you set `bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, + queries that read rows with nanosecond values fail. + + + diff --git a/docs/data-tests/dbt/dbt-package.mdx b/docs/data-tests/dbt/dbt-package.mdx index 009f32fd7..dd868994b 100644 --- a/docs/data-tests/dbt/dbt-package.mdx +++ b/docs/data-tests/dbt/dbt-package.mdx @@ -56,85 +56,3 @@ the following var: vars: elementary_full_refresh: true ``` - -## BigQuery settings - -The following settings only affect BigQuery. Other warehouses ignore them. - -### Test table expiration - -Elementary stores test results in temporary tables during a dbt invocation, and the `on-run-end` hooks read them when -the invocation ends. On BigQuery, these tables expire after 1 hour. If your dbt invocation runs longer than that, the -tables can expire before the hooks read them, and the hooks fail. - -To keep the tables longer, set their expiration in hours: - -```yaml -vars: - temp_table_expiration_hours: 6 -``` - -This applies to all temporary tables Elementary creates on BigQuery, not only test tables. - -### Nanosecond timestamps - -dbt Fusion and dbt Core 1.11 record run timings (such as `execute_started_at`) with nanosecond precision, which -BigQuery timestamps don't support. Starting with version 0.26.1, Elementary truncates these values to microseconds -when it uploads them. Rows uploaded by earlier versions may still contain nanosecond values, so by default Elementary -also truncates timestamps whenever it casts them on BigQuery. - -The truncation converts each timestamp to a string and back, so BigQuery can't prune partitions on it. As a result, -anomaly tests on partitioned tables scan the whole table, and tables with `require_partition_filter` reject the query. - -To use plain timestamp casts again: - -1. Fix the nanosecond values that earlier versions already uploaded: - - ```shell - dbt run-operation elementary.fix_nanosecond_timing_values - ``` - - This updates the timing columns of `dbt_run_results` and `dbt_source_freshness_results` in place, truncating them - to microseconds. It scans both tables, and only needs to run once. - -2. Disable the truncation: - - ```yaml - vars: - bigquery_truncate_nanosecond_timestamps: false - ``` - - - Run `fix_nanosecond_timing_values` before you set `bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, - queries that read rows with nanosecond values fail. - - -### Partitioning of test result tables - -Starting with version 0.26.1, `elementary_test_results` and `test_result_rows` are partitioned by day on -`created_at`, like `dbt_run_results` and `dbt_invocations`. This reduces the data scanned when Elementary reads -recent test results. - -dbt only applies partitioning when it creates a table, so tables created by earlier versions keep their current -layout and keep working as before. To partition an existing table and keep its rows, rebuild it while no dbt -invocation is running: - -```sql -create table `..elementary_test_results__partitioned` -partition by timestamp_trunc(created_at, day) -as select * from `..elementary_test_results`; -``` - -Then check that the row counts match, drop the original table, and rename the new table to `elementary_test_results`. -Repeat for `test_result_rows`. - - - Don't use a full refresh to partition these tables. They hold your test history, and a full refresh deletes it. - - -To create Elementary tables without partitioning, set: - -```yaml -vars: - bigquery_disable_partitioning: true -``` diff --git a/docs/data-tests/dbt/on-run-end_hooks.mdx b/docs/data-tests/dbt/on-run-end_hooks.mdx index e56940704..c28df3420 100644 --- a/docs/data-tests/dbt/on-run-end_hooks.mdx +++ b/docs/data-tests/dbt/on-run-end_hooks.mdx @@ -79,6 +79,21 @@ vars: The size of the queries depends on the amount of models/tests executed in the run. The time the run results adds to the invocation shouldn't be significant. +## Long invocations on BigQuery + +Elementary stores test results in temporary tables during a dbt invocation, and the `on-run-end` hooks read them when +the invocation ends. On BigQuery, these tables expire after 1 hour. If your dbt invocation runs longer than that, the +tables can expire before the hooks read them, and the hooks fail. + +To keep the tables longer, set their expiration in hours: + +```yaml dbt_project.yml +vars: + temp_table_expiration_hours: 6 +``` + +This applies to all temporary tables Elementary creates on BigQuery, not only test tables. Other warehouses ignore it. + ## Can I disable the `on-run-end` hooks? Yes, but note that this may cause missing results and/or outdated metadata in Elementary report and alerts. diff --git a/docs/data-tests/dbt/package-models.mdx b/docs/data-tests/dbt/package-models.mdx index de3d248c2..c54aa39fe 100644 --- a/docs/data-tests/dbt/package-models.mdx +++ b/docs/data-tests/dbt/package-models.mdx @@ -158,6 +158,31 @@ Failed test row samples. Each row contains a sample of data that caused a test t - `created_at` (timestamp) - When the record was created. + + +Starting with version 0.26.1, `elementary_test_results` and `test_result_rows` are partitioned by day on +`created_at`, like `dbt_run_results` and `dbt_invocations`. This reduces the data scanned when Elementary reads +recent test results. + +dbt only applies partitioning when it creates a table, so tables created by earlier versions keep their current +layout and keep working as before. To partition an existing table and keep its rows, rebuild it while no dbt +invocation is running: + +```sql +create table `..elementary_test_results__partitioned` +partition by timestamp_trunc(created_at, day) +as select * from `..elementary_test_results`; +``` + +Then check that the row counts match, drop the original table, and rename the new table to `elementary_test_results`. +Repeat for `test_result_rows`. + + + Don't use a full refresh to partition these tables. They hold your test history, and a full refresh deletes it. + + + + ### dbt_source_freshness_results _Incremental model_ From 0fed0aa5770b6b0d2db2d27c5b53be26138f37c7 Mon Sep 17 00:00:00 2001 From: Elad Barkay Date: Sun, 11 Oct 2026 16:22:27 +0300 Subject: [PATCH 3/3] docs: say when fix_nanosecond_timing_values can be skipped Co-Authored-By: Claude Opus 5.5 --- .../Anomaly-troubleshooting-guide.mdx | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx b/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx index 4c7382cbf..cc9bbe967 100644 --- a/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx +++ b/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx @@ -177,7 +177,7 @@ anomaly tests on partitioned tables scan the whole table, and tables with `requi To use plain timestamp casts again: -1. Fix the nanosecond values that earlier versions already uploaded: +1. If needed, fix the nanosecond values that earlier versions already uploaded: ```shell dbt run-operation elementary.fix_nanosecond_timing_values @@ -186,6 +186,10 @@ To use plain timestamp casts again: This updates the timing columns of `dbt_run_results` and `dbt_source_freshness_results` in place, truncating them to microseconds. It scans both tables, and only needs to run once. + You can skip this step if no rows with nanosecond values are within the time range that your reports and alerts + read (`days_back`). For example, if you upgraded to 0.26.1 longer ago than that time range, the older rows are + never read. The same is true if you didn't run an earlier Elementary version with dbt Fusion or dbt Core 1.11. + 2. Disable the truncation: ```yaml dbt_project.yml @@ -194,8 +198,8 @@ To use plain timestamp casts again: ``` - Run `fix_nanosecond_timing_values` before you set `bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, - queries that read rows with nanosecond values fail. + If rows with nanosecond values are within that time range, run `fix_nanosecond_timing_values` before you set + `bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, queries that read those rows fail.