diff --git a/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx b/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx index 200b5f884..cc9bbe967 100644 --- a/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx +++ b/docs/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide.mdx @@ -164,3 +164,42 @@ If you change it after executing elementary tests, you will need to run a full r If you want the Elementary UI to show data for a longer period of time, use the days-back option of the CLI: **`edr report --days-back 45`** + + + +dbt Fusion and dbt Core 1.11 record run timings (such as `execute_started_at`) with nanosecond precision, which +BigQuery timestamps don't support. Starting with version 0.26.1, Elementary truncates these values to microseconds +when it uploads them. Rows uploaded by earlier versions may still contain nanosecond values, so by default Elementary +also truncates timestamps whenever it casts them on BigQuery. + +The truncation converts each timestamp to a string and back, so BigQuery can't prune partitions on it. As a result, +anomaly tests on partitioned tables scan the whole table, and tables with `require_partition_filter` reject the query. + +To use plain timestamp casts again: + +1. If needed, fix the nanosecond values that earlier versions already uploaded: + + ```shell + dbt run-operation elementary.fix_nanosecond_timing_values + ``` + + This updates the timing columns of `dbt_run_results` and `dbt_source_freshness_results` in place, truncating them + to microseconds. It scans both tables, and only needs to run once. + + You can skip this step if no rows with nanosecond values are within the time range that your reports and alerts + read (`days_back`). For example, if you upgraded to 0.26.1 longer ago than that time range, the older rows are + never read. The same is true if you didn't run an earlier Elementary version with dbt Fusion or dbt Core 1.11. + +2. Disable the truncation: + + ```yaml dbt_project.yml + vars: + bigquery_truncate_nanosecond_timestamps: false + ``` + + + If rows with nanosecond values are within that time range, run `fix_nanosecond_timing_values` before you set + `bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, queries that read those rows fail. + + + diff --git a/docs/data-tests/dbt/on-run-end_hooks.mdx b/docs/data-tests/dbt/on-run-end_hooks.mdx index e56940704..c28df3420 100644 --- a/docs/data-tests/dbt/on-run-end_hooks.mdx +++ b/docs/data-tests/dbt/on-run-end_hooks.mdx @@ -79,6 +79,21 @@ vars: The size of the queries depends on the amount of models/tests executed in the run. The time the run results adds to the invocation shouldn't be significant. +## Long invocations on BigQuery + +Elementary stores test results in temporary tables during a dbt invocation, and the `on-run-end` hooks read them when +the invocation ends. On BigQuery, these tables expire after 1 hour. If your dbt invocation runs longer than that, the +tables can expire before the hooks read them, and the hooks fail. + +To keep the tables longer, set their expiration in hours: + +```yaml dbt_project.yml +vars: + temp_table_expiration_hours: 6 +``` + +This applies to all temporary tables Elementary creates on BigQuery, not only test tables. Other warehouses ignore it. + ## Can I disable the `on-run-end` hooks? Yes, but note that this may cause missing results and/or outdated metadata in Elementary report and alerts. diff --git a/docs/data-tests/dbt/package-models.mdx b/docs/data-tests/dbt/package-models.mdx index de3d248c2..c54aa39fe 100644 --- a/docs/data-tests/dbt/package-models.mdx +++ b/docs/data-tests/dbt/package-models.mdx @@ -158,6 +158,31 @@ Failed test row samples. Each row contains a sample of data that caused a test t - `created_at` (timestamp) - When the record was created. + + +Starting with version 0.26.1, `elementary_test_results` and `test_result_rows` are partitioned by day on +`created_at`, like `dbt_run_results` and `dbt_invocations`. This reduces the data scanned when Elementary reads +recent test results. + +dbt only applies partitioning when it creates a table, so tables created by earlier versions keep their current +layout and keep working as before. To partition an existing table and keep its rows, rebuild it while no dbt +invocation is running: + +```sql +create table `..elementary_test_results__partitioned` +partition by timestamp_trunc(created_at, day) +as select * from `..elementary_test_results`; +``` + +Then check that the row counts match, drop the original table, and rename the new table to `elementary_test_results`. +Repeat for `test_result_rows`. + + + Don't use a full refresh to partition these tables. They hold your test history, and a full refresh deletes it. + + + + ### dbt_source_freshness_results _Incremental model_