Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -164,3 +164,42 @@ If you change it after executing elementary tests, you will need to run a full r

If you want the Elementary UI to show data for a longer period of time, use the days-back option of the CLI:聽**`edr report --days-back 45`**
</Accordion>

<Accordion title="BigQuery: anomaly tests scan the whole table or fail on require_partition_filter">

dbt Fusion and dbt Core 1.11 record run timings (such as `execute_started_at`) with nanosecond precision, which
BigQuery timestamps don't support. Starting with version 0.26.1, Elementary truncates these values to microseconds
when it uploads them. Rows uploaded by earlier versions may still contain nanosecond values, so by default Elementary
also truncates timestamps whenever it casts them on BigQuery.

The truncation converts each timestamp to a string and back, so BigQuery can't prune partitions on it. As a result,
anomaly tests on partitioned tables scan the whole table, and tables with `require_partition_filter` reject the query.

To use plain timestamp casts again:

1. If needed, fix the nanosecond values that earlier versions already uploaded:

```shell
dbt run-operation elementary.fix_nanosecond_timing_values
```

This updates the timing columns of `dbt_run_results` and `dbt_source_freshness_results` in place, truncating them
to microseconds. It scans both tables, and only needs to run once.

You can skip this step if no rows with nanosecond values are within the time range that your reports and alerts
read (`days_back`). For example, if you upgraded to 0.26.1 longer ago than that time range, the older rows are
never read. The same is true if you didn't run an earlier Elementary version with dbt Fusion or dbt Core 1.11.

2. Disable the truncation:

```yaml dbt_project.yml
vars:
bigquery_truncate_nanosecond_timestamps: false
```

<Warning>
If rows with nanosecond values are within that time range, run `fix_nanosecond_timing_values` before you set
`bigquery_truncate_nanosecond_timestamps` to `false`. Otherwise, queries that read those rows fail.
</Warning>

</Accordion>
15 changes: 15 additions & 0 deletions docs/data-tests/dbt/on-run-end_hooks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,21 @@ vars:
The size of the queries depends on the amount of models/tests executed in the run.
The time the run results adds to the invocation shouldn't be significant.

## Long invocations on BigQuery

Elementary stores test results in temporary tables during a dbt invocation, and the `on-run-end` hooks read them when
the invocation ends. On BigQuery, these tables expire after 1 hour. If your dbt invocation runs longer than that, the
tables can expire before the hooks read them, and the hooks fail.

To keep the tables longer, set their expiration in hours:

```yaml dbt_project.yml
vars:
temp_table_expiration_hours: 6
```

This applies to all temporary tables Elementary creates on BigQuery, not only test tables. Other warehouses ignore it.

## Can I disable the `on-run-end` hooks?

Yes, but note that this may cause missing results and/or outdated metadata in Elementary report and alerts.
Expand Down
25 changes: 25 additions & 0 deletions docs/data-tests/dbt/package-models.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,31 @@ Failed test row samples. Each row contains a sample of data that caused a test t
- `created_at` (timestamp) - When the record was created.
</Accordion>

<Accordion title="Partitioning of elementary_test_results and test_result_rows on BigQuery">

Starting with version 0.26.1, `elementary_test_results` and `test_result_rows` are partitioned by day on
`created_at`, like `dbt_run_results` and `dbt_invocations`. This reduces the data scanned when Elementary reads
recent test results.

dbt only applies partitioning when it creates a table, so tables created by earlier versions keep their current
layout and keep working as before. To partition an existing table and keep its rows, rebuild it while no dbt
invocation is running:

```sql
create table `<project>.<elementary_schema>.elementary_test_results__partitioned`
partition by timestamp_trunc(created_at, day)
as select * from `<project>.<elementary_schema>.elementary_test_results`;
```

Then check that the row counts match, drop the original table, and rename the new table to `elementary_test_results`.
Repeat for `test_result_rows`.

<Warning>
Don't use a full refresh to partition these tables. They hold your test history, and a full refresh deletes it.
</Warning>

</Accordion>

### dbt_source_freshness_results

_Incremental model_
Expand Down
Loading