Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Airflow in Datacoves: Key Things to Know First"
sidebar_label: What to know
description: "Important notes before writing Airflow DAGs in Datacoves: Ruff linting, Datacoves decorators, and My Airflow for faster personal DAG development."
sidebar_position: 1
sidebar_position: 10
id: airflow-index
---
# Airflow - What to know
Expand Down
83 changes: 83 additions & 0 deletions docs/how-tos/airflow/airflow-3-migration-check.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
title: Check your DAGs for Airflow 3 migration issues
sidebar_label: Airflow 3 migration check
description: "Use the datacoves airflow3-check command to find and auto-fix Airflow 2 to 3 migration issues in your DAG code before upgrading your environment."
sidebar_position: 20
---
# Check your DAGs for Airflow 3 migration issues

:::info
The `datacoves airflow3-check` command is available in Datacoves 6.1 and later. It works regardless of the Airflow version your environment runs, so you can (and should) use it while still on Airflow 2.
:::

When an environment is upgraded from Airflow 2 to Airflow 3, DAG code written for Airflow 2 may rely on imports, arguments, or behaviors that were removed or deprecated in Airflow 3, for example:

| Airflow 2 pattern | Airflow 3 replacement |
| --- | --- |
| `from airflow.decorators import dag, task` | `from airflow.sdk import dag, task` |
| `from airflow.operators.bash import BashOperator` | `from airflow.providers.standard.operators.bash import BashOperator` |
| `schedule_interval="@daily"` | `schedule="@daily"` |
| `email_on_failure` / `email_on_retry` in `default_args` | SMTP notifications (`SmtpNotifier`) |
| Runtime-varying values in DAG definitions (e.g. `datetime.today()` in `start_date`) | Static values; Airflow 3 creates a new DAG version on every parse otherwise |

Finding every occurrence across a repository by hand is tedious. The Datacoves code-server image ships the `datacoves airflow3-check` command, which scans your DAG code for all of these using [ruff's Airflow lint rules](https://docs.astral.sh/ruff/rules/#airflow-air), maintained alongside the Airflow project itself.

## Run the check

Open a terminal in your Datacoves workbench and run:

```bash
datacoves airflow3-check
```

By default it checks the DAGs folder configured for your environment. Each finding shows the file, line, offending pattern, and the suggested replacement:

```
airflow3-suggested-update: `airflow.decorators.dag` is removed in Airflow 3.0
--> orchestrate/dags/daily_run.py:13:2
help: `dag` has been moved to `airflow.sdk` since Airflow 3.0
```

The command exits with code `0` when no issues are found and `1` when there are findings, so you can also use it in scripts or CI.

To check a different directory (for example a scratch folder, or a repo with a non-standard layout), pass it explicitly:

```bash
datacoves airflow3-check path/to/dags
```

## Apply automatic fixes

Some issues have unambiguous, safe fixes (for example `schedule_interval` -> `schedule`). Apply them with:

```bash
datacoves airflow3-check --fix
```

Changes that could alter behavior, such as rewriting imports to their new locations, are reported but not applied automatically; review each finding's `help:` line and update the code by hand.

:::tip
Commit or stash your work before running `--fix` so you can review the changes with `git diff`.
:::

## Recommended workflow

1. Run `datacoves airflow3-check` on your repository while the environment is still on Airflow 2 and work through the findings. Fixes like `schedule_interval` -> `schedule` already work on recent Airflow 2 releases; for new-location imports such as `airflow.sdk` (Airflow 3 only), use a portable pattern while both versions are in play:

```python
try:
# Airflow 3
from airflow.sdk import dag, task
except ImportError:
# Airflow 2
from airflow.decorators import dag, task
```

The check recognizes this pattern and will not flag the Airflow 2 import inside the `except` branch.

2. Re-run it until it reports no issues.
3. Coordinate the environment upgrade with your Datacoves administrator.

:::note
The check covers Python DAG files. DAGs generated from YAML definitions (`dbt-coves generate airflow-dags`) are regenerated by the tooling and do not usually need manual migration.
:::
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/api-triggered-dag.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: How to Trigger an Airflow DAG Using Datasets
sidebar_label: Trigger DAG via Datasets
description: "Step-by-step guide to triggering Airflow DAGs via dataset events in Datacoves, enabling data-driven pipeline orchestration without time-based schedules."
sidebar_position: 6
sidebar_position: 140
---
# How to Trigger a DAG using Datasets

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/customize-worker-environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Set Up a Custom Environment for Airflow Workers
sidebar_label: Worker - Custom Environment
description: "Configure a custom Docker image for Airflow workers in Datacoves to include specific Python packages, libraries, or dependencies your DAGs require."
sidebar_position: 39
sidebar_position: 150
---
# How to set up a custom environment for your Airflow workers

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/dags/_category_.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
label: DAGs
position: 11
position: 40
link:
type: generated-index
title: DAGs
Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/initial-setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Airflow Configuration Guide for Datacoves Admins
sidebar_label: Initial Setup
description: "Configure Airflow in Datacoves: set DAG paths, git sync branch, Kubernetes executor settings, and environment-level service connections for dbt jobs."
sidebar_position: 1
sidebar_position: 50
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/request-resources-on-workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Request Custom CPU & Memory for Airflow Workers
sidebar_label: Worker - Request CPU & Memory
description: "Configure Kubernetes resource requests for individual Airflow tasks in Datacoves to allocate specific CPU and memory limits per DAG task."
sidebar_position: 40
sidebar_position: 160
---
# How to request more memory or cpu resources on a particular DAG task

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/send-emails.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Send Email Notifications on Airflow DAG Failure
sidebar_label: Notifications - Email
description: "Configure email alerts for Airflow DAG failures in Datacoves: set up SMTP integration, define recipient lists, and handle task-level notifications."
sidebar_position: 34
sidebar_position: 70
---
# How to send email notifications on DAG's failure

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/send-ms-teams-notifications.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Send MS Teams Alerts on Airflow DAG Failure
sidebar_label: Notifications - MS Teams
description: "Configure Microsoft Teams notifications for Airflow DAGs in Datacoves to receive alerts when pipeline tasks succeed, fail, or are retried."
sidebar_position: 35
sidebar_position: 80
---
# How to send Microsoft Teams notifications on DAG's status

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/send-slack-notifications.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Send Slack Notifications on Airflow DAG Status
sidebar_label: Notifications - Slack
description: "Set up Slack alerts for Airflow DAG events in Datacoves: connect a workspace, configure channel routing, and receive success or failure messages."
sidebar_position: 36
sidebar_position: 90
---
# How to send Slack notifications on DAG's status

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/sync-database.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Sync the Internal Airflow Database in Datacoves
sidebar_label: Sync Internal Database
description: "How to sync the internal Airflow metadata database in Datacoves to apply configuration changes, reset state, or recover from database inconsistencies."
sidebar_position: 3
sidebar_position: 130
---

# Sync Airflow database
Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/use-airflow-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Access the Airflow REST API in Datacoves
sidebar_label: Airflow REST API
description: "Authenticate with and call the Airflow REST API in Datacoves: generate API tokens, trigger DAGs, check run status, and query task instances programmatically."
sidebar_position: 2
sidebar_position: 30
---
# How to use the Airflow API

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/use-aws-secrets-manager.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Use AWS Secrets Manager for Airflow in Datacoves
sidebar_label: Secrets - AWS Secrets Manager
description: "Configure AWS Secrets Manager as the secrets backend for Apache Airflow in Datacoves to securely store and retrieve connections and variables."
sidebar_position: 36
sidebar_position: 100
---
# How to use AWS Secrets Manager in Airflow

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/use-azure-key-vault.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Use Azure Key Vault for Airflow in Datacoves
sidebar_label: Secrets - Azure Key Vault
description: "Configure Azure Key Vault as the secrets backend for Apache Airflow in Datacoves to securely store and retrieve connections and variables using Managed Identity."
sidebar_position: 37
sidebar_position: 110
---
# How to use Azure Key Vault in Airflow

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/use-datacoves-secrets-manager.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Use the Datacoves Secrets Manager in Airflow
sidebar_label: Secrets - Datacoves Manager
description: "How to use the built-in Datacoves Secrets Manager to store and inject secrets into Airflow connections and variables without exposing credentials in code."
sidebar_position: 38
sidebar_position: 120
---

# How to use Datacoves Secrets Manager in Airflow
Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/use-key-pair-authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Use Key-Pair Authentication in Airflow
sidebar_label: Key-Pair Authentication
description: "Configure Snowflake key-pair authentication for Airflow connections in Datacoves instead of password-based credentials for improved security."
sidebar_position: 10
sidebar_position: 60
---
# Using Key-Pair Authentication in Airflow

Expand Down
Loading