Skip to content

Drop airflow-core's own dev-group dependency on 6 providers #71641

Description

@coleheflin

What

airflow-core/pyproject.toml's own dev dependency-group hardcodes 6 providers so that uv sync --project airflow-core (a scoped, core-only sync) can still run the full airflow-core test suite:

[dependency-groups]
dev = [
    "apache-airflow-core[all]",
    "apache-airflow-ctl",
    "apache-airflow-devel-common",
    "apache-airflow-task-sdk",
    # TODO(potiuk): eventually we do not want any providers nor apache-airflow extras to be needed for
    # airflow-core tests
    "apache-airflow[pandas,polars]",
    "apache-airflow-providers-amazon",
    "apache-airflow-providers-celery",
    "apache-airflow-providers-cncf-kubernetes",
    "apache-airflow-providers-fab>=2.2.0",
    "apache-airflow-providers-git",
    "apache-airflow-providers-ftp",
]

There's an inline TODO already acknowledging this should go away "eventually," but no tracking issue exists for it.

Why this matters

This is the remaining piece of the "standalone core testing" goal from #60770. That issue reported that uv sync --project airflow-core succeeds but the test suite doesn't actually run clean in that scoped environment. #71637 fixed the concrete, reproducible failures (a missing shared-package import and 11 tests that hard-coded full-provider-set assumptions), so the test suite now passes in this scoped sync — but only because these 6 providers are still installed alongside airflow-core's own dependencies. The environment isn't actually provider-free yet.

Scope of the follow-up

  1. Identify which airflow-core tests genuinely need amazon, celery, cncf-kubernetes, fab, git, and ftp (e.g. via apache.hive-style plugin tests, FAB-based auth-manager tests, connection form widgets that only render via flask_appbuilder, etc.).
  2. For each: either move the test into the relevant provider's own test suite (if it's really testing provider behavior that leaked into airflow-core), or make it skip/adapt gracefully when that specific provider isn't installed (matching the pattern used for the tests fixed in Fix airflow-core tests that silently required the full provider set #71637).
  3. Once nothing in airflow-core/tests needs a provider unconditionally, drop the 6 provider entries (and apache-airflow[pandas,polars]) from airflow-core/pyproject.toml's dev group and remove the TODO comment.
  4. Update contributing-docs/07_local_virtualenv.rst's existing note about standalone core development not being fully supported, once resolved.

This is a larger, multi-PR migration (touching many test files across fab/amazon/celery/cncf.kubernetes/git/ftp-adjacent behavior) rather than a single mechanical change, which is why it's split out from #60770 instead of being done in the same PR.

related: #60770

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions