Git-sync: Support multiple repositories in one AirflowCluster

Current Situation

We have in our CRD's the possibility to configure git-sync using:

  clusterConfig:
    dagsGitSync:
      # Define your repository here. Currently, the public stackable
      - repo: https://github.com/stackabletech/airflow-operator
        branch: "main"
        gitFolder: "tests/templates/kuttl/mount-dags-gitsync/dags"
        depth: 10
        wait: 20s
        # You may uncomment this line, if your repository is private
        # In this case, please fill in config/airflow-git-credentials.yaml
        # credentialsSecret: airflow-git-credentials
        gitSyncConf:
          --rev: HEAD

As of now, we only support one git-sync although this is already, thankfully, a list.

Favoured Situation

Currently, customers would need either support multiple repositories or support multiple branches per repository but ideally both. If we'd need to decide, I'd rather take multiple repositories as this would suffice for ( in my opinion ) much more brought audience.

Issues

I've tested the current status with two separate gitsync repos and it is possible to list multiple gitsync repos, but the target folders get sync-ed into separate folders, whereas Airflow expects everything under AIRFLOW__CORE__DAGS_FOLDER.

Two gitsync containers are created:

[stackable@airflow-scheduler-default-0 ~]$ ls -al /stackable/app/
total 1076
...
drwxrwsrwx 4 root      stackable    4096 Dec  5 08:03 git-0
drwxrwsrwx 4 root      stackable    4096 Dec  5 08:03 git-1
...

But only one location is picked up:

[stackable@airflow-webserver-default-0 ~]$ env | grep DAGS
AIRFLOW__CORE__DAGS_FOLDER=/stackable/app/git-0/current/mount-dags-gitsync/dags_airflow3

See here.

Fixed with: #729

@adwk67

Multi-repo GitSync Support — Design Discussion

Background

We (@adwk67 and @Maleware) had an internal discussion centering on this proposed feature (adding support for multiple git repositories in the GitSync configuration of the Airflow operator). Here's a short summary:

Airflow philosophy

Pro:

Enterprise customers frequently have existing multi-repo structures. A managed platform should accommodate real-world organisational layouts, not just idealised ones. Airflow's philosophy is understandable but may create more work for the customer.
The implementation covered by this ticket still uses a single top-level DAG folder, with repositories placed in subdirectories beneath it, so Airflow's single-folder design is preserved in spirit.
If the feature is not merged, the option to configure multiple GitSync repositories in Airflow should be removed for consistency (it should remain for other products such as NiFi).
Customers should be given a constructive explanation as to why the problem will not be addressed in SDP.

Con:

The implementation requires DAG authors to embed filesystem path details (e.g. git-0/current/dags_airflow3) directly in their DAG code. This couples DAG logic to infrastructure, which customers may find to be fragile and undesirable. If this is breaking, it is not going to be obvious to us as it may require DAG refactoring, a customer responsibility.
- This also applies to customers using single gitsync targets, as the internal path changes.
Consolidating multiple repositories into a single, clean sync target is a customer responsibility. Git tooling such as submodules is the appropriate place to solve this, and that solution remains entirely in the customer's hands. Airflow's own community has argued this position.
Airflow's charts and codebase assume a single DAG folder backed by a shared volume. Supporting multiple repositories is a meaningful departure that could create ongoing maintenance burden and conflict with future upstream Airflow direction. Operator patches should remain patches, not forks.

To be decided

Whether to merge, reject, or restructure the feature
If rejected, whether to also remove the multi-GitSync configuration option
How to communicate the decision (either way, as the feature may break DAGs) to affected customers with a constructive rationale
If we decide against implementing this, we could try and achieve the same using overrides. This would be messy but would at least allow us to offer a solution that does not involve us changing operator code.

Multi-repo GitSync Support — Design Discussion: update following meeting 12.03.2026

we have decided not to implement multi gitsyncs
alternatives are:
- multiple Airflow instances, each with one git backend
- possibly achieve this with overrides
- DAG bundles
we will open an issue to look at DAG bundles (generally, but specifically in this context)
we will open an issue to prevent a CR from defining multiple gitsyncs and document accordingly

After the discussion, we decided to punt on it and explore other (non-inversive) options.

Navigation Menu

Repository navigation

Git-sync: Support multiple repositories in one AirflowCluster #721

Current Situation

Favoured Situation

Issues

Multi-repo GitSync Support — Design Discussion

Background

Airflow philosophy

Pro:

Con:

To be decided

Multi-repo GitSync Support — Design Discussion: update following meeting 12.03.2026

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Git-sync: Support multiple repositories in one AirflowCluster #721

Description

Current Situation

Favoured Situation

Issues

Activity

adwk67 commented on Dec 5, 2025

Maleware commented on Feb 27, 2026

adwk67 commented on Mar 11, 2026

Multi-repo GitSync Support — Design Discussion

Background

Airflow philosophy

Pro:

Con:

To be decided

adwk67 commented on Mar 12, 2026

Multi-repo GitSync Support — Design Discussion: update following meeting 12.03.2026

Maleware commented on Mar 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions