Skip to content

createUserJob is not using create_metadata_engine from airflowLocalSettings #71664

Description

@raphaelauv

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.3.1

What happened and how to reproduce it?

when using a custom airflowLocalSettings ( airflow_local_settings.py ) in the helm chart

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN : postgresql://airflow_iam_user:@XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.rds.amazonaws.com:5432/airflow_db?sslmode=require
apiServer:
  apiServerConfig: |
    from sqlalchemy import event
    from sqlalchemy.engine import Engine
    import boto3


    def generate_rds_token(host, port, user, region):
      client = boto3.client("rds", region_name=region)
      return client.generate_db_auth_token(
      DBHostname=host, Port=port, DBUsername=user, Region=region
      )


    @event.listens_for(Engine, "do_connect")
    def provide_rds_token(dialect, conn_rec, cargs, cparams):
      print("SYNC FAB TOKEN RDS")
      if dialect.name != "postgresql":
        return
      host = cparams.get("host", "")
      if not host.endswith(".rds.amazonaws.com"):
        return

      cparams["password"] = generate_rds_token(
        user=cparams["user"],
        host=host,
        port=cparams.get("port", 5432),
        region=host.split(".")[2],
      )
      cparams["sslmode"] = "require"

airflowLocalSettings: |-
  from sqlalchemy import create_engine, event
  import boto3


  def generate_rds_token(host, port, user, region):
      client = boto3.client('rds', region_name=region)
      return client.generate_db_auth_token(
          DBHostname=host,
          Port=port,
          DBUsername=user,
          Region=region
      )


  def create_metadata_engine(sql_alchemy_conn, *, engine_args, connect_args):
      connect_args["sslmode"]="require"
      engine = create_engine(
          sql_alchemy_conn,
          connect_args=connect_args,
          **engine_args,
          future=True,
      )

      @event.listens_for(engine, "do_connect")
      def provide_token(dialect, conn_rec, cargs, cparams):
          token = generate_rds_token(
              user=cparams["user"],
              host=cparams["host"],
              port=cparams["port"],
              region=cparams["host"].split(".")[2])
          cparams["password"] = token

      return engine

the createUserJob fail to run but the migrateDatabaseJob run successfully ( the api-server also works thanks to apiServerConfig )

2026-08-16T00:03:43.008206Z [warning  ] FileType is deprecated. Simply open files after parsing arguments. [py.warnings] category=PendingDeprecationWarning filename=/home/airflow/.local/lib/python3.14/site-packages/airflow/cli/cli_config.py lineno=646
2026-08-16T00:03:43.009323Z [warning  ] FileType is deprecated. Simply open files after parsing arguments. [py.warnings] category=PendingDeprecationWarning filename=/home/airflow/.local/lib/python3.14/site-packages/airflow/cli/cli_config.py lineno=868
2026-08-16T00:03:44.406735Z [warning  ] 'return' in a 'finally' block  [py.warnings] category=SyntaxWarning filename=/home/airflow/.local/lib/python3.14/site-packages/flask_appbuilder/baseviews.py lineno=1258
2026-08-16T00:03:44.407012Z [warning  ] 'return' in a 'finally' block  [py.warnings] category=SyntaxWarning filename=/home/airflow/.local/lib/python3.14/site-packages/flask_appbuilder/baseviews.py lineno=1305
2026-08-16T00:03:48.736653Z [warning  ] starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement. [py.warnings] category=StarletteDeprecationWarning filename=/home/airflow/.local/lib/python3.14/site-packages/fastapi/middleware/wsgi.py lineno=1
2026-08-16T00:03:48.839747Z [warning  ] Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. [py.warnings] category=UserWarning filename=/home/airflow/.local/lib/python3.14/site-packages/flask_limiter/_extension.py lineno=364
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 144, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 3319, in raw_connection
    return self.pool.connect()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 448, in connect
    return _ConnectionFairy._checkout(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 1272, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/impl.py", line 178, in _do_get
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/util/langhelpers.py", line 122, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/impl.py", line 176, in _do_get
    return self._create_connection()
           ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 389, in _create_connection
    return _ConnectionRecord(self)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
    self.__connect()
    ~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/util/langhelpers.py", line 122, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/create.py", line 667, in connect
    return dialect.connect(*cargs_tup, **cparams)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/default.py", line 630, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)  # type: ignore[no-any-return]  # NOQA: E501
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.rds.amazonaws.com" (YYY.YY.YY.YYY), port 5432 failed: fe_sendauth: no password supplied


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/__main__.py", line 61, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/utils/cli.py", line 113, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/providers/fab/auth_manager/cli_commands/user_command.py", line 69, in users_create
    with get_application_builder() as appbuilder:
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/python/lib/python3.14/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/providers/fab/auth_manager/cli_commands/utils.py", line 69, in get_application_builder
    yield _return_appbuilder(flask_app, db)
          ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/providers/fab/auth_manager/cli_commands/utils.py", line 47, in _return_appbuilder
    init_airflow_session_interface(app, db)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/providers/fab/www/extensions/init_session.py", line 48, in init_airflow_session_interface
    app.session_interface = AirflowDatabaseSessionInterface(
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        app=app,
        ^^^^^^^^
    ...<7 lines>...
        use_signer=True,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/airflow/.local/lib/python3.14/site-packages/airflow/providers/fab/www/session.py", line 62, in __init__
    super().__init__(*args, **kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/flask_session/sqlalchemy/sqlalchemy.py", line 112, in __init__
    self.sql_session_model.__table__.create(bind=engine, checkfirst=True)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/sql/schema.py", line 1289, in create
    bind._run_ddl_visitor(ddl.SchemaGenerator, self, checkfirst=checkfirst)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 3269, in _run_ddl_visitor
    with self.begin() as conn:
         ~~~~~~~~~~^^
  File "/usr/python/lib/python3.14/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 3259, in begin
    with self.connect() as conn:
         ~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 3295, in connect
    return self._connection_cls(self)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
    Connection._handle_dbapi_exception_noconnection(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        err, dialect, engine
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 2450, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 144, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 3319, in raw_connection
    return self.pool.connect()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 448, in connect
    return _ConnectionFairy._checkout(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 1272, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/impl.py", line 178, in _do_get
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/util/langhelpers.py", line 122, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/impl.py", line 176, in _do_get
    return self._create_connection()
           ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 389, in _create_connection
    return _ConnectionRecord(self)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
    self.__connect()
    ~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
    with util.safe_reraise():
         ~~~~~~~~~~~~~~~~~^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/util/langhelpers.py", line 122, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/create.py", line 667, in connect
    return dialect.connect(*cargs_tup, **cparams)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/sqlalchemy/engine/default.py", line 630, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)  # type: ignore[no-any-return]  # NOQA: E501
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.14/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.rds.amazonaws.com" (YYY.YY.YY.YYY), port 5432 failed: fe_sendauth: no password supplied

(Background on this error at: https://sqlalche.me/e/20/e3q8)

Versions of Apache Airflow Providers

apache-airflow-providers-postgres==7.0.1
apache-airflow-providers-celery==3.23.1
apache-airflow-providers-redis==4.5.0
apache-airflow-providers-cncf-kubernetes==10.21.0
apache-airflow-providers-snowflake==6.16.0
apache-airflow-providers-http==6.0.5
apache-airflow-providers-common-sql==2.1.0
apache-airflow-providers-standard==1.17.0
apache-airflow-providers-fab==3.8.0
apache-airflow-providers-common-compat==1.18.0

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

1.36

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions