feat(data-ngin): upgrade Airflow 2.10 to 3.3.2

[pranavb05]

Summary

Airflow 2 is end-of-life; this moves the data-ngin Airflow image and compose stack to 3.3.2.

• Dockerfile.airflow: base apache/airflow:3.3.2-python3.11; apache-airflow==3.3.2 is pinned in the same pip install so a service dependency can't silently move Airflow (this is what surfaced the SQLAlchemy conflict below).
• docker-compose.yml rewritten for Airflow 3:
  • airflow-webserver → airflow-api-server; new airflow-dag-processor.
  • FAB auth manager kept, so existing users carry over and Dex OIDC can be added later.
  • Shared AIRFLOW_JWT_SECRET / AIRFLOW_SECRET_KEY and EXECUTION_API_SERVER_URL.
  • Removes the hardcoded postgres superuser DSN: AIRFLOW_DB_CONN now comes from .env. The password is still in git history and should be rotated.
  • Only /opt/airflow/logs is a volume. Mounting all of /opt/airflow meant DAGs baked into the image were shadowed by the volume's first-run copy.
• DAG: imports move to airflow.sdk.
• SQLAlchemy >=2.0,<3 (was <2.0; Airflow 3 requires 2.x). The only SQLAlchemy code is db_models.py, which now uses the 2.0-compatible declarative_base import. The workspace lock goes to 2.1.3.
• platform_db: DatabaseConfig.url() uses postgresql+psycopg2://, because SQLAlchemy 2.1 made bare postgresql:// mean psycopg 3.

Not yet deployable as-is

• The DAG's default DATA_NGIN_CONFIG_PATH (/opt/airflow/data_engine/...) and the contracts CSVs match the standalone repo's layout on the box, not this image. Triggered tasks fail with Configuration file not found. This predates the PR: the monorepo Airflow stack has never been deployed.
• Back up the airflow_metadata schema before running airflow db migrate against prod.
• The box .env needs AIRFLOW_JWT_SECRET and AIRFLOW_SECRET_KEY (openssl rand -hex 32).
• AlgoGators/data-ngin has an earlier, unmerged Airflow 3.1 migration (feat/airflow-3-sqlalchemy-2-migration); reconcile with that work before cutting over.

Test plan

• just lint / typecheck / test on data-ngin, platform/db, research-api and libs/algosystem
• Image builds; airflow version reports 3.3.2, FAB provider installed
• Full stack against a throwaway postgres:16: init migrates, api-server/scheduler/dag-processor healthy, DAG parses with no import errors, a triggered run executes tasks through the execution API (they fail only on the missing config path above)
• Deploy to the algocloud box after the config path is resolved

[pranavb05]

Moving this to the standalone AlgoGators/data-ngin repo, which is what the algocloud box deploys from.