Skip to content

Update Databento to support v3 and report pipeline failures - #14

Closed
gabbypolito wants to merge 1 commit into
mainfrom
no-data
Closed

gabbypolito wants to merge 1 commit into
mainfrom
no-data

Conversation

@gabbypolito

Copy link
Copy Markdown

Airflow logs showed DBN v3 responses failing to decode with the older Databento packages. The pipeline also swallowed symbol-processing errors and reported success even when these errors occurred.

Changes:

  • Pin databento to 0.83.0 and databento-dbn to 0.65.0; regenerate the nested project's Poetry lockfile.
  • Update Docker's Poetry version to 2.3.3.
  • Let all symbols finish, then fail the task if any symbol raised an error.
  • Add a synthetic DBN v3 fixture, decoder test, and failure-reporting regression tests.

Validation:

  • Old decoder reproduced the logged DBN version error.
  • Updated decoder passed the synthetic test, including in the Linux amd64 Airflow image.
  • Five failure-reporting regression tests passed.

Production containers and server code remain unchanged. A real Databento fetch, database insertion, and historical backfill still need verification after deployment.

@gabbypolito gabbypolito closed this Oct 1, 2026
@gabbypolito

Copy link
Copy Markdown
Author

I didn't commit all my changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant