[joshuaRiefman]

FEATURE: Worker and Server Orchestration

• Module Documentation
• Method/Class/Function Documentation
• Pull Request Completed

What's New

• Added deliberate orchestration and two-way communication between the server and workers. Workers report progress, and the server can run and manage workers. A server less mode remains where the worker does not report to the server as a fallback.

Deprecated/Removed

• The broker has been replaced by the server. An orphan broker container might remain.

Dependencies

• alembic has been added to manage database migrations.
• The server has had many additions to run a dashboard, such as fastapi.
• mypy, ruff, and pytest now live in a test dependency group.

Notes

[Anything else relevant?]

[jonahjlee]

This PR is too large for me to review in detail, but I read the docs and skimmed through the code and it looks good.

Committing API tokens feels strange even though it's safe in our case...

In terms of the orchestration module: it seems to be working well, but I think that it's a more complex solution that what we really need, since we will probably not be running offline Sunbeam very often. Regardless, there are lots of good changes and other additions in this PR. Thanks for all your work!

[jonahjlee]

Maybe this should be removed, or made to point to ALEMBIC.md

[jonahjlee]

The stages for FSGP 2024 day 1 should match what's currently in realtime for this to work.

[jonahjlee]

I know it's not orchestration related but can we change this to - pgdata:/var/lib/postgresql/data? It makes running on WSL easier.

[jonahjlee]

Add: pgdata:

[joshuaRiefman]

Committing API tokens feels strange even though it's safe in our case...

Agreed. I could remove them but then we'd have to deal with .env files which is just frankly a pain (maybe necessary though).

Agreed that it is definitely more complex than what I set out to do, but I've been struggling to find places to reduce the complexity and I'd be happy to follow through with any suggestions.

since we will probably not be running offline Sunbeam very often

I used to think this too, but my opinion has flipped. We'll be using offline Sunbeam quite often in the process of developing Sunbeam and especially for any future data analysis.

[jonahjlee]

since we will probably not be running offline Sunbeam very often

I used to think this too, but my opinion has flipped. We'll be using offline Sunbeam quite often in the process of developing Sunbeam and especially for any future data analysis.

Good points! I suppose changes to the derivation of values or additions of new ones would mean we would want to rerun the historical pipeline.

Agreed that it is definitely more complex than what I set out to do, but I've been struggling to find places to reduce the complexity and I'd be happy to follow through with any suggestions.

I'll keep this in mind - maybe I'll have better ideas when I've worked more with the repo hands-on.