Question: Custom runners for worker processes

[pastelsky]

Hi there,

If one want to run OpenWorkflow workers on using custom runners / workers (e.g., GitHub Actions or Bitbucket Pipelines). The idea is to provision workers via our runner APIs and have them poll the DB. Is this a supported pattern, or are workers expected to be long-running processes that the open workflow services host itself?

If it's supported, any guidance on best practices for short-lived/ephemeral runners vs. long-lived workers would be appreciated.

Thanks!

[jamescmartinez]

well, it's not something I've tried before, but it should work.

OpenWorkflow workers are stateless and coordinate entirely via the database, so they can run in any environment that can run Node and connect to your database.

they're designed to be long-running processes, but when a worker is shut down (by you or runtime timeouts), the work will resume when the next worker spins up.

if you are ok with spinning them up on a regular basis to perform the work (e.g., maybe using a GitHub Actions cron?), then everything should work.

one caveat is that the npx @openworkflow/cli worker start command runs workers indefinitely. for ephemeral runners, we probably want to an expose options for the worker to process pending runs and then exit to save costs. until then, expect the worker to run until it times out.

[jamescmartinez]

it's an interesting use case - I'd love to hear how this works so we can make it better & support it natively

if you want to work on this together, find me in our Discord here: https://discord.openworkflow.dev