Switch production deployment to gunicorn

[EvanDietzMorris]

Unless there is another helm chart living somewhere else, we're running Django's development server (manage.py runserver) in production. That is not recommended by Django and can cause serious performance issues:

• It's just one python process with one GIL. CPU intensive tasks block every other process.
• It doesn't have great restart and recycling capabilities, if it gets to a completely hung state there's no recourse and the best case scenario is k8s kills it and spins up an entirely new instance with hard down time in between

Switching to gunicorn with workers solves a lot of those issues:

• Workers have their own python processes with their own GIL which enables real concurrency that can scale by adding more cpu & workers (cpu hogging tasks only block their own worker)
• Workers can crash and restart and are automatically recycled periodically intentionally, while other workers are still processing requests
• Better logging and understanding of these restarts

If we adopt this, we'll want to add the new gunicorn values to our ncats-values files, and it's strongly recommended to tune the number of workers according to the amount of CPU we have allocated. Comments in the chart show where/how to do that.

Along with this, it is highly recommended that we double check the settings not committed to this repo (deploy/configs/settings.py) and make sure Django is not being run in DEBUG mode for CI, test, or prod, and probably we don't need DJANGO_LOG_LEVEL=DEBUG there either. It might be nice to use environment variables configurable from helm values for these to make it easy to set them per-maturity-level.

[github-actions]

Well done! You have taken the first step into a larger world

[MarkDWilliams]

Trying this out and then we can test in CI to see if how it affects performance.

[EvanDietzMorris]

The other thing we might want to look at here is the worker type - https://gunicorn.org/design/#worker-types

gthread is used in this PR because it is the most compatible with the widest variety of other python libraries (least likely to break things in weird ways) but it is entirely possible (even likely) that gevent workers would give us better performance for scaling up more concurrent requests with limited hardware. Either should be an improvement over the Django server though, so we shouldn't let this consideration stop us from deploying with gthread.