Performance with intermediate table

[LaggAt]

I'm currently working on a live system (with low resources) and do profiling the code on a large dataset. This issue is just a reminder to review the cProfile results and optimize performance as soon as the past data is done. Seems biggest issues are lot of SQL's when working on intermediate tables - this is a less optimized portion of code for now.

So - stay tuned for cProfile / SnakeViz analysis.

[LaggAt]

Ideas for performance improvements are:

• thread for every Dimension we work on (still getting new data while working on old once)
• producer/consumer for intermediate table updating - doing it later but more performant using bulk updates.