Since we will be handling much of Hermes via the UI we need to have access to the status of tasks. Luigi has a "Centralized Scheduler" that is meant for production code that exposes an api to get this info. Currently we are using the "Local Scheduler" which is good enough for basic pipelines on local machines. Switching to the centralized scheduler is generally simple but requires attention for some points:
Since we will be handling much of Hermes via the UI we need to have access to the status of tasks. Luigi has a "Centralized Scheduler" that is meant for production code that exposes an api to get this info. Currently we are using the "Local Scheduler" which is good enough for basic pipelines on local machines. Switching to the centralized scheduler is generally simple but requires attention for some points: