Skip to content

Moving to luigi's centralized scheduler #917

@lior-antonov

Description

@lior-antonov

Since we will be handling much of Hermes via the UI we need to have access to the status of tasks. Luigi has a "Centralized Scheduler" that is meant for production code that exposes an api to get this info. Currently we are using the "Local Scheduler" which is good enough for basic pipelines on local machines. Switching to the centralized scheduler is generally simple but requires attention for some points:

  • We will have to add some sort of a "dispatch id" to each run of the pipeline. This is because the centralized scheduler takes all the users' tasks and only differentitates them based on parameters and node names. Providing a "dispatch id" parameter on each execution means each execution can be uniquely identified. Add option for centralized scheduler in hermes execution #918
  • Making sure the centralized scheduler is setup correctly on our end
  • Keeping the option to run Luigi pipelines via the "Local Scheduler" for simple tasks. The change will just give the user an option to use the centralized scheduler. Even though the Centralized Scheduler also adds a many utilities such as a UI, option to programmatically inform on events and progress of tasks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions