The communication protocol ought to have an ID associated with it so we can cancel DAGs mid-execution.
Workflow might look something like this:
- Creates ID for new execution
- Adds ID to active requests
- Runs DAG - each node type should make some check to ensure the request ID is still in the active requests
- When DAG finishes, remove ID from active requests
A user could then submit a cancel request that removes the ID from the active requests pool. The next node that checks the active pool will see the ID is no longer around and stop execution. This is especially useful for long-running executions, such as querying an LLM.