feat: DAG workflow engine (phases 2-8)#51
Merged
pratyush618 merged 12 commits intomasterfrom Apr 16, 2026
Merged
Conversation
New taskito-workflows crate with WorkflowDefinition, WorkflowRun, WorkflowNode types, SQLite storage, topology module, and state machine. PyO3 bindings: PyWorkflowBuilder, PyWorkflowHandle, PyWorkflowRunStatus, plus PyQueue methods for submit, mark result, cancel, fan-out expansion, deferred job creation, approval gates, sub-workflow support, incremental caching, and visualization DAG retrieval.
- Workflow builder with step(), gate(), fan_out/fan_in, conditions - WorkflowTracker with conditional orchestration, skip propagation - Fan-out strategy module (each), fan-in aggregation - Approval gate support with timeout and auto-resolve - Sub-workflow composition via WorkflowProxy.as_step() - WorkflowContext for callable conditions - Incremental caching with dirty-set computation - Graph analysis: critical path, execution plan, bottleneck - Mermaid and DOT visualization renderers - Cron-scheduled workflow support via bridge task pattern - Type stubs and event types updated
- test_workflows_linear: 15 tests (Phase 2) - test_workflows_fan_out: 11 tests (Phase 3) - test_workflows_conditions: 13 tests (Phase 4) - test_workflows_gates: 6 tests (Phase 5A) - test_workflows_subworkflow: 4 tests (Phase 5B) - test_workflows_cron: 1 test (Phase 5D) - test_workflows_analysis: 12 tests (Phase 6) - test_workflows_caching: 7 tests (Phase 7) - test_workflows_visualization: 5 tests (Phase 8)
Expand guide/core/workflows.md with DAG workflows covering fan-out, conditions, gates, sub-workflows, cron scheduling, incremental runs, visualization, and graph analysis. Canvas primitives moved to bottom. New api/workflows.md with full API reference for Workflow, WorkflowRun, WorkflowProxy, queue methods, and all type definitions.
The local path ../dagron/crates/dagron-core doesn't exist in CI. Switch to a git dependency pointing at the ByteVeda/dagron repo.
New docs/workflows/ section with 9 pages: - Overview with system architecture diagram - Building Workflows (step config, decorator, node state machine) - Fan-Out & Fan-In (sequence diagram, empty/failure handling) - Conditions & Error Handling (fail_fast vs continue diagrams) - Approval Gates (timeout, events) - Sub-Workflows & Scheduling (composition diagram, cron) - Incremental Runs (dirty-set propagation diagrams) - Analysis & Visualization (critical path, bottleneck, Mermaid/DOT) - Canvas Primitives (comparison table vs DAG workflows) Also fixes mypy CI failures: - Widen Workflow.step() task param to Any (accepts duck-typed objects) - Fix test type annotations for sorted(), tuple hints, bool returns
Define a runtime-checkable Protocol instead of using Any for the task parameter. Accepts TaskWrapper, SubWorkflowRef, or any object with a _task_name attribute — type-safe without being overly rigid.
Break architecture.md into 8 pages: overview, job lifecycle, worker pool, storage (with ER diagram), scheduler, resource system, failure model, and serialization. Replace raw SQL schema with Mermaid erDiagram. Add docs/examples/workflows.md with 6 patterns: ML pipeline with approval gate, map-reduce fan-out, continue mode, multi-region sub-workflows, incremental runs, and pre-execution analysis. Fix workflow index diagram (flowchart TD without subgraph overlap).
CI and publish workflows passed --features explicitly, which replaces pyproject's default set. This dropped 'workflows' from the compiled extension, causing PyWorkflowBuilder ImportError in tests and breaking workflow support in released wheels. - ci.yml: add extension-module,postgres,redis,workflows - publish.yml: add native-async,workflows (4 platforms)
- Swatinem/rust-cache v2.8.2 -> v2.9.1 (Node.js 20 deprecation) - actions/setup-node v4 -> v6 (Node.js 20 deprecation) - cache-dependency-glob uv.lock -> pyproject.toml (uv.lock is gitignored) - jobs.tsx: drop unused statsError, refetchStats destructure
New feature release for the DAG workflow ecosystem. See changelog for full feature list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
taskito-workflowscrate with WorkflowDefinition, WorkflowRun, WorkflowNode types, SQLite storage, topology module, and state machine (25 Rust tests)depends_onjob chains, WorkflowTracker event-driven completionfan_out="each"), result aggregation (fan_in="all")condition="on_success"/"on_failure"/"always"+ callable conditions withWorkflowContext,on_failure="continue"mode, recursive skip propagationWorkflowProxy.as_step(), cron-scheduled workflowscritical_path(),execution_plan(),bottleneck_analysis(),ancestors()/descendants(),topological_levels(),stats()CACHE_HITnode status, dirty-set propagation, cache TTLWorkflowandWorkflowRunTest plan
cargo test --workspace --features workflows— 73 Rust testsuv run python -m pytest tests/python/ -v— 431 Python tests (74 new workflow tests)cargo check --workspace/--features postgres/--features redis— all feature combos compileuv run ruff check py_src/ tests/python/— cleanuv run mypy py_src/taskito/ --no-incremental— 0 errors in 91 files