Skip to content

Improve workflow DX, manifests, and generated APIs#20

Merged
kingwill101 merged 23 commits intomasterfrom
feat/workflow-dx-1-6
Mar 19, 2026
Merged

Improve workflow DX, manifests, and generated APIs#20
kingwill101 merged 23 commits intomasterfrom
feat/workflow-dx-1-6

Conversation

@kingwill101
Copy link
Owner

Summary

This branch overhauls Stem's workflow/task DX and related docs.

It does four main things:

  • makes tasks: and generated bundles the default wiring path instead of manual registries
  • adds stronger workflow introspection and runtime metadata views
  • improves annotated workflow/task generation with typed refs, modules, and payload codecs
  • refreshes the docs/examples around durable workflows, adapters, and operational guidance

Highlights

Workflow runtime and DX

  • add workflow manifests and runtime metadata views for runs, steps, and persisted state
  • clarify flow steps vs script checkpoints across runtime, dashboard, docs, and generated output
  • add StemModule, typed WorkflowRef/WorkflowStartCall, typed task definition helpers, and generated bundle-first builder output
  • add typed workflow resume helpers, typed task result waits, typed workflow event emit helpers, and codec-backed DTO workflow/task payload support
  • keep tasks:-first wiring across Stem, Worker, Canvas, and StemWorkflowApp

Examples and docs

  • add a full ecommerce workflow example
  • expand annotated workflow examples to cover nested calls, context-aware checkpoints, DTO payloads, and typed task helpers
  • add a dedicated workflows section in .site and refresh READMEs/snippets to match the current APIs
  • update docs branding/favicon and clean stale repo/example references

Adapters and stores

  • prepare SQLite/Postgres integrations for Ormed 0.2.0
  • simplify SQLite/Postgres datasource bootstrapping using the newer Ormed helpers
  • remove explicit adapter registration calls where no longer needed
  • fix workflow runtime/store regressions around persisted continuation queue metadata and duplicate caller-supplied run ids

Compatibility and safety

  • restore SimpleTaskRegistry as a deprecated compatibility alias
  • add adapter contract coverage for duplicate workflow run ids and resume queue routing

Testing

Ran targeted validation across the changed areas, including:

  • dart test for packages/stem, packages/stem_builder, packages/stem_memory, packages/stem_sqlite, packages/stem_redis, and packages/stem_postgres workflow/store/bootstrap suites
  • runnable example checks for annotated workflows, durable watchers, stem_builder examples, and ecommerce
  • dart analyze on the changed package sets
  • npm --prefix .site run build

Notes

  • Oracle scratch files were intentionally left uncommitted.
  • The branch is based on master.

@coderabbitai
Copy link

coderabbitai bot commented Mar 19, 2026

Important

Review skipped

Too many files!

This PR contains 265 files, which is 115 over the limit of 150.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 894502c4-b720-4caf-b04c-2e0d6f4540f6

📥 Commits

Reviewing files that changed from the base of the PR and between e8d2f7f and afc51c1.

⛔ Files ignored due to path filters (2)
  • .site/static/img/favicon.ico is excluded by !**/*.ico
  • .site/static/img/stem-icon.png is excluded by !**/*.png
📒 Files selected for processing (265)
  • .gitignore
  • .site/docs/brokers/overview.md
  • .site/docs/comparisons/stem-vs-bullmq.md
  • .site/docs/core-concepts/canvas.md
  • .site/docs/core-concepts/cli-control.md
  • .site/docs/core-concepts/index.md
  • .site/docs/core-concepts/observability.md
  • .site/docs/core-concepts/persistence.md
  • .site/docs/core-concepts/rate-limiting.md
  • .site/docs/core-concepts/routing.md
  • .site/docs/core-concepts/signing.md
  • .site/docs/core-concepts/stem-builder.md
  • .site/docs/core-concepts/tasks.md
  • .site/docs/core-concepts/workflows.md
  • .site/docs/getting-started/developer-environment.md
  • .site/docs/getting-started/first-steps.md
  • .site/docs/getting-started/index.md
  • .site/docs/getting-started/intro.md
  • .site/docs/getting-started/next-steps.md
  • .site/docs/getting-started/observability-and-ops.md
  • .site/docs/getting-started/production-checklist.md
  • .site/docs/getting-started/quick-start.md
  • .site/docs/getting-started/troubleshooting.md
  • .site/docs/scheduler/index.md
  • .site/docs/workers/daemonization.md
  • .site/docs/workers/index.md
  • .site/docs/workers/programmatic-integration.md
  • .site/docs/workers/worker-control.md
  • .site/docs/workflows/annotated-workflows.md
  • .site/docs/workflows/context-and-serialization.md
  • .site/docs/workflows/errors-retries-and-idempotency.md
  • .site/docs/workflows/flows-and-scripts.md
  • .site/docs/workflows/getting-started.md
  • .site/docs/workflows/how-it-works.md
  • .site/docs/workflows/index.md
  • .site/docs/workflows/observability.md
  • .site/docs/workflows/starting-and-waiting.md
  • .site/docs/workflows/suspensions-and-events.md
  • .site/docs/workflows/troubleshooting.md
  • .site/sidebars.ts
  • Taskfile.yml
  • packages/dashboard/CHANGELOG.md
  • packages/dashboard/lib/src/services/models.dart
  • packages/dashboard/lib/src/ui/task_detail.dart
  • packages/dashboard/lib/src/ui/workflows.dart
  • packages/dashboard/pubspec.yaml
  • packages/stem/CHANGELOG.md
  • packages/stem/README.md
  • packages/stem/example/annotated_workflows/README.md
  • packages/stem/example/annotated_workflows/bin/main.dart
  • packages/stem/example/annotated_workflows/lib/definitions.dart
  • packages/stem/example/annotated_workflows/lib/definitions.stem.g.dart
  • packages/stem/example/annotated_workflows/lib/stem_registry.g.dart
  • packages/stem/example/annotated_workflows/pubspec.yaml
  • packages/stem/example/autoscaling_demo/bin/producer.dart
  • packages/stem/example/autoscaling_demo/bin/worker.dart
  • packages/stem/example/autoscaling_demo/lib/shared.dart
  • packages/stem/example/canvas_patterns/chain_example.dart
  • packages/stem/example/canvas_patterns/chord_example.dart
  • packages/stem/example/canvas_patterns/group_example.dart
  • packages/stem/example/dlq_sandbox/bin/producer.dart
  • packages/stem/example/dlq_sandbox/bin/worker.dart
  • packages/stem/example/dlq_sandbox/lib/shared.dart
  • packages/stem/example/docs_snippets/lib/best_practices.dart
  • packages/stem/example/docs_snippets/lib/developer_environment.dart
  • packages/stem/example/docs_snippets/lib/namespaces.dart
  • packages/stem/example/docs_snippets/lib/observability.dart
  • packages/stem/example/docs_snippets/lib/persistence.dart
  • packages/stem/example/docs_snippets/lib/producer.dart
  • packages/stem/example/docs_snippets/lib/production_checklist.dart
  • packages/stem/example/docs_snippets/lib/routing.dart
  • packages/stem/example/docs_snippets/lib/scheduler.dart
  • packages/stem/example/docs_snippets/lib/signing.dart
  • packages/stem/example/docs_snippets/lib/tasks.dart
  • packages/stem/example/docs_snippets/lib/worker_control.dart
  • packages/stem/example/docs_snippets/lib/workers_programmatic.dart
  • packages/stem/example/docs_snippets/lib/workflows.dart
  • packages/stem/example/durable_watchers.dart
  • packages/stem/example/ecommerce/.gitignore
  • packages/stem/example/ecommerce/README.md
  • packages/stem/example/ecommerce/analysis_options.yaml
  • packages/stem/example/ecommerce/bin/server.dart
  • packages/stem/example/ecommerce/database/.gitkeep
  • packages/stem/example/ecommerce/lib/ecommerce.dart
  • packages/stem/example/ecommerce/lib/src/app.dart
  • packages/stem/example/ecommerce/lib/src/database/datasource.dart
  • packages/stem/example/ecommerce/lib/src/database/migrations.dart
  • packages/stem/example/ecommerce/lib/src/database/migrations/m_20260226010000_create_ecommerce_tables.dart
  • packages/stem/example/ecommerce/lib/src/database/models/cart.dart
  • packages/stem/example/ecommerce/lib/src/database/models/cart.orm.dart
  • packages/stem/example/ecommerce/lib/src/database/models/cart_item.dart
  • packages/stem/example/ecommerce/lib/src/database/models/cart_item.orm.dart
  • packages/stem/example/ecommerce/lib/src/database/models/catalog_sku.dart
  • packages/stem/example/ecommerce/lib/src/database/models/catalog_sku.orm.dart
  • packages/stem/example/ecommerce/lib/src/database/models/models.dart
  • packages/stem/example/ecommerce/lib/src/database/models/order.dart
  • packages/stem/example/ecommerce/lib/src/database/models/order.orm.dart
  • packages/stem/example/ecommerce/lib/src/database/models/order_item.dart
  • packages/stem/example/ecommerce/lib/src/database/models/order_item.orm.dart
  • packages/stem/example/ecommerce/lib/src/database/orm_registry.g.dart
  • packages/stem/example/ecommerce/lib/src/domain/catalog.dart
  • packages/stem/example/ecommerce/lib/src/domain/repository.dart
  • packages/stem/example/ecommerce/lib/src/tasks/manual_tasks.dart
  • packages/stem/example/ecommerce/lib/src/workflows/annotated_defs.dart
  • packages/stem/example/ecommerce/lib/src/workflows/annotated_defs.stem.g.dart
  • packages/stem/example/ecommerce/lib/src/workflows/checkout_flow.dart
  • packages/stem/example/ecommerce/ormed.yaml
  • packages/stem/example/ecommerce/pubspec.yaml
  • packages/stem/example/ecommerce/test/server_test.dart
  • packages/stem/example/email_service/bin/enqueuer.dart
  • packages/stem/example/email_service/bin/worker.dart
  • packages/stem/example/encrypted_payload/docker/main.dart
  • packages/stem/example/encrypted_payload/enqueuer/bin/enqueue.dart
  • packages/stem/example/encrypted_payload/worker/bin/worker.dart
  • packages/stem/example/image_processor/bin/api.dart
  • packages/stem/example/image_processor/bin/worker.dart
  • packages/stem/example/microservice/enqueuer/bin/main.dart
  • packages/stem/example/microservice/worker/bin/worker.dart
  • packages/stem/example/mixed_cluster/enqueuer/bin/enqueue.dart
  • packages/stem/example/mixed_cluster/postgres_worker/bin/worker.dart
  • packages/stem/example/mixed_cluster/redis_worker/bin/worker.dart
  • packages/stem/example/monolith_service/bin/service.dart
  • packages/stem/example/ops_health_suite/bin/producer.dart
  • packages/stem/example/ops_health_suite/bin/worker.dart
  • packages/stem/example/ops_health_suite/lib/shared.dart
  • packages/stem/example/otel_metrics/bin/worker.dart
  • packages/stem/example/postgres_tls/bin/enqueue.dart
  • packages/stem/example/postgres_tls/bin/worker.dart
  • packages/stem/example/postgres_worker/enqueuer/bin/enqueue.dart
  • packages/stem/example/postgres_worker/worker/bin/worker.dart
  • packages/stem/example/progress_heartbeat/bin/producer.dart
  • packages/stem/example/progress_heartbeat/bin/worker.dart
  • packages/stem/example/progress_heartbeat/lib/shared.dart
  • packages/stem/example/rate_limit_delay/bin/producer.dart
  • packages/stem/example/rate_limit_delay/bin/worker.dart
  • packages/stem/example/rate_limit_delay/lib/shared.dart
  • packages/stem/example/redis_postgres_worker/enqueuer/bin/enqueue.dart
  • packages/stem/example/redis_postgres_worker/worker/bin/worker.dart
  • packages/stem/example/retry_task/bin/producer.dart
  • packages/stem/example/retry_task/bin/worker.dart
  • packages/stem/example/retry_task/lib/shared.dart
  • packages/stem/example/routing_parity/bin/publisher.dart
  • packages/stem/example/routing_parity/bin/worker.dart
  • packages/stem/example/routing_parity/lib/routing_demo.dart
  • packages/stem/example/scheduler_observability/bin/worker.dart
  • packages/stem/example/scheduler_observability/lib/shared.dart
  • packages/stem/example/signals_demo/bin/producer.dart
  • packages/stem/example/signals_demo/bin/worker.dart
  • packages/stem/example/signals_demo/lib/shared.dart
  • packages/stem/example/signing_key_rotation/bin/producer.dart
  • packages/stem/example/signing_key_rotation/bin/worker.dart
  • packages/stem/example/signing_key_rotation/lib/shared.dart
  • packages/stem/example/stem_example.dart
  • packages/stem/example/task_context_mixed/bin/enqueue.dart
  • packages/stem/example/task_context_mixed/bin/worker.dart
  • packages/stem/example/task_context_mixed/lib/shared.dart
  • packages/stem/example/task_usage_patterns.dart
  • packages/stem/example/unique_tasks/unique_task_example.dart
  • packages/stem/example/worker_control_lab/bin/producer.dart
  • packages/stem/example/worker_control_lab/bin/worker.dart
  • packages/stem/example/worker_control_lab/lib/shared.dart
  • packages/stem/example/workflow_runtime_features_example.dart
  • packages/stem/example/workflows/cancellation_policy.dart
  • packages/stem/example/workflows/runtime_metadata_views.dart
  • packages/stem/example/workflows/sleep_and_event.dart
  • packages/stem/lib/src/bootstrap/stem_app.dart
  • packages/stem/lib/src/bootstrap/stem_client.dart
  • packages/stem/lib/src/bootstrap/stem_module.dart
  • packages/stem/lib/src/bootstrap/workflow_app.dart
  • packages/stem/lib/src/canvas/canvas.dart
  • packages/stem/lib/src/core/contracts.dart
  • packages/stem/lib/src/core/payload_codec.dart
  • packages/stem/lib/src/core/stem.dart
  • packages/stem/lib/src/worker/worker.dart
  • packages/stem/lib/src/workflow/annotations.dart
  • packages/stem/lib/src/workflow/core/flow.dart
  • packages/stem/lib/src/workflow/core/flow_step.dart
  • packages/stem/lib/src/workflow/core/run_state.dart
  • packages/stem/lib/src/workflow/core/workflow_definition.dart
  • packages/stem/lib/src/workflow/core/workflow_ref.dart
  • packages/stem/lib/src/workflow/core/workflow_resume.dart
  • packages/stem/lib/src/workflow/core/workflow_runtime_metadata.dart
  • packages/stem/lib/src/workflow/core/workflow_script.dart
  • packages/stem/lib/src/workflow/core/workflow_step_entry.dart
  • packages/stem/lib/src/workflow/core/workflow_store.dart
  • packages/stem/lib/src/workflow/runtime/workflow_introspection.dart
  • packages/stem/lib/src/workflow/runtime/workflow_manifest.dart
  • packages/stem/lib/src/workflow/runtime/workflow_runtime.dart
  • packages/stem/lib/src/workflow/runtime/workflow_views.dart
  • packages/stem/lib/src/workflow/workflow.dart
  • packages/stem/lib/stem.dart
  • packages/stem/spec.md
  • packages/stem/test/bootstrap/stem_app_test.dart
  • packages/stem/test/bootstrap/stem_client_test.dart
  • packages/stem/test/performance/throughput_test.dart
  • packages/stem/test/soak/soak_test.dart
  • packages/stem/test/unit/canvas/canvas_test.dart
  • packages/stem/test/unit/core/contracts_test.dart
  • packages/stem/test/unit/core/stem_core_test.dart
  • packages/stem/test/unit/core/stem_enqueue_options_test.dart
  • packages/stem/test/unit/core/stem_unique_task_test.dart
  • packages/stem/test/unit/core/task_registry_test.dart
  • packages/stem/test/unit/observability/logging_test.dart
  • packages/stem/test/unit/observability/metrics_integration_test.dart
  • packages/stem/test/unit/redis_components_test.dart
  • packages/stem/test/unit/scheduler/beat_test.dart
  • packages/stem/test/unit/tracing/tracing_test.dart
  • packages/stem/test/unit/worker/task_context_enqueue_integration_test.dart
  • packages/stem/test/unit/worker/task_retry_policy_test.dart
  • packages/stem/test/unit/worker/worker_test.dart
  • packages/stem/test/unit/workflow/in_memory_event_bus_test.dart
  • packages/stem/test/unit/workflow/workflow_manifest_test.dart
  • packages/stem/test/unit/workflow/workflow_metadata_views_test.dart
  • packages/stem/test/unit/workflow/workflow_resume_test.dart
  • packages/stem/test/workflow/workflow_runtime_test.dart
  • packages/stem/tool/proxy_runtime_check.dart
  • packages/stem_adapter_tests/CHANGELOG.md
  • packages/stem_adapter_tests/lib/src/workflow_script_facade_suite.dart
  • packages/stem_adapter_tests/lib/src/workflow_store_contract_suite.dart
  • packages/stem_adapter_tests/pubspec.yaml
  • packages/stem_builder/CHANGELOG.md
  • packages/stem_builder/README.md
  • packages/stem_builder/build.yaml
  • packages/stem_builder/example/README.md
  • packages/stem_builder/example/bin/main.dart
  • packages/stem_builder/example/bin/runtime_metadata_views.dart
  • packages/stem_builder/example/lib/definitions.dart
  • packages/stem_builder/example/lib/definitions.stem.g.dart
  • packages/stem_builder/example/pubspec.yaml
  • packages/stem_builder/lib/src/stem_registry_builder.dart
  • packages/stem_builder/lib/stem_builder.dart
  • packages/stem_builder/pubspec.yaml
  • packages/stem_builder/test/stem_registry_builder_test.dart
  • packages/stem_cli/lib/src/cli/utilities.dart
  • packages/stem_cli/test/unit/cli/cli_tasks_test.dart
  • packages/stem_cli/test/unit/cli/cli_worker_stats_test.dart
  • packages/stem_cli/test/unit/cli/cli_worker_status_test.dart
  • packages/stem_cli/test/unit/cli/cli_workflow_agent_help_test.dart
  • packages/stem_cli/test/unit/cli/cli_workflow_test.dart
  • packages/stem_cli/test/unit/cli/dlq_cli_test.dart
  • packages/stem_memory/CHANGELOG.md
  • packages/stem_memory/lib/src/workflow/store/in_memory_workflow_store.dart
  • packages/stem_memory/pubspec.yaml
  • packages/stem_postgres/CHANGELOG.md
  • packages/stem_postgres/lib/src/connection.dart
  • packages/stem_postgres/lib/src/database/datasource.dart
  • packages/stem_postgres/lib/src/database/seed_runtime.dart
  • packages/stem_postgres/lib/src/workflow/postgres_workflow_store.dart
  • packages/stem_postgres/lib/src/workflow/postgres_workflow_store_new.dart
  • packages/stem_postgres/pubspec.yaml
  • packages/stem_postgres/test/support/postgres_test_harness.dart
  • packages/stem_redis/CHANGELOG.md
  • packages/stem_redis/lib/src/workflow/redis_workflow_store.dart
  • packages/stem_redis/pubspec.yaml
  • packages/stem_redis/test/chaos/worker_resilience_test.dart
  • packages/stem_sqlite/CHANGELOG.md
  • packages/stem_sqlite/lib/src/connection.dart
  • packages/stem_sqlite/lib/src/database/datasource.dart
  • packages/stem_sqlite/lib/src/database/seed_runtime.dart
  • packages/stem_sqlite/lib/src/workflow/sqlite_workflow_store.dart
  • packages/stem_sqlite/pubspec.yaml
  • packages/stem_sqlite/test/backend/sqlite_result_backend_test.dart
  • packages/stem_sqlite/test/broker/sqlite_broker_test.dart
  • packages/stem_sqlite/test/control/sqlite_revoke_store_test.dart
  • packages/stem_sqlite/test/workflow/sqlite_workflow_store_test.dart

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/workflow-dx-1-6
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

You can get early access to new features in CodeRabbit.

Enable the early_access setting to enable early access features such as new models, tools, and more.

@github-actions
Copy link

Package publishing

Package Version Status Publish tag (post-merge)
package:stem 0.1.1 ready to publish stem-v0.1.1
package:stem_adapter_tests 0.1.1 ready to publish stem_adapter_tests-v0.1.1
package:stem_builder 0.1.0 ready to publish stem_builder-v0.1.0
package:stem_cli 0.1.0 already published at pub.dev
package:stem_memory 0.1.0 ready to publish stem_memory-v0.1.0
package:stem_postgres 0.1.1 ready to publish stem_postgres-v0.1.1
package:stem_redis 0.1.1 ready to publish stem_redis-v0.1.1
package:stem_sqlite 0.1.1 ready to publish stem_sqlite-v0.1.1

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@kingwill101 kingwill101 merged commit e22e1de into master Mar 19, 2026
10 checks passed
@kingwill101 kingwill101 deleted the feat/workflow-dx-1-6 branch March 19, 2026 18:31
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: afc51c1f1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1272 to +1276
buffer.writeln(' static final PayloadCodec<$typeCode> $symbol =');
buffer.writeln(' PayloadCodec<$typeCode>(');
buffer.writeln(' encode: (value) => value.toJson(),');
buffer.writeln(
' decode: (payload) => $typeCode.fromJson('

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject or unwrap nullable DTO types in generated codecs

_payloadCodecTypeCode accepts codec-backed DTOs even when the annotated type is nullable, so this emitter will generate PayloadCodec<Foo?> plus value.toJson() and Foo?.fromJson(...). That is invalid Dart, which means any annotated task/workflow parameter, result, or checkpoint that uses an optional DTO will fail codegen/builds instead of producing a usable helper.

Useful? React with 👍 / 👎.

Comment on lines 524 to 528
return StemWorkflowApp.create(
module: module,
workflows: workflows,
flows: flows,
scripts: scripts,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Forward task registrations in StemWorkflowApp.fromClient

This factory now accepts a tasks iterable, but the StemWorkflowApp.create(...) call never passes it through. In the fromClient path, any workflow that depends on handlers supplied via tasks: will start without those handlers in the shared registry, so step-enqueued work can be dead-lettered as an unregistered task even though the caller provided the handler list.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant