Skip to content

Commonize IR generation and validation to Rust#166

Draft
markovejnovic wants to merge 9 commits into
mainfrom
marko/cli-45-commonize-ir-generation-and-validation-to-rust
Draft

Commonize IR generation and validation to Rust#166
markovejnovic wants to merge 9 commits into
mainfrom
marko/cli-45-commonize-ir-generation-and-validation-to-rust

Conversation

@markovejnovic

Copy link
Copy Markdown
Contributor

Summary

  • Move pipeline IR lowering from Python to Rust_pipeline.py, _keys.py, keygen.py, json_emit.py replaced by lower.rs and keygen.rs in hm-dsl-engine. Rust is now the single source of truth for IR generation.
  • Change the Python→Rust boundary from v0 IR JSON to a simpler "step chain" format. Python serializes raw Step chains; Rust lowers them into PipelineGraph.
  • Cache key resolution ported to Rust with byte-exact parity (verified via hardcoded reference hashes from the Python implementation).
  • Net -858 lines (59 files changed, +2678 / -3536). Eliminates ~6 test files and 5 fixture JSONs that existed solely to bridge Python↔Rust IR compatibility.

What stays the same

  • User-facing Python DSL API (@hm.pipeline, toolchains, Step.sh(), etc.)
  • DslEngine trait (still returns JSON strings)
  • PipelineGraph IR schema
  • Subprocess-based Python invocation model

What changes

  • hm.pipeline([...]) factory form removed (decorator-only now) — was only used for Python-side IR generation
  • SubprocessPythonEngine now deserializes raw step chain envelope and lowers in Rust
  • dump_registry_json() is now zero-arg (cache key params moved to Rust)

PyO3 investigation

Researched three PyO3 approaches (extension module, embedded interpreter, pythonize bridge). Conclusion: PyO3 is viable but unnecessary — moving lowering to Rust without PyO3 solved the dual-maintenance problem with zero new build dependencies. Documented as a future Phase 2 option in the plan.

Test plan

  • cargo test --workspace — all green
  • uv run --extra dev python -m pytest — 520 passed, 1 skipped
  • CLI integration tests (cmd_render, cmd_pipelines, cmd_init) pass
  • Cache key byte-exact parity with Python (12 keygen tests with hardcoded reference hashes)
  • Full Python→Rust roundtrip verified (step chain serialize → lower → PipelineGraph)
  • make docs-generate from simci root (docstrings updated, MDX regen needed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant