Implement Obol, a compiler pipeline for compiling imperative Python to Styx code

README.md

@@ -53,6 +53,9 @@ pip install -r requirements.txt
 *   [`worker`](https://github.com/delftdata/styx/tree/main/worker)
     Styx worker.
 
+*   [`obol`](https://github.com/delftdata/styx/tree/main/obol)
+    The Obol source-to-source compiler that turns sequential Python into Styx operator functions.
+
 ## Container images
 
 The coordinator and worker images are published to the GitHub Container Registry (GHCR).
@@ -180,6 +183,35 @@ with the Kafka, coordinator, and S3-compatible storage environment variables.
 
 To clear the SE: `docker compose down --volumes`
 
+## Obol: write Styx programs as ordinary sequential Python
+
+[`obol`](obol/) is a source-to-source compiler that lets you write distributed
+stateful workflows as plain, type-annotated, object-oriented Python and compiles
+them into the asynchronous, message-passing operator functions that the Styx
+runtime expects.
+
+Instead of hand-decomposing a method into a chain of callbacks, you write
+entities and call their methods, and Obol generates the routing, state
+persistence, and continuation management. Compiled programs inherit Styx's
+exactly-once guarantees.
+
+- **Entities** are classes annotated with `@entity`; their `__init__` attributes
+  are their persistent state (essentially the database schema), and `__key__()`
+  returns the routing key Styx partitions on.
+- **Methods** are plain synchronous Python. A call on a value typed as an
+  `@entity` compiles to a remote dispatch; everything else stays local.
+- **Concurrency** is expressed with two constructs: `send_async`
+  (fire-and-forget) and `gather` (fan-out/fan-in), the latter compiling to a
+  failure-durable synchronization barrier.
+
+Under the hood, Obol is a multi-stage pipeline over the libcst syntax tree
+(syntactic preparation, `mypy`-based type resolution, live-variable analysis,
+and CPS-style function splitting at every remote-call boundary). Every
+cross-entity call in the source compiles to exactly one asynchronous dispatch in
+the output, with no extra round-trips introduced.
+
+See the [`obol/` README](obol/README.md) for more.
+
 ##### Cite Styx
 
 ```bibtex

docker-compose-combined.yml

@@ -0,0 +1,249 @@
+name: styx
+
+services:
+  # =========================================
+  # ZOOKEEPER QUORUM
+  # =========================================
+  zoo1:
+    image: confluentinc/cp-zookeeper:7.4.0
+    hostname: zoo1
+    container_name: zoo1
+    ports:
+      - "2181:2181"
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_SERVER_ID: 1
+      ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
+    healthcheck:
+      test: ["CMD", "cub", "zk-ready", "localhost:2181", "10"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  zoo2:
+    image: confluentinc/cp-zookeeper:7.4.0
+    hostname: zoo2
+    container_name: zoo2
+    ports:
+      - "2182:2182"
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2182
+      ZOOKEEPER_SERVER_ID: 2
+      ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
+    healthcheck:
+      test: ["CMD", "cub", "zk-ready", "localhost:2182", "10"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  zoo3:
+    image: confluentinc/cp-zookeeper:7.4.0
+    hostname: zoo3
+    container_name: zoo3
+    ports:
+      - "2183:2183"
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2183
+      ZOOKEEPER_SERVER_ID: 3
+      ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
+    healthcheck:
+      test: ["CMD", "cub", "zk-ready", "localhost:2183", "10"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  # =========================================
+  # KAFKA CLUSTER
+  # =========================================
+  kafka1:
+    image: confluentinc/cp-kafka:7.4.0
+    hostname: kafka1
+    container_name: kafka1
+    ports:
+      - "9092:9092"
+    stop_grace_period: 10s  
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
+      KAFKA_BROKER_ID: 1
+      KAFKA_MESSAGE_MAX_BYTES: 134217728      
+      KAFKA_REPLICA_FETCH_MAX_BYTES: 134217728  
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
+      KAFKA_MIN_INSYNC_REPLICAS: 1
+      KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 6000     
+      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000  
+    healthcheck:
+      test: ["CMD", "cub", "kafka-ready", "-b", "localhost:9092", "1", "20"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    depends_on:
+      zoo1:
+        condition: service_healthy
+      zoo2:
+        condition: service_healthy
+      zoo3:
+        condition: service_healthy
+
+  kafka2:
+    image: confluentinc/cp-kafka:7.4.0
+    hostname: kafka2
+    container_name: kafka2
+    ports:
+      - "9093:9093"
+    stop_grace_period: 10s  
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:19093,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
+      KAFKA_BROKER_ID: 2
+      KAFKA_MESSAGE_MAX_BYTES: 134217728      
+      KAFKA_REPLICA_FETCH_MAX_BYTES: 134217728  
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
+      KAFKA_MIN_INSYNC_REPLICAS: 1
+      KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 6000    
+      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000 
+    healthcheck:
+      test: ["CMD", "cub", "kafka-ready", "-b", "localhost:9093", "1", "20"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    depends_on:
+      zoo1:
+        condition: service_healthy
+      zoo2:
+        condition: service_healthy
+      zoo3:
+        condition: service_healthy
+
+  kafka3:
+    image: confluentinc/cp-kafka:7.4.0
+    hostname: kafka3
+    container_name: kafka3
+    ports:
+      - "9094:9094"
+    stop_grace_period: 10s  
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:19094,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
+      KAFKA_BROKER_ID: 3
+      KAFKA_MESSAGE_MAX_BYTES: 134217728      # 128MB
+      KAFKA_REPLICA_FETCH_MAX_BYTES: 134217728  # 128MB
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
+      KAFKA_MIN_INSYNC_REPLICAS: 1
+      KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 6000     
+      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000  
+    healthcheck:
+      test: ["CMD", "cub", "kafka-ready", "-b", "localhost:9094", "1", "20"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    depends_on:
+      zoo1:
+        condition: service_healthy
+      zoo2:
+        condition: service_healthy
+      zoo3:
+        condition: service_healthy
+
+  # =========================================
+  # STORAGE & OBSERVABILITY
+  # =========================================
+  rustfs:
+    image: rustfs/rustfs:latest
+    ports:
+      - "9000:9000"
+    volumes:
+      - rustfs-data:/data
+
+  prometheus:
+    image: prom/prometheus:v3.2.1
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+
+  grafana:
+    image: grafana/grafana:11.5.2-ubuntu
+    ports:
+      - "3001:3000"
+    environment:
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+      - GF_AUTH_DISABLE_LOGIN_FORM=true
+    volumes:
+      - grafana_data:/var/lib/grafana
+      - ./grafana/provisioning:/etc/grafana/provisioning
+      - ./grafana/dashboards:/etc/grafana/dashboards
+
+  # =========================================
+  # STYX COMPONENTS
+  # =========================================
+  coordinator:
+    build:
+      context: .
+      dockerfile: coordinator/coordinator.dockerfile
+    image: dev/styx-coordinator:latest
+    ports:
+      - "8886:8888"
+      - "8000:8000"
+    environment:
+      - KAFKA_URL=kafka1:19092
+      - HEARTBEAT_LIMIT=5000
+      - HEARTBEAT_CHECK_INTERVAL=500
+      - S3_ENDPOINT=http://rustfs:9000
+      - S3_ACCESS_KEY=rustfsadmin
+      - S3_SECRET_KEY=rustfsadmin
+    depends_on:
+      kafka1:
+        condition: service_healthy
+      kafka2:
+        condition: service_healthy
+      kafka3:
+        condition: service_healthy
+      rustfs:
+        condition: service_started
+      prometheus:
+        condition: service_started
+      grafana:
+        condition: service_started
+
+  worker:
+    build:
+      context: .
+      dockerfile: worker/worker.dockerfile
+    image: dev/styx:latest
+    environment:
+      - INGRESS_TYPE=KAFKA
+      - KAFKA_URL=kafka1:19092
+      - DISCOVERY_HOST=coordinator
+      - DISCOVERY_PORT=8888
+      - S3_ENDPOINT=http://rustfs:9000
+      - S3_ACCESS_KEY=rustfsadmin
+      - S3_SECRET_KEY=rustfsadmin
+    depends_on:
+      coordinator:
+        condition: service_started
+      kafka1:
+        condition: service_healthy
+      rustfs:
+        condition: service_started
+    deploy:
+      replicas: 3
+
+volumes:
+  rustfs-data:
+  grafana_data:

docs/obol/index.md

@@ -0,0 +1,49 @@
+# Obol
+
+Obol is a source-to-source compiler that turns sequential, type-annotated,
+object-oriented Python into the asynchronous, message-passing Styx operator
+functions. You write entities and call their methods as if everything ran in a
+single process, and Obol synthesizes the routing, state persistence, and
+continuation management required by the Styx runtime — without weakening its
+serializable, exactly-once transactional guarantees.
+
+```python
+@entity
+class User:
+    def __init__(self, name: str, balance: int):
+        self.name = name
+        self.balance = balance
+        self.myitems: list[Item] = []
+
+    def __key__(self) -> str:
+        return self.name
+
+    def buy_item(self, amount: int, item: Item) -> bool:
+        total_price = amount * item.get_price()   # cross-entity call, written as a normal call
+        if self.balance < total_price:
+            raise NotEnoughBalance("Not enough balance.")
+        item.update_stock(-amount)
+        self.balance -= total_price
+        self.myitems.append(item)
+        return True
+```
+
+Obol compiles this into a chain of registered Styx step functions, split at each
+remote call, with live variables threaded across every asynchronous boundary via
+an explicit `reply_to` continuation stack.
+
+The compiler lives under [`obol/`](https://github.com/delftdata/styx/tree/main/obol)
+in the repository; see its `README.md` for the full programming model, the
+compilation pipeline, limitations, and development instructions.
+
+## API Reference
+
+The Obol surface a program uses is the set of DSL intrinsics in `obol.api`:
+`entity`, `send_async`, `gather`, `get_entity_by_key`, and `exists`. (The
+compiler internals under `obol/src/obol` are not part of the public API.)
+
+::: obol.api
+    options:
+      show_submodules: false
+      show_root_heading: true
+      show_overloads: false

mkdocs.yml

@@ -5,6 +5,7 @@ nav:
   - Docs:
     - Overview: styx-docs/overview.md
     - Quickstart: styx-docs/quickstart.md
+    - Obol: obol/index.md
     - Styx Operator: styx-docs/styx-operator.md
     - Stateful Function: styx-docs/styx-stateful-function.md
     - Stateflow Graph: styx-docs/styx-stateflow-graph.md
@@ -78,7 +79,7 @@ plugins:
   - mkdocstrings:
       handlers:
         python:
-          paths: ["styx-package/styx"]
+          paths: ["styx-package/styx", "obol/src"]
           options:
             show_source: false
             merge_init_into_class: true