Skip to content

Latest commit

 

History

History
220 lines (159 loc) · 8.17 KB

README.md

File metadata and controls

220 lines (159 loc) · 8.17 KB

Function Stream

中文 | English

Function Stream is a high-performance, event-driven stream processing framework built in Rust. It provides a modular runtime to orchestrate serverless-style processing functions compiled to WebAssembly (WASM), supporting functions written in Go, Python, and Rust.

Table of Contents

Key Features

  • Event-Driven WASM Runtime: Executes polyglot functions (Go, Python, Rust) with near-native performance and sandboxed isolation.
  • Durable State Management: Built-in support for RocksDB-backed state stores for stateful stream processing.
  • SQL-Powered CLI: Interactive REPL for job management and stream inspection using SQL-like commands.

Repository Layout

function-stream/
├── src/                     # Core runtime, coordinator, server, config
├── protocol/                # Protocol Buffers definitions
├── cli/                     # SQL REPL client
├── conf/                    # Default runtime configuration
├── docs/                    # Documentation (English & Chinese)
├── examples/                # Sample processors
├── go-sdk/                  # Go SDK and generated WIT bindings
├── python/                  # Python API, Client, and Runtime (WASM)
├── scripts/                 # Build and environment automation scripts
├── Makefile                 # Unified build system
└── Cargo.toml               # Workspace manifest

Prerequisites

  • Rust Toolchain: Stable >= 1.77 (via rustup).
  • Python 3.9+: Required for building the Python WASM runtime.
  • Protoc: Protocol Buffers compiler (for generating gRPC bindings).
  • Build Tools: cmake, pkg-config, OpenSSL headers (for rdkafka).

Quick Start (Local Development)

1. Initialize Environment

We provide an automated script to set up the Python virtual environment (.venv), install dependencies, and compile necessary sub-modules.

make env

2. Build & Run Server

Start the control plane in release mode. The build system will automatically compile the Python WASM runtime if it's missing.

cargo run --release --bin function-stream

3. Run CLI

In a separate terminal, launch the SQL REPL:

cargo run -p function-stream-cli -- --host 127.0.0.1 --port 8080

Building & Packaging

The project uses a standard Makefile to handle compilation and distribution. Artifacts are generated in the dist/ directory.

Build Targets

Command Description
make Alias for make build. Compiles Rust binaries and Python WASM runtime.
make build Builds the Full version (Rust + Python Support).
make build-lite Builds the Lite version (Rust only, no Python dependencies).

Distribution (Packaging)

To create production-ready archives (.tar.gz and .zip), use the dist targets.

make dist

Output:

  • dist/function-stream-<version>.tar.gz
  • dist/function-stream-<version>.zip

For a lightweight distribution without Python WASM runtime:

make dist-lite

Output:

  • dist/function-stream-<version>-lite.tar.gz
  • dist/function-stream-<version>-lite.zip

Running the Distribution

After extracting the release package, you will see the following structure:

function-stream-<version>/
├── bin/
│   ├── function-stream      # The compiled binary
│   ├── cli                  # The compiled CLI binary
│   ├── start-server.sh      # Production startup script
│   └── start-cli.sh         # CLI startup script
├── conf/
│   └── config.yaml          # Default configuration
├── data/                    # Runtime data (e.g. WASM cache)
└── logs/                    # Log directory (created on runtime)

Using the Startup Script (bin/start-server.sh)

We provide a robust shell script to manage the server process, capable of handling environment injection, daemonization, and configuration overrides.

Usage:

./bin/start-server.sh [options]

Options:

Option Description
-d, --daemon Run the server in the background (Daemon mode).
-c, --config <path> Specify a custom configuration file path.
-D <KEY>=<VALUE> Inject environment variables (e.g., -D RUST_LOG=debug).

Examples:

  1. Foreground Mode (Docker / Debugging) — Runs the server in the current shell. Useful for containerized environments (Kubernetes/Docker) or debugging.

    ./bin/start-server.sh

  2. Daemon Mode (Production) — Runs the server in the background, redirects stdout/stderr to logs/.

    ./bin/start-server.sh -d

  3. Custom Configuration — Start with a specific config file.

    ./bin/start-server.sh -d -c config/config.yml

Maintenance

Command Description
make clean Deep clean. Removes target/, dist/, .venv/, and all temporary artifacts.
make env-clean Removes only the Python virtual environment and Python artifacts.
make test Runs the Rust test suite.

Documentation

Document Description
Server Configuration Server Configuration & Operations
Function Configuration Task Definition Specification
SQL CLI Guide Interactive Management Guide
Function Development Management & Development Guide
Python SDK Guide Python SDK Guide

Configuration

The runtime behavior is controlled by conf/config.yaml. You can override the configuration location using environment variables:

export FUNCTION_STREAM_CONF=/path/to/custom/config.yaml

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.