Skip to content

README.md

Latest commit

 

History

History
183 lines (133 loc) · 7.97 KB

README.md

File metadata and controls

183 lines (133 loc) · 7.97 KB
Raw

⚡ Agentic API

The stateful, agentic API layer for vLLM, written in Rust 🦀

Run OpenAI-grade agentic workloads (Responses API, server-side tools, Codex) on your own GPUs.

License Rust CI pre-commit

🧠 Overview

vLLM gives you state-of-the-art inference throughput. But real agentic applications need more than raw tokens: they need conversation state, tool-call loops, and multi-turn orchestration. Today, all of that complexity lives in your client code.

Agentic API moves it server-side. It is a Rust-native gateway that sits in front of vLLM and owns the stateful agentic APIs, starting with an OpenAI-compatible Responses API. Your application makes one API call and the server handles the rest: state hydration, tool execution, streaming, and continuation.

flowchart LR
    C(["🧑‍💻 Client<br/>Codex · SDKs · curl"]) -->|"📮 <code>POST /v1/responses</code><br/>🌐 HTTP&nbsp;&nbsp;📡 SSE&nbsp;&nbsp;🔌 WebSocket"| A
    subgraph A ["⚡ Agentic API (Rust 🦀)"]
        direction TB
        S["🔄 State hydration<br/><code>previous_response_id</code>"]
        T["🛠️ Server-side tools<br/>web search · functions"]
        P["💾 Persistence<br/>SQLite response store"]
    end
    A -->|"🚀 <code>POST /v1/responses</code><br/>⚙️ stateless&nbsp;&nbsp;🤝 OpenAI-compatible"| V(["🚀 vLLM core<br/>inference engine"])

    classDef client fill:#FFE8B3,stroke:#F59E0B,stroke-width:2px,color:#7C2D12
    classDef inner fill:#E0E7FF,stroke:#6366F1,stroke-width:2px,color:#312E81
    classDef engine fill:#DCFCE7,stroke:#22C55E,stroke-width:2px,color:#14532D

    class C client
    class S,T,P inner
    class V engine
    style A fill:#F5F3FF,stroke:#8B5CF6,stroke-width:2px,color:#5B21B6
    linkStyle 0 stroke:#F59E0B,stroke-width:2px
    linkStyle 1 stroke:#22C55E,stroke-width:2px
Loading

Tip

Point OpenAI Codex at Agentic API and drive it entirely with open models served by vLLM. No OpenAI account required.

✨ Key Features

  • 🔄 Stateful conversations: the server manages history via previous_response_id. No client-side message tracking, no replaying full transcripts.
  • 🛠️ Server-side tool execution: an explicit tool-ownership model (gateway / client / provider) decides exactly what runs where. Web search ships today via You.com, and the model executes multi-step tool chains automatically.
  • 📡 Every transport: non-streaming HTTP, server-sent events for token streaming, and full WebSocket support for interactive clients.
  • 🧰 Codex-ready: accepts Codex-shaped Responses traffic out of the box, preserving the tool declarations and response item shapes Codex depends on.
  • 🏃 Background execution: fire-and-forget requests that keep processing server-side.
  • ✅ Compatibility tested: validated against the Open Responses compatibility suite, with replay-cassette tests for real OpenAI and vLLM traffic.

🧭 API Surface

Endpoint Description Status
POST /v1/responses OpenAI-compatible Responses API with state, tools, and streaming ✅
GET /v1/responses WebSocket transport for the Responses API ✅
POST /v1/conversations Conversation management ✅
GET /v1/models Model listing proxied from vLLM ✅
GET /health · GET /ready Liveness and readiness probes ✅
Messages API Anthropic-style stateful messages on shared primitives 🚧 Planned
Interactions API Higher-level agentic workflow surface ⏳ Planned

🚀 Quickstart

1. Serve a model with vLLM. Any recipe from recipes.vllm.ai works:

vllm serve Qwen/Qwen3-30B-A3B-FP8 \
  --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --reasoning-parser qwen3 --port 5050

2. Start Agentic API, pointing it at the vLLM server (set the YOU_* variables to enable built-in web search):

YOU_API_KEY=<your-you.com-api-key> YOU_API_BASE_URL=<you.com-api-base-url> \
  cargo run -p agentic-server -- --llm-api-base http://0.0.0.0:5050

3. Make a stateful call:

curl http://localhost:9000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-FP8",
    "input": "What is new in vLLM this month?",
    "tools": [{"type": "web_search"}]
  }'

Continue the conversation by passing the returned id as previous_response_id, and the server rehydrates everything for you.

🤖 Codex on your own GPUs

Agentic API speaks the Responses wire protocol Codex expects, including WebSockets, so you can run the full Codex experience against open models.

Add a provider to ~/.codex/config.toml:

[model_providers.agentic-api]
name = "agentic-api"
base_url = "http://localhost:9000/v1"
wire_api = "responses"
requires_openai_auth = false
supports_websockets = true

Then launch Codex:

codex --disable image_generation -c model_provider=agentic-api -m Qwen/Qwen3-30B-A3B-FP8

🧩 Tool Ownership Model

Every tool call has exactly one execution path, so nothing runs by accident:

Ownership Who executes it Examples
Gateway-owned Agentic API executes it server-side and continues the loop Web search, file search, MCP-backed tools
Client-owned Preserved and returned to the client Codex shell / editor tools, your functions
Provider-owned Passed through to vLLM or an upstream provider Provider-native tools

Unknown or ambiguous tool shapes are never executed by default. They are preserved and returned.

🏗️ Repository Layout

crates/
├── agentic-server/       # Axum binary, transport handlers (HTTP/SSE/WS), configuration
├── agentic-server-core/  # Protocol types, executor, tool framework, persistence
└── agentic-praxis/       # Praxis gateway integration
docs/                     # MkDocs documentation, ADRs, and design notes

🛠️ Developing

cargo build                                  # build
cargo test                                   # test
cargo clippy --all-targets -- -D warnings    # lint
cargo fmt -- --check                         # format check

Docs are built with MkDocs:

uv venv
uv pip install -r docs/requirements.txt
uv run mkdocs serve

Design and migration decisions are tracked as ADRs in docs/adr/, with deeper design notes in docs/design/. See the full ROADMAP for where the project is heading, and CONTRIBUTING to get involved.

🗺️ Roadmap at a Glance

  • Responses API hydration: stateful continuation with previous_response_id
  • Codex support: practical Codex sessions through the Responses API
  • Server-side tool execution: explicit ownership, web search built in
  • Messages API: built on the same persistence and execution primitives
  • Interactions API: durable, higher-level agentic workflows
  • Production hardening: storage backends, observability, cached-prefix continuation

📄 License

Licensed under the Apache License 2.0.

⭐ If Agentic API saves you from writing one more client-side tool loop, star the repo! ⭐