Skip to content

ROADMAP.md

Latest commit

 

History

History
173 lines (126 loc) · 6.96 KB

ROADMAP.md

File metadata and controls

173 lines (126 loc) · 6.96 KB
Raw

agentic-api Roadmap

agentic-api, also referred to as vLLM Agentic API, is the stateful agentic API layer for vLLM core. It is optimized for vLLM core and implemented in Rust. Its job is to own stateful agentic APIs, including Responses, Messages, and Interactions, and to orchestrate server-side tool execution for tool calls generated by vLLM core.

This roadmap uses:

  • vLLM core for vllm-project/vllm, the inference engine and model-serving project.
  • vLLM Agentic API for this repository, vllm-project/agentic-api.

vLLM Agentic API should let vLLM core focus on inference while this repository handles API state, continuation, tool-call execution, and the higher-level orchestration needed by stateful agentic applications.

Project Goals

1. Responses API Hydration

The first project goal is reliable state hydration for the Responses API.

This means supporting previous_response_id so clients can continue from a stored response without replaying the full conversation history. The server must persist the model-visible items needed for continuation, rehydrate them in the right order, append new input, and send the resulting request to vLLM core.

Core work:

  • Store Responses API state needed for continuation.
  • Rehydrate prior input, assistant output, tool calls, and tool outputs.
  • Preserve OpenAI-compatible request and response shapes.
  • Support streaming and non-streaming Responses API flows.
  • Keep storage semantics explicit and testable.

2. Codex Support Through Responses

The second project goal is to support Codex through the Responses API.

Codex depends on stateful continuation, tool-call preservation, and a broad set of agentic tool and response shapes. agentic-api should accept Codex-compatible Responses traffic, preserve the fields Codex needs, and execute or return tool calls according to the ownership model for each tool.

Core work:

  • Preserve Codex-used tool declarations and response item shapes.
  • Support client-owned tools, gateway-owned tools, and provider-owned tools.
  • Implement the tool calls and API behavior needed for practical Codex sessions.

3. Server-Side Tool Execution

The project should execute server-side tool calls generated by vLLM core.

When vLLM core produces a tool call that is owned by the gateway, vLLM Agentic API should dispatch it, capture the result, append the tool output, and continue the agentic loop. The tool system should make ownership explicit so each tool call has a clear execution path.

Tool ownership means:

  • Gateway-owned tools are executed by vLLM Agentic API.
  • Client-owned tools are preserved and returned to the client for execution.
  • Provider-owned tools are passed through to vLLM core or another upstream provider surface; vLLM Agentic API does not execute them.

Ownership should be resolved from the request, the configured tool registry, and the tool type. vLLM Agentic API should only execute tools that resolve to a configured gateway-owned handler. Unknown, unsupported, or ambiguous tool shapes are preserved and returned or passed through; they are never executed by default.

Initial and expected tool areas include:

  • File search.
  • Web search.
  • Computer use.
  • Shell tools.
  • MCP-backed tools and resources.
  • Function tools owned by the gateway.

4. Messages API

The third major API goal is Messages.

Messages should build on the same persistence, rehydration, and execution foundations as Responses, while exposing the API surface expected by agentic clients that organize state around message objects.

Messages should adapt into shared execution primitives for items, tool calls, and tool outputs instead of forking a Responses-specific execution loop.

Core work:

  • Define the Messages API state model.
  • Map Messages state into the shared execution and storage primitives.
  • Preserve compatibility with Responses where the APIs overlap.
  • Add tests that cover continuation and tool-call behavior through Messages.

5. Interactions API

The fourth major API goal is Interactions.

Interactions should provide the higher-level agentic workflow surface once the Responses and Messages foundations are solid. This work should reuse the same state, orchestration, and tool execution model rather than creating a parallel stack.

Interactions should compose the same API-agnostic execution primitives used by Responses and Messages, adding workflow structure without duplicating the underlying model-call and tool-call loop.

Core work:

  • Define the Interactions API contract.
  • Reuse the shared execution loop for model calls and tool calls.
  • Support durable interaction state.
  • Identify which interaction-level concepts belong in this repository and which belong in clients or gateways.

6. Broader Stateful Agentic APIs

After Responses, Codex support, Messages, and Interactions, the project should add the remaining stateful APIs needed by core agentic applications.

This should be driven by concrete client needs and vLLM core integration points, not by adding broad abstractions ahead of use. Each new API should reuse the common storage, continuation, and tool-execution foundations wherever possible.

Near-Term Focus

The near-term focus is to make the Rust implementation dependable and easy to extend.

  • Keep orchestration logic reusable across standalone and gateway deployments.
  • Keep HTTP serving, gateway integration, and tool execution thin around the shared stateful agentic API implementation.
  • Keep the implementation Rust-first and optimized around vLLM core's inference surface.
  • Treat model aliasing and routing as deployment concerns that may live at the standalone server or gateway edge; they are not part of tool execution semantics.
  • Expand compatibility tests for Responses, Codex-shaped traffic, tool calls, streaming, and continuation.
  • Document behavior as it stabilizes, especially where compatibility or tool ownership rules are subtle.

Longer-Term Direction

Longer-term work should improve production readiness and performance without moving inference responsibilities out of vLLM core.

  • Production storage backends, retention policy, and compaction.
  • Observability for request lifecycle, tool execution, and continuation.
  • Gateway integration with Praxis and other deployment surfaces.
  • Cached prefix continuation and other latency optimizations where vLLM core owns rendering, tokenization, and KV-cache execution.
  • Coordination with vLLM core and llm-d where token identity, prefix routing, and renderer boundaries matter.

Non-Goals For Now

These are intentionally out of scope until the core APIs and execution model are stable.

  • Reimplementing tokenization, chat templates, or model-specific rendering in this repository.
  • Building a broad plugin platform before the built-in execution model is settled.
  • Adding stateful agentic APIs without a concrete client or vLLM core integration need.
  • Moving vLLM core inference responsibilities into vLLM Agentic API.