diff --git a/Taskfile.yaml b/Taskfile.yaml index fd3366f..1e9f041 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -80,5 +80,5 @@ tasks: --mypy_out={{.OUT_DIR}} \ --grpc_python_out={{.OUT_DIR}} \ --mypy_grpc_out={{.OUT_DIR}} \ - -I proto + -I proto \ proto/redpanda/runtime/proto/runtime.proto diff --git a/docs/06_rpkmcpendpoint_.md b/docs/06_rpkmcpendpoint_.md deleted file mode 100644 index 8489006..0000000 --- a/docs/06_rpkmcpendpoint_.md +++ /dev/null @@ -1,134 +0,0 @@ -# Chapter 6: RPKMCPEndpoint - -In the [previous chapter: MCPEndpoint](05_mcpendpoint_.md), you learned how an Agent can communicate with external servers or local processes using different connection methods. Now, let’s explore a specialized version called “RPKMCPEndpoint.” Think of it as a shortcut to launch a local “rpk connect mcp-server” command in a folder, so you don’t have to host a remote service elsewhere—perfect for local testing or development. - ---- - -## 1. Why RPKMCPEndpoint? - -Suppose you have a local folder that contains some specialized tools or data transformations you want to test. Instead of separately starting the server and then connecting the Agent to it, you can let RPKMCPEndpoint do both steps at once: -1. Launch the command "rpk connect mcp-server ." -2. Communicate through stdin/stdout as if it were a remote service. - -This means your Agent can “call out” to Tools in that folder, but everything happens locally and on-demand. - ---- - -## 2. A Simple Use Case - -Imagine you have some local data tasks that your Agent can’t (or shouldn’t) handle all by itself (like data cleansing or formatting). Instead of coding it into the Agent directly, you set up a local MCP server script that does it. With RPKMCPEndpoint, one line of code spawns that server process in a specified folder, then the Agent can immediately talk to it. - -### Minimal Example - -Below is a super-short snippet that shows how to: -1. Create an RPKMCPEndpoint for a local directory. -2. Fire up the “mcp-server” command in that directory. -3. Use an Agent to query the newly launched server’s Tools. - -```python -import asyncio -from pathlib import Path -from redpanda.agents import Agent -from redpanda.agents._mcp import RPKMCPEndpoint, mcp_client - -async def local_mcp_demo(): - endpoint = RPKMCPEndpoint(directory=Path("/path/to/my_local_project")) - async with mcp_client(endpoint) as client: - await client.initialize() - available_tools = await client.list_tools() - print("Local Tools:", [t.name for t in available_tools]) - -asyncio.run(local_mcp_demo()) -``` -Explanation: -• We import `RPKMCPEndpoint` and pass a folder path to it. -• Under the hood, it runs “rpk connect mcp-server .” in that folder (the dot refers to the folder’s path). -• We create an `mcp_client(endpoint)` context to talk to that local server. -• We list the Tools to confirm the server is up and running. - ---- - -## 3. How It Works (Step-by-Step) - -Let’s see a simple flow of what happens when you run the code above: - -```mermaid -sequenceDiagram - participant You - participant Endpoint as RPKMCPEndpoint - participant LocalServer as Local MCP Server - participant Client as MCPClient - You->>Endpoint: "Launch and connect" - Endpoint->>LocalServer: Start "rpk connect mcp-server" - Endpoint->>Client: Provide connection via stdin/stdout - Client->>LocalServer: "list_tools" request - LocalServer-->>Client: Tools data - Client-->>You: Tools found -``` - -1. You create an RPKMCPEndpoint and request a connection. -2. RPKMCPEndpoint spawns the local “rpk connect mcp-server” command in your chosen folder. -3. The endpoint sets up stdin/stdout communication with that process. -4. The MCPClient uses this channel to send commands like “list_tools” or “call_tool.” -5. The process replies with the data about the available Tools. -6. You see the final list of Tools in your Python code. - ---- - -## 4. Under the Hood - -RPKMCPEndpoint is simply a subclass of [StdioMCPEndpoint](05_mcpendpoint_.md#simple-use-case) with the command and arguments set to `"rpk connect mcp-server"`. Here’s a simplified look (shortened for clarity) in [src/redpanda/agents/_mcp.py](https://github.com/redpanda-data/agent/blob/main/src/redpanda/agents/_mcp.py): - -```python -class RPKMCPEndpoint(StdioMCPEndpoint): - def __init__(self, directory: Path | str | None, cache_tools: bool = True): - super().__init__( - params=StdioServerParameters( - command="redpanda-connect", - args=["mcp-server", "."], - env={...}, - cwd=directory, - ), - cache_enabled=cache_tools, - ) -``` - -• It passes the local folder as `cwd`. -• It sets up the arguments so that “mcp-server .” is launched inside that folder. -• Communication is done through standard I/O, just like any other MCPEndpoint. - ---- - -## 5. Putting It All Together with an Agent - -Once you have this endpoint, you can attach it to your Agent to seamlessly use any Tools defined in your local server folder. For example: - -```python -from redpanda.agents import Agent -from redpanda.agents._mcp import RPKMCPEndpoint - -endpoint = RPKMCPEndpoint(directory="my_local_project") -my_agent = Agent( - name="LocalAgent", - model="openai/gpt-3.5-turbo", - mcp_endpoints=[endpoint] # We can list multiple endpoints if we want -) - -# Now my_agent can discover Tools from this local server automatically! -``` -Explanation: -• We pass `[endpoint]` to `mcp_endpoints` so our Agent knows where to find remote (or local-remote) Tools. -• The Agent will call out to that folder’s Tools if it sees a request that matches them. - ---- - -## 6. Key Takeaways and Next Steps - -By using RPKMCPEndpoint: -• You can spin up a local server environment on demand—no separate service needed. -• It behaves just like any other remote or local server from the Agent’s perspective. -• Great for testing or when your data pipelines live in the same repo as your Agent. - -Next time, we’ll look more closely at [mcp_client](07_mcp_client_.md), which is the library helper that actually does most of the handshake and messaging between your Python code and the server. You’ll see how it all fits together behind the scenes! - -Enjoy experimenting with your locally spawned server—happy testing! diff --git a/docs/index.md b/docs/index.md index 3f8ce98..9c6a056 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,52 +1,103 @@ -# Welcome to MkDocs +# Redpanda Agents -For full documentation visit [mkdocs.org](https://www.mkdocs.org). +The [Redpanda Agent SDK](https://github.com/redpanda-data/agent) allows you to build agentic [Redpanda Connect](https://www.redpanda.com/connect) pipelines. -## Commands +You can use [`rpk`](https://docs.redpanda.com/current/get-started/intro-to-rpk/) to generate the initial boilerplate for an agentic pipeline. -* `mkdocs new [dir-name]` - Create a new project. -* `mkdocs serve` - Start the live-reloading docs server. -* `mkdocs build` - Build the documentation site. -* `mkdocs -h` - Print help message and exit. +```bash +$ rpk connect agent init my_first_agent -## Project layout +$ ls --tree ./my_first_agent +my_first_agent +├── agents +│ └── weather.py +├── mcp +│ └── resources +│ └── processors +│ └── check_weather_tool.yaml +├── pyproject.toml +├── README.md +├── redpanda_agents.yaml +└── uv.lock +``` - mkdocs.yml # The configuration file. - docs/ - index.md # The documentation homepage. - ... # Other markdown pages, images and other files. +## Project Structure +The project structure is as follows: -**This project** provides a flexible system where an *Agent* can answer questions and use specialized *Tools* to tackle tasks. -The Agent can also connect to external services through different **MCPEndpoints**, letting it integrate with remote or local resources. +### `redpanda_agents.yaml` -A **RuntimeServer** exposes the Agent over gRPC, so outside applications can interact with it. -The project includes *hooks* to track key events (like before or after a tool is called) and a *ToolResponse* structure for sending back text or images neatly. +The main entry point for the agent pipeline. The file looks like the following: -The **RPKMCPEndpoint** even simplifies testing by running a local command to simulate a server inside your development folder. +```yaml +# Each agent is an entry under `agents` - there can be multiple for multi-agent flows. +agents: + # The name of the agent determines what python file is executed for the agent. + : + # Any Redpanda Connect input is valid here + # See them all at: https://docs.redpanda.com/redpanda-connect/components/inputs/about/ + input: + + # Any tool labels defined in the `mcp` directory, see notes below for more. + tools: + - + # Any Redpanda Connect output is valid here + # See them all at https://docs.redpanda.com/redpanda-connect/components/outputs/about/ + output: + + # See options here: https://docs.redpanda.com/redpanda-connect/components/tracer/about/ + tracer: + +``` +### `agents/*.py` +Each agent recieves input from `input` and sends it's output to `output`. You can create an agent +by importing from `redpanda.agents`. Creating an `Agent` looks like: -## Chapters +```python +my_agent = Agent( + name="my_first_agent", + model="openai/gpt-4o", + instructions="These are your instructions - good luck!", +) +``` -1. [Agent -](01_agent_.md) -2. [Tool -](02_tool_.md) -3. [ToolResponse -](03_toolresponse_.md) -4. [AgentHooks -](04_agenthooks_.md) -5. [MCPEndpoint -](05_mcpendpoint_.md) -6. [RPKMCPEndpoint -](06_rpkmcpendpoint_.md) -7. [mcp_client -](07_mcp_client_.md) -8. [RuntimeServer -](08_runtimeserver_.md) +Once you've created the agent, you pass it off to the runtime to handle messages in the pipeline like so: +```python +asyncio.run(redpanda.runtime.serve(my_agent)) +``` ---- +### `mcp/resources/processors/*.yaml` + +In order to give tools to your agent, you need to first define them as yaml files with the following structure: + +```yaml +label: '