redpanda-data · rockwotj · Apr 29, 2025 · Apr 25, 2025 · Apr 29, 2025 · Apr 29, 2025
diff --git a/Taskfile.yaml b/Taskfile.yaml
@@ -80,5 +80,5 @@ tasks:
           --mypy_out={{.OUT_DIR}} \
           --grpc_python_out={{.OUT_DIR}} \
           --mypy_grpc_out={{.OUT_DIR}} \
-          -I proto
+          -I proto \
           proto/redpanda/runtime/proto/runtime.proto
diff --git a/docs/06_rpkmcpendpoint_.md b/docs/06_rpkmcpendpoint_.md
diff --git a/docs/index.md b/docs/index.md
@@ -1,52 +1,103 @@
-# Welcome to MkDocs
+# Redpanda Agents
 
-For full documentation visit [mkdocs.org](https://www.mkdocs.org).
+The [Redpanda Agent SDK](https://github.com/redpanda-data/agent) allows you to build agentic [Redpanda Connect](https://www.redpanda.com/connect) pipelines.
 
-## Commands
+You can use [`rpk`](https://docs.redpanda.com/current/get-started/intro-to-rpk/) to generate the initial boilerplate for an agentic pipeline.
 
-* `mkdocs new [dir-name]` - Create a new project.
-* `mkdocs serve` - Start the live-reloading docs server.
-* `mkdocs build` - Build the documentation site.
-* `mkdocs -h` - Print help message and exit.
+```bash
+$ rpk connect agent init my_first_agent
 
-## Project layout
+$ ls --tree ./my_first_agent
+my_first_agent
+├── agents
+│   └── weather.py
+├── mcp
+│   └── resources
+│       └── processors
+│           └── check_weather_tool.yaml
+├── pyproject.toml
+├── README.md
+├── redpanda_agents.yaml
+└── uv.lock
+```
 
-    mkdocs.yml    # The configuration file.
-    docs/
-        index.md  # The documentation homepage.
-        ...       # Other markdown pages, images and other files.
+## Project Structure
 
+The project structure is as follows:
 
-**This project** provides a flexible system where an *Agent* can answer questions and use specialized *Tools* to tackle tasks. 
-The Agent can also connect to external services through different **MCPEndpoints**, letting it integrate with remote or local resources. 
+### `redpanda_agents.yaml`
 
-A **RuntimeServer** exposes the Agent over gRPC, so outside applications can interact with it. 
-The project includes *hooks* to track key events (like before or after a tool is called) and a *ToolResponse* structure for sending back text or images neatly. 
+The main entry point for the agent pipeline. The file looks like the following:
 
-The **RPKMCPEndpoint** even simplifies testing by running a local command to simulate a server inside your development folder.
+```yaml
+# Each agent is an entry under `agents` - there can be multiple for multi-agent flows.
+agents:
+  # The name of the agent determines what python file is executed for the agent.
+  <agent_name>:
+    # Any Redpanda Connect input is valid here
+    # See them all at: https://docs.redpanda.com/redpanda-connect/components/inputs/about/
+    input:
+      <input>
+    # Any tool labels defined in the `mcp` directory, see notes below for more.
+    tools:
+      - <tool label>
+    # Any Redpanda Connect output is valid here
+    # See them all at https://docs.redpanda.com/redpanda-connect/components/outputs/about/
+    output:
+      <output>
+    # See options here: https://docs.redpanda.com/redpanda-connect/components/tracer/about/
+    tracer:
+        <tracer>
+```
 
+### `agents/*.py`
 
+Each agent recieves input from `input` and sends it's output to `output`. You can create an agent
+by importing from `redpanda.agents`. Creating an `Agent` looks like:
 
-## Chapters
+```python
+my_agent = Agent(
+    name="my_first_agent",
+    model="openai/gpt-4o",
+    instructions="These are your instructions - good luck!",
+)
+```
 
-1. [Agent
-](01_agent_.md)
-2. [Tool
-](02_tool_.md)
-3. [ToolResponse
-](03_toolresponse_.md)
-4. [AgentHooks
-](04_agenthooks_.md)
-5. [MCPEndpoint
-](05_mcpendpoint_.md)
-6. [RPKMCPEndpoint
-](06_rpkmcpendpoint_.md)
-7. [mcp_client
-](07_mcp_client_.md)
-8. [RuntimeServer
-](08_runtimeserver_.md)
+Once you've created the agent, you pass it off to the runtime to handle messages in the pipeline like so:
 
+```python
+asyncio.run(redpanda.runtime.serve(my_agent))
+```
 
----
+### `mcp/resources/processors/*.yaml`
+
+In order to give tools to your agent, you need to first define them as yaml files with the following structure:
+
+```yaml
+label: '<label>'
+processors:
+  - <processors>
+meta:
+  mcp:
+    enabled: true
+    description: '<description>'
+```
+
+The `<label>` is what must be provided under the list of `tools` in the agent YAML file, while the
+processors can be any [Redpanda Connect processor](https://docs.redpanda.com/redpanda-connect/components/processors/about/).
+
+<!--
+TODO
+
+### `mcp/resources/caches/*.yaml`
+
+### `mcp/o11y/tracer.yaml`
+
+### `mcp/o11y/metrics.yaml`
+
+-->
+
+## Running
+
+To run our agent, we can do that with `rpk connect agent run my_first_agent`
 
-Generated by [AI Codebase Knowledge Builder](https://github.com/The-Pocket/Tutorial-Codebase-Knowledge)
diff --git a/docs/01_agent_.md → docs/tutorial/01_agent_.md b/docs/01_agent_.md → docs/tutorial/01_agent_.md
diff --git a/docs/02_tool_.md → docs/tutorial/02_tool_.md b/docs/02_tool_.md → docs/tutorial/02_tool_.md
diff --git a/docs/03_toolresponse_.md → docs/tutorial/03_toolresponse_.md b/docs/03_toolresponse_.md → docs/tutorial/03_toolresponse_.md
diff --git a/docs/04_agenthooks_.md → docs/tutorial/04_agenthooks_.md b/docs/04_agenthooks_.md → docs/tutorial/04_agenthooks_.md
diff --git a/docs/05_mcpendpoint_.md → docs/tutorial/05_mcpendpoint_.md b/docs/05_mcpendpoint_.md → docs/tutorial/05_mcpendpoint_.md
@@ -12,7 +12,6 @@ For example:
 • You can connect via standard input/output (like launching a program and talking with it).  
 • You can connect via Server-Sent Events (SSE).  
 • You can connect via WebSocket.  
-• Or you can do something more specialized like RPKMCPEndpoint (we’ll see that in [Chapter 6: RPKMCPEndpoint](06_rpkmcpendpoint_.md)).
 
 Think of MCPEndpoint as the base blueprint. Different subclasses (e.g., StdioMCPEndpoint, SSEMCPEndpoint, WebsocketMCPEndpoint) each handle one style of communication.
 
@@ -149,4 +148,4 @@ Explanation:
 • Different subclasses handle different connection methods: Stdio, SSE, WebSocket, etc.  
 • You gain the flexibility of calling local or remote Tools in a uniform way.
 
-Now that you’ve seen the basic Endpoint, let’s explore a special variation that launches a local process for you: the [RPKMCPEndpoint](06_rpkmcpendpoint_.md). This can be handy when you want to bundle the server’s logic right with your Agent without hosting externally. Get ready for the next step!
+Next time, we’ll look more closely at [mcp_client](07_mcp_client_.md), which is the library helper that actually does most of the handshake and messaging between your Python code and the server. You’ll see how it all fits together behind the scenes!
diff --git a/docs/07_mcp_client_.md → docs/tutorial/07_mcp_client_.md b/docs/07_mcp_client_.md → docs/tutorial/07_mcp_client_.md
@@ -1,6 +1,6 @@
-# Chapter 7: mcp_client
+# Chapter 6: mcp_client
 
-In the previous chapter, [RPKMCPEndpoint](06_rpkmcpendpoint_.md), we saw how an Agent can spin up a local MCP process using the “rpk connect mcp-server” command. Now, let's explore the core function that actually sets up the conversation with any MCP server—whether it’s local, remote, using Standard I/O, SSE, or WebSocket:
+In the [previous chapter: MCPEndpoint](05_mcpendpoint_.md), you learned how an Agent can communicate with external servers or local processes using different connection methods. Now, let's explore the core function that actually sets up the conversation with any MCP server—whether it’s local, remote, using Standard I/O, SSE, or WebSocket:
 
 • The function name: `mcp_client`  
 • Purpose: Creates a temporary communication channel to an MCP server.  

diff --git a/docs/08_runtimeserver_.md → docs/tutorial/08_runtimeserver_.md b/docs/08_runtimeserver_.md → docs/tutorial/08_runtimeserver_.md
@@ -1,6 +1,6 @@
-# Chapter 8: RuntimeServer
+# Chapter 7: RuntimeServer
 
-In [Chapter 7: mcp_client](07_mcp_client_.md), we learned how to tap into an existing server that hosts tools or services. But what if you want others to tap into your Agent in the same way—via a simple, well-defined interface? That’s exactly where “RuntimeServer” shines.
+In [Chapter 6: mcp_client](07_mcp_client_.md), we learned how to tap into an existing server that hosts tools or services. But what if you want others to tap into your Agent in the same way—via a simple, well-defined interface? That’s exactly where “RuntimeServer” shines.
 
 Imagine you have an Agent with all sorts of logic or tooling behind it, and you want to let an external system send requests. The external system might not speak Python, or it might be on a totally different tech stack. By using RuntimeServer, you can expose your Agent as a gRPC service—letting outside callers invoke it in real time, just like a translator who relays messages back and forth.
 

diff --git a/docs/tutorial/index.md b/docs/tutorial/index.md
@@ -0,0 +1,28 @@
+# Overview
+
+**Redpanda Agents** provides a flexible system where an *Agent* can answer questions and use specialized *Tools* to tackle tasks. 
+The Agent can also connect to external services through different **MCPEndpoints**, letting it integrate with remote or local resources. 
+
+A **RuntimeServer** exposes the Agent over gRPC, so Redpanda Connect can interact with it. 
+The project includes *hooks* to track key events (like before or after a tool is called) and a *ToolResponse* structure for sending back text or images neatly. 
+
+## Chapters
+
+1. [Agent
+](01_agent_.md)
+2. [Tool
+](02_tool_.md)
+3. [ToolResponse
+](03_toolresponse_.md)
+4. [AgentHooks
+](04_agenthooks_.md)
+5. [MCPEndpoint
+](05_mcpendpoint_.md)
+6. [mcp_client
+](07_mcp_client_.md)
+7. [RuntimeServer
+](08_runtimeserver_.md)
+
+---
+
+Generated by [AI Codebase Knowledge Builder](https://github.com/The-Pocket/Tutorial-Codebase-Knowledge)
diff --git a/examples/observability/README.md b/examples/observability/README.md
@@ -0,0 +1,27 @@
+## Observability demo
+
+This demo shows how to view open telemetry traces within your pipelines.
+
+First start a local [jaeger] instance like this:
+
+```
+docker run --rm --name jaeger \
+  -p 16686:16686 \
+  -p 4317:4317 \
+  -p 4318:4318 \
+  -p 5778:5778 \
+  -p 9411:9411 \
+  jaegertracing/jaeger:2.5.0
+```
+
+Then run the agent like so:
+
+```
+rpk connect agent run .
+```
+
+After asking a question like `What is the weather in Chicago?` and pressing enter,
+you should be able to see traces shortly in the jaeger UI at localhost:16686
+
+[jaeger]: https://www.jaegertracing.io/
+
diff --git a/examples/observability/agents/weather.py b/examples/observability/agents/weather.py
@@ -0,0 +1,14 @@
+import asyncio
+
+import redpanda.runtime
+from redpanda.agents import Agent
+
+my_agent = Agent(
+    name="WeatherAgent",
+    model="openai/gpt-4o",
+    instructions="""
+    You are a helpful AI agent for finding out about the weather.
+    """.strip(),
+)
+
+asyncio.run(redpanda.runtime.serve(my_agent))
diff --git a/examples/observability/mcp/o11y/tracer.yaml b/examples/observability/mcp/o11y/tracer.yaml
@@ -0,0 +1,5 @@
+open_telemetry_collector:
+  service: rp-mcp
+  grpc:
+    - address: localhost:4317
+      secure: false
diff --git a/...ources/processors/check_weather_tool.yaml → ...ources/processors/check_weather_tool.yaml b/...ources/processors/check_weather_tool.yaml → ...ources/processors/check_weather_tool.yaml