Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions skills/netra-best-practices/references/python/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,25 +96,32 @@ dataset = Dataset(items=fetched.items)
The `task` is a callable that receives a single dataset item's `input` and returns the output to be evaluated. It can be sync or async.

```python
from netra import Netra
from openai import OpenAI

client = OpenAI()

def my_task(input):
Netra.set_root_input(input)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": input}],
)
return response.choices[0].message.content
output = response.choices[0].message.content
Netra.set_root_output(output)
return output
```

```python
async def my_async_task(input):
Netra.set_root_input(input)
response = await async_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": input}],
)
return response.choices[0].message.content
output = response.choices[0].message.content
Netra.set_root_output(output)
return output
```

Each task invocation is automatically wrapped in a `TestRun.{name}` span, so the call is traced end-to-end without extra instrumentation.
Expand Down Expand Up @@ -267,11 +274,14 @@ Netra.init(
client = OpenAI()

def qa_task(input):
Netra.set_root_input(input)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": input}],
)
return response.choices[0].message.content
output = response.choices[0].message.content
Netra.set_root_output(output)
return output

class ContainsExpectedEvaluator(BaseEvaluator):
def evaluate(self, context: EvaluatorContext) -> EvaluatorOutput:
Expand Down Expand Up @@ -315,7 +325,8 @@ Netra.shutdown()
2. `NETRA_API_KEY` and `NETRA_OTLP_ENDPOINT` environment variables are set.
3. Every `DatasetItem` has a non-empty `input`.
4. The `task` function accepts a single argument and returns the output.
5. Custom evaluator `evaluate()` returns `EvaluatorOutput` with `evaluator_name` matching `config.name`.
5. The `task` function calls `Netra.set_root_input(input)` at the start and `Netra.set_root_output(output)` before returning.
6. Custom evaluator `evaluate()` returns `EvaluatorOutput` with `evaluator_name` matching `config.name`.
6. `ScoreType` matches the type of `result` (bool for `BOOLEAN`, number for `NUMERICAL`, string for `CATEGORICAL`).
7. `Netra.shutdown()` is called on graceful termination.

Expand Down
67 changes: 65 additions & 2 deletions skills/netra-best-practices/references/python/instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,13 +92,17 @@ from netra.decorators import workflow, agent, task, span

@workflow(name="order-fulfillment")
def fulfill_order(order: dict):
Netra.set_root_input(order)

current = Netra.get_current_span()
if current:
current.set_attribute("order.id", order.get("id"))
current.add_event("order.received", {"item_count": len(order.get("items", []))})

result = OrderAgent().orchestrate(order)

Netra.set_root_output(result)

if current:
current.add_event("order.completed")
return result
Expand Down Expand Up @@ -137,6 +141,8 @@ Python uses context managers — the span is automatically ended when the `with`
from netra import Netra, SpanType, UsageModel

def chat_with_ai(user_message: str) -> str:
Netra.set_root_input(user_message)

with Netra.start_span(
"chat-completion",
as_type=SpanType.GENERATION,
Expand All @@ -160,6 +166,7 @@ def chat_with_ai(user_message: str) -> str:
)
])
span.set_success()
Netra.set_root_output(response_text)
return response_text
except Exception as exc:
span.set_error(str(exc))
Expand All @@ -177,6 +184,61 @@ Netra.set_tenant_id("tenant-xyz")
Netra.set_custom_attributes("feature", "chat-v2")
```

## Root Input & Output

Use `set_root_input` and `set_root_output` to record the top-level input and output on the **root span** of the current trace. These values appear as the `input` and `output` attributes on the trace in the Netra dashboard, making it easy to see what went in and what came out at a glance.

**Always set root input at the entry point of your workflow and root output before returning the final result.** This should be the default practice for every instrumented application.

```python
from netra import Netra

Netra.set_root_input({"query": "What is the weather today?"})

# ... run your pipeline / agent / chain ...

Netra.set_root_output("The weather today is sunny with a high of 72°F.")
```

Both methods accept any serializable value (strings, dicts, lists, etc.). The SDK serializes the value to a string internally.

### Streaming output

When the final output is a stream (e.g., an LLM streaming response), use `set_root_output_stream` instead. It wraps the stream transparently and records the accumulated output on the root span when iteration completes.

```python
stream = client.chat.completions.create(model="gpt-4o", messages=messages, stream=True)
stream = Netra.set_root_output_stream(stream)

for chunk in stream:
print(chunk) # output is auto-committed to root span when iteration ends
```

`set_root_output_stream` supports both sync and async iterables. If no active trace context exists or the value is not iterable, it returns the value unchanged — so it is always safe to reassign.

### Generator / SSE streaming (FastAPI, Starlette, etc.)

`set_root_output_stream` works by wrapping an iterable that yields raw LLM chunks. It does **not** work when the generator yields already-formatted SSE strings (e.g., `"data: ...\n\n"`), because the accumulated text would include the SSE framing.

In this pattern — common in FastAPI `StreamingResponse` generators — **manually accumulate** the content chunks and call `set_root_output` after iteration:

```python
@workflow(name="chat-stream")
def generate():
Netra.set_root_input(user_message)
collected: list[str] = []

for chunk in agent.run(message, stream=True):
if chunk and chunk.content:
collected.append(chunk.content)
yield f"data: {chunk.content}\n\n"

Netra.set_root_output("".join(collected))
yield "data: [DONE]\n\n"
```

**Rule of thumb:** Use `set_root_output_stream` when you can wrap the raw LLM iterable before consuming it. Use manual accumulation + `set_root_output` when the generator transforms or formats chunks before yielding (SSE, WebSocket frames, custom protocols).

## Recommended Rollout Strategy

1. Start with auto-instrumentation for broad, immediate coverage.
Expand All @@ -188,8 +250,9 @@ Netra.set_custom_attributes("feature", "chat-v2")
1. `Netra.init()` is called once at startup.
2. Initialization happens **before** instrumented library usage.
3. High-level operations appear as workflow spans.
4. `Netra.shutdown()` is called on graceful app termination.
5. All `InstrumentSet` values and `SpanType` values used are verified against the official Netra documentation listed in the **References** section below.
4. `Netra.set_root_input()` is called at the entry point with the user's input, and `Netra.set_root_output()` (or `Netra.set_root_output_stream()` for streaming) is called with the final result before returning. For SSE/generator streaming, chunks are accumulated and `set_root_output` is called after iteration completes.
5. `Netra.shutdown()` is called on graceful app termination.
6. All `InstrumentSet` values and `SpanType` values used are verified against the official Netra documentation listed in the **References** section below.

## References

Expand Down
11 changes: 10 additions & 1 deletion skills/netra-best-practices/references/python/simulation.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,14 @@ Netra.init(
Subclass `BaseTask` and implement `run()`. The method can be sync or async.

```python
from netra import Netra
from netra.simulation import BaseTask, TaskResult

class MyAgentTask(BaseTask):
def run(self, message, session_id=None, files=None):
Netra.set_root_input(message)
response = my_agent.chat(message, session_id=session_id)
Netra.set_root_output(response.text)
return TaskResult(
message=response.text,
session_id=response.session_id or session_id or "default",
Expand Down Expand Up @@ -77,7 +80,9 @@ def run(
```python
class MyAsyncTask(BaseTask):
async def run(self, message, session_id=None, files=None):
Netra.set_root_input(message)
response = await my_async_agent.chat(message, session_id=session_id)
Netra.set_root_output(response.text)
return TaskResult(
message=response.text,
session_id=response.session_id or session_id or "default",
Expand All @@ -87,14 +92,17 @@ class MyAsyncTask(BaseTask):
### Task with file handling

```python
from netra import Netra
from netra.simulation import BaseTask, TaskResult, ProcessedFile

class FileTask(BaseTask):
def run(self, message, session_id=None, files=None):
Netra.set_root_input(message)
if files:
for f in files:
print(f.file_name, f.content_type, len(f.data))
response = my_agent.chat(message, session_id=session_id, files=files)
Netra.set_root_output(response.text)
return TaskResult(
message=response.text,
session_id=response.session_id or session_id or "default",
Expand Down Expand Up @@ -131,7 +139,8 @@ Datasets for simulation are created and managed via the Netra dashboard.
1. `Netra.init()` is called before accessing `Netra.simulation`.
2. `NETRA_API_KEY` and `NETRA_OTLP_ENDPOINT` are set.
3. Task's `run()` always returns a `TaskResult` with both `message` and `session_id`.
4. `Netra.shutdown()` is called on graceful termination.
4. Task's `run()` calls `Netra.set_root_input(message)` at the start and `Netra.set_root_output(response)` before returning.
5. `Netra.shutdown()` is called on graceful termination.

## References

Expand Down
14 changes: 11 additions & 3 deletions skills/netra-best-practices/references/typescript/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,16 +90,20 @@ const dataset = { items: fetched!.items };
A function receiving a single dataset item's `input` and returning the output. Can be sync or async.

```typescript
import { Netra } from "netra-sdk";
import OpenAI from "openai";

const client = new OpenAI();

async function myTask(input: any): Promise<string> {
Netra.setRootInput(input);
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: input }],
});
return response.choices[0].message.content!;
const output = response.choices[0].message.content!;
Netra.setRootOutput(output);
return output;
}
```

Expand Down Expand Up @@ -187,11 +191,14 @@ await Netra.init({
const client = new OpenAI();

async function qaTask(input: any): Promise<string> {
Netra.setRootInput(input);
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: input }],
});
return response.choices[0].message.content!;
const output = response.choices[0].message.content!;
Netra.setRootOutput(output);
return output;
}

const containsExpected = {
Expand Down Expand Up @@ -239,7 +246,8 @@ await Netra.shutdown();
2. `NETRA_API_KEY` and `NETRA_OTLP_ENDPOINT` environment variables are set.
3. Every dataset item has a non-empty `input`.
4. The task function accepts a single argument and returns the output.
5. Evaluator `.evaluate()` returns an object with `evaluatorName` matching `config.name`.
5. The task function calls `Netra.setRootInput(input)` at the start and `Netra.setRootOutput(output)` before returning.
6. Evaluator `.evaluate()` returns an object with `evaluatorName` matching `config.name`.
6. `await Netra.shutdown()` is called on graceful termination.

## References
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,13 +80,17 @@ TypeScript decorators require `"experimentalDecorators": true` in `tsconfig.json
- `@span`: generic/custom span type.

```typescript
import { SpanType } from "netra-sdk";
import { Netra, SpanType } from "netra-sdk";
import { workflow, agent, task, span } from "netra-sdk/decorators";

@workflow({ name: "order-fulfillment" })
class OrderWorkflow {
async run(order: { id: string; items: unknown[] }) {
Netra.setRootInput(order);

const result = await new OrderAgent().orchestrate(order);

Netra.setRootOutput(result);
return result;
}
}
Expand Down Expand Up @@ -123,6 +127,8 @@ import { Netra, SpanType } from "netra-sdk";
import type { UsageModel } from "netra-sdk";

async function chatWithAI(userMessage: string): Promise<string> {
Netra.setRootInput(userMessage);

const span = Netra.startSpan("chat-completion", {
asType: SpanType.GENERATION,
moduleName: "chat",
Expand All @@ -146,6 +152,7 @@ async function chatWithAI(userMessage: string): Promise<string> {
},
]);
span.setSuccess();
Netra.setRootOutput(responseText);
return responseText;
} catch (error: any) {
span.setError(error?.message || "unknown error");
Expand Down Expand Up @@ -182,6 +189,63 @@ Netra.setTenantId("tenant-xyz");
Netra.setCustomAttributes("feature", "chat-v2");
```

## Root Input & Output

Use `setRootInput` and `setRootOutput` to record the top-level input and output on the **root span** of the current trace. These values appear as the `input` and `output` attributes on the trace in the Netra dashboard, making it easy to see what went in and what came out at a glance.

**Always set root input at the entry point of your workflow and root output before returning the final result.** This should be the default practice for every instrumented application.

```typescript
import { Netra } from "netra-sdk";

Netra.setRootInput({ query: "What is the weather today?" });

// ... run your pipeline / agent / chain ...

Netra.setRootOutput("The weather today is sunny with a high of 72°F.");
```

Both methods accept any serializable value (strings, objects, arrays, etc.). The SDK serializes the value to a string internally.

### Generator / SSE streaming (Express, Next.js, Hono, etc.)

In web frameworks, streaming responses are often sent as formatted SSE strings (`"data: ...\n\n"`) or via `res.write()`. Because the output is transformed before being sent, you cannot simply wrap the raw stream — you need to **manually accumulate** the content chunks and call `setRootOutput` after iteration completes.

```typescript
import { Netra } from "netra-sdk";

app.post("/api/chat/stream", async (req, res) => {
const span = Netra.startSpan("chat-stream");
Netra.setSessionId(req.body.sessionId);
Netra.setRootInput(req.body.message);

const collected: string[] = [];

try {
const stream = await agent.run(req.body.message, { stream: true });

for await (const chunk of stream) {
if (chunk.content) {
collected.push(chunk.content);
res.write(`data: ${chunk.content}\n\n`);
}
}

Netra.setRootOutput(collected.join(""));
res.write("data: [DONE]\n\n");
res.end();
span.setSuccess();
} catch (error: any) {
span.setError(error?.message || "unknown error");
res.end();
} finally {
span.end();
}
});
```

**Rule of thumb:** Use manual accumulation + `setRootOutput` when the handler transforms or formats chunks before sending (SSE, WebSocket frames, custom protocols). This ensures the root span captures the clean response content without framing artifacts.

## Root Span Wrapper

When `enableRootSpan` is enabled, use `runWithRootSpan()` to execute code within the root span context. Any spans created inside the callback are automatically **parented to the root span**.
Expand All @@ -207,8 +271,9 @@ Netra.runWithRootSpan(() => {
2. Initialization happens **before** instrumented library usage.
3. `experimentalDecorators: true` is set in `tsconfig.json` if using decorators.
4. Manual spans always call `span.end()` in a `finally` block.
5. `await Netra.shutdown()` is called on graceful app termination.
6. All `NetraInstruments` values and `SpanType` values used are verified against the official Netra documentation listed in the **References** section below.
5. `Netra.setRootInput()` is called at the entry point with the user's input, and `Netra.setRootOutput()` is called with the final result before returning. For SSE/generator streaming, chunks are accumulated and `setRootOutput` is called after iteration completes.
6. `await Netra.shutdown()` is called on graceful app termination.
7. All `NetraInstruments` values and `SpanType` values used are verified against the official Netra documentation listed in the **References** section below.

## References

Expand Down
Loading