Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,14 +93,17 @@ The web docs live in `docs/` and are built with [MkDocs Material](https://squidf
**Structure:**
- `docs/index.html` — custom HTML landing page (not generated by MkDocs, hand-maintained)
- `docs/getting-started.md` — system requirements, build instructions, CMake options
- `docs/concepts.md` — terminology glossary (kernel bypass, GPUDirect, packet/burst/segment, flow/queue, memory region, zero-copy ownership, RX reorder). Meant to be opened in parallel with the rest of the docs.
- `docs/concepts.md` — terminology glossary (stream types and protocols, GPUDirect, packet/burst/segment, flow/queue, memory region, zero-copy ownership, RX reorder). Meant to be opened in parallel with the rest of the docs.
- `docs/api-reference/index.md` — API guide (6-step application lifecycle, configuration-first model)
- `docs/api-reference/configuration.md`, `docs/api-reference/cpp.md`, `docs/api-reference/python.md` — YAML schema, C++ API, and Python bindings docs
- `docs/tutorials/` — tutorial walkthroughs (system config, benchmarking, config files)
- `docs/tutorials/` — tutorial walkthroughs (system config, config-file walkthrough)
- `docs/tutorials/benchmarking_examples.md` — surfaced as a top-level "Benchmarks" nav entry in `mkdocs.yml` and `docs/index.html`; file kept at its original path for inbound-link stability
- `docs/stylesheets/extra.css` — custom theme overrides

**User-facing vocabulary:** docs and the YAML schema use `stream_type` (`raw`, `socket`, future `pcie`) and `protocol` (`udp`, `tcp`, `roce`). The word "backend" is internal-only — accurate for `src/managers/<name>/`, the `Manager` ABC, CMake `DAQIRI_MGR`, and API-reference function blurbs, but should not appear in tutorials, the landing page, or concept pages. The mapping: `stream_type: "raw"` is implemented by the `dpdk` manager; `stream_type: "socket"` with `protocol: "udp"` / `"tcp"` is implemented by the `socket` manager; `stream_type: "socket"` with `protocol: "roce"` is implemented by the `rdma` manager.

**Keeping docs in sync with code:** before committing changes, scan for the recurring drift hotspots:
- **Backend list** (`src/managers/*/`) — README Backends table, `docs/getting-started.md`, `docs/concepts.md` (Kernel Bypass section + Backend Maturity admonition), `docs/api-reference/configuration.md`
- **Stream-type list** (`src/managers/*/`) — README Backends table, `docs/getting-started.md`, `docs/concepts.md` (Stream Types section + Support and testing admonition), `docs/api-reference/configuration.md`
- **CMake options / `DAQIRI_MGR` default** (`src/CMakeLists.txt:137`) — README Quick Start, `docs/getting-started.md`, this file's Build & run section
- **Benchmark binary or YAML names** (`examples/`) — the benchmark table above, `docs/tutorials/benchmarking_examples.md`, and the "Choosing an example config" decision tree in `docs/tutorials/configuration-walkthrough.md` (every YAML must have a leaf; CI's `scripts/check_doc_refs.py` enforces coverage)
- **Public API include** (`#include <daqiri/daqiri.h>`; source files under `include/daqiri/`) — `docs/api-reference/index.md`, `docs/api-reference/cpp.md`, `docs/api-reference/python.md`; if the change adds or renames a user-facing concept, also `docs/concepts.md`
Expand Down
32 changes: 20 additions & 12 deletions docs/api-reference/configuration.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
hide:
- navigation
---

# Configuration YAML Reference

DAQIRI is configured through a YAML file or a `NetworkConfig` struct built in code.
Expand Down Expand Up @@ -68,9 +73,10 @@ and their `kind` determines the receive mode (CPU-only, header-data split, or ba
- values: `local`, `rdma_read`, `rdma_write`
- **`num_bufs`**: Number of buffers in this region. Higher values give more processing
headroom but consume more memory (GPU BAR1 for `device`). Too low risks dropped packets
on RX or higher latency on TX. Rule of thumb: 3x-5x `batch_size`. For the DPDK
backend, `num_bufs` below 1.5x the NIC ring size deadlocks the worker; `daqiri_init`
auto-bumps such MRs to 3x the ring (24576 with the default 8192) and logs a `WARN`.
on RX or higher latency on TX. Rule of thumb: 3x-5x `batch_size`. For Raw Ethernet
(`stream_type: "raw"`), `num_bufs` below 1.5x the NIC ring size deadlocks the worker;
`daqiri_init` auto-bumps such MRs to 3x the ring (24576 with the default 8192) and
logs a `WARN`.
- type: `integer`
- **`buf_size`**: Size of each buffer in bytes. Should match the expected packet size, or
the segment size when using header-data split.
Expand Down Expand Up @@ -104,8 +110,9 @@ memory_regions:

- **`name`**: Interface name. Used to look up port IDs at runtime via `get_port_id()`.
- type: `string`
- **`address`**: PCIe BDF address (from `lspci`) or Linux interface name for DPDK, or IP
address for RDMA.
- **`address`**: PCIe BDF address (from `lspci`) or Linux interface name for Raw Ethernet
(`stream_type: "raw"`), or IP address for RoCE (`stream_type: "socket"`,
`protocol: "roce"`).
- type: `string`

### RDMA Configuration
Expand Down Expand Up @@ -201,7 +208,8 @@ Unmatched packets are dropped. When `false`, unmatched packets go to a default q

### Hardware Timestamps

`rx.hardware_timestamps:` — Enable per-packet hardware RX timestamps for the DPDK backend.
`rx.hardware_timestamps:` — Enable per-packet hardware RX timestamps for Raw Ethernet
(`stream_type: "raw"`).
When enabled, DAQIRI requires `RTE_ETH_RX_OFFLOAD_TIMESTAMP` support from the NIC/PMD and
initialization fails if DAQIRI cannot provide nanosecond timestamps for the selected PMD.
Timestamps are returned by `get_packet_rx_timestamp()` in nanoseconds in the NIC timestamp
Expand All @@ -210,12 +218,12 @@ clock domain, not wall-clock time.
- type: `boolean`
- default: `false`

### RX Reorder Configs (DPDK v1)
### RX Reorder Configs

`rx.reorder_configs:` — Optional automatic packet reordering/aggregation plans. In v1 this is
implemented for the DPDK backend only. GPU reorder requires CUDA-addressable packet buffers
(`device` or `host_pinned` memory regions). CPU reorder requires CPU-addressable packet buffers
(`host`, `host_pinned`, or `huge` memory regions).
`rx.reorder_configs:` — Optional automatic packet reordering/aggregation plans. Implemented
for Raw Ethernet (`stream_type: "raw"`) only in v1. GPU reorder requires CUDA-addressable
packet buffers (`device` or `host_pinned` memory regions). CPU reorder requires CPU-addressable
packet buffers (`host`, `host_pinned`, or `huge` memory regions).

v1 source-memory requirement:
- Reorder queues must use exactly one RX source memory region.
Expand Down Expand Up @@ -316,7 +324,7 @@ enabled, use `set_packet_tx_time()` to schedule packets. Requires ConnectX-7 or
- type: `boolean`
- default: `false`

## Complete Example (DPDK, Header-Data Split)
## Complete Example (Raw Ethernet, Header-Data Split)

```yaml
%YAML 1.2
Expand Down
21 changes: 13 additions & 8 deletions docs/api-reference/cpp.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
hide:
- navigation
---

# C++ API Usage

This guide covers C++ initialization, RX/TX workflows, buffer lifecycle calls, file
Expand Down Expand Up @@ -28,9 +33,9 @@ auto status = daqiri::daqiri_init(config);
After `daqiri_init()` returns `Status::SUCCESS`, all memory regions are allocated, NIC
queues are configured, and worker threads are running.

If GPU RX `reorder_configs` are configured for the DPDK backend, set one CUDA stream
per GPU reorder plan before pulling reordered bursts. CPU reorder configs do not use a
CUDA stream. See the [Configuration YAML Reference](configuration.md#rx-reorder-configs-dpdk-v1)
If GPU RX `reorder_configs` are configured for Raw Ethernet (`stream_type: "raw"`), set
one CUDA stream per GPU reorder plan before pulling reordered bursts. CPU reorder configs do not use a
CUDA stream. See the [Configuration YAML Reference](configuration.md#rx-reorder-configs)
for reorder configuration constraints.

```cpp
Expand Down Expand Up @@ -81,11 +86,11 @@ for (int i = 0; i < daqiri::get_num_packets(burst); i++) {
}
```

RX hardware timestamps are available only when the DPDK backend is configured with
`rx.hardware_timestamps: true` and the NIC supports `RTE_ETH_RX_OFFLOAD_TIMESTAMP`.
DAQIRI converts the NIC timestamp counter to nanoseconds internally using DPDK's
matching device clock when available, or the PMD's nanosecond timestamp format when
the driver already supplies nanoseconds. DAQIRI does not expose NIC clock reads or
RX hardware timestamps are available only when Raw Ethernet (`stream_type: "raw"`) is
configured with `rx.hardware_timestamps: true` and the NIC supports
`RTE_ETH_RX_OFFLOAD_TIMESTAMP`. DAQIRI converts the NIC timestamp counter to nanoseconds
internally using the matching device clock when available, or the PMD's nanosecond
timestamp format when the driver already supplies nanoseconds. DAQIRI does not expose NIC clock reads or
convert timestamps to wall-clock time. For reordered aggregate bursts,
`get_packet_rx_timestamp(burst, 0, &ts)` returns the timestamp of the first source
packet accepted into the aggregate.
Expand Down
16 changes: 11 additions & 5 deletions docs/api-reference/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
hide:
- navigation
---

# API Guide

DAQIRI is a library that moves bursts of packets between NICs and CPU or
Expand All @@ -15,16 +20,17 @@ For the terminology and conceptual background it relies on

A DAQIRI application starts from a YAML configuration file (or an
equivalent `NetworkConfig` struct built in code). The configuration
defines the active backend, NIC interfaces, RX and TX queues, memory
regions, flow steering rules, flow isolation, header-data split, and
optional reorder plans. After initialization, the language API operates
on those configured ports, queues, buffers, and flows.
defines the active stream type and protocol, NIC interfaces, RX and TX
queues, memory regions, flow steering rules, flow isolation,
header-data split, and optional reorder plans. After initialization,
the language API operates on those configured ports, queues, buffers,
and flows.

The language APIs do **not** discover queues, memory, or flow steering
rules on their own. They are runtime handles over the topology declared
in the configuration (YAML file or `NetworkConfig` struct). The
configuration is the source of truth for queue IDs, memory placement,
protocol/backend selection, and flow routing.
stream-type / protocol selection, and flow routing.

The configuration schema lives in the
[Configuration YAML Reference](configuration.md). For an annotated
Expand Down
2 changes: 1 addition & 1 deletion docs/api-reference/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ if status == daqiri.Status.SUCCESS:
If GPU RX `reorder_configs` are configured for the DPDK backend, set one CUDA
stream per GPU reorder plan before pulling reordered bursts. Pass the CUDA
stream as an integer address; pass `0` to use the default stream. See the
[Configuration YAML Reference](configuration.md#rx-reorder-configs-dpdk-v1)
[Configuration YAML Reference](configuration.md#rx-reorder-configs)
for reorder configuration constraints.

```python
Expand Down
Loading
Loading