Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/extensions/official/ros-ext/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ description: Extension for ReductStore to extract and transform data in ROS form
# ReductROS

The ReductROS [**extension**](../../glossary#extension) provides tools that can be used to extract and transform data stored in ReductStore into formats that are compatible with the Robot Operating System (ROS).
It allows users to query and convert historical data, such as sensor readings and camera images, into JSON or filtered MCAP episodes for analysis, visualisation or integration with other systems.
It allows users to query and convert historical data, such as sensor readings and camera images, into JSON or filtered MCAP episodes for analysis, visualisation or integration with other systems across both ROS1 and ROS2 workflows.

import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote";

Expand All @@ -23,11 +23,11 @@ import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote";

You can use the ReductROS extension to extract and transform data in the following formats:

| Format | Typical input | What you can do |
| --------------------------------- | ------------------------ | ------------------------------------------------------------ |
| **[Raw Messages](./ros-ext/raw)** | Raw ROS records | Extract ROS messages as JSON or export them as MCAP. |
| **[MCAP](./ros-ext/mcap)** | MCAP files | Extract messages as JSON or create filtered MCAP episodes. |
| **[Rosbag](./ros-ext/rosbag)** | ROS 2 `rosbag2` archives | Extract messages as JSON or create filtered rosbag episodes. |
| Format | Typical input | What you can do |
| --------------------------------- | ----------------------- | -------------------------------------------------------------------- |
| **[Raw Messages](./ros-ext/raw)** | Raw ROS1/ROS2 records | Extract ROS1/ROS2 messages as JSON or export them as MCAP. |
| **[MCAP](./ros-ext/mcap)** | MCAP files | Extract ROS1/ROS2 messages as JSON or create filtered MCAP episodes. |
| **[Rosbag](./ros-ext/rosbag)** | ROS2 `rosbag2` archives | Extract messages as JSON or create filtered rosbag episodes. |

import DocCardList from "@theme/DocCardList";

Expand Down
21 changes: 12 additions & 9 deletions docs/extensions/official/ros-ext/mcap.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Parameters and examples for working with MCAP files in ReductROS.

# MCAP

MCAP is a container format for recorded robotics data. In ROS 2, it commonly stores CDR-encoded messages across many topics with metadata like timestamps and schemas. ReductROS can extract and transform ROS 2 messages from MCAP files stored in ReductStore (content type `application/mcap`).
MCAP is a container format for recorded robotics data. It can store ROS1 and ROS2 messages across many topics together with timestamps, schemas, and channel metadata. ReductROS can extract and transform ROS1 and ROS2 messages from MCAP files stored in ReductStore (content type `application/mcap`).

import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote";

Expand Down Expand Up @@ -39,8 +39,7 @@ A user can use the `ext` query parameter to activate the `ros` extension and def
### Data extraction

The `extract` property allows you to specify the ROS topic from which to extract messages and convert them to JSON format.
The extension will read the MCAP file, decode the messages from the specified topic, and return each message as a JSON record with
the timestamp from the message header (when present).
The extension reads the MCAP file, selects the parser from the schema encoding of the requested channel, and returns each decoded message as a JSON record with the timestamp from the message header (when present).

| Parameter | Type | Mandatory | Description |
| ---------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
Expand All @@ -49,6 +48,10 @@ the timestamp from the message header (when present).
| `encode.<field>` | `string` | No | The encoding format for binary fields in the message. Supported values are `base64` and `jpeg`. If not specified, the field will be returned as a JSON list. |
| `as_label` | `object` | No | An object that specifies dynamic computed labels that will be delivered to the client. Can be a basis for output filtering. |

:::info
MCAP extraction supports both ROS1 (`ros1msg`) and ROS2 (`cdr`) schema encodings. Mixed MCAP files are supported, so ROS1 and ROS2 topics can coexist in the same input file and still be extracted correctly topic by topic.
:::

#### Encoding binary data

The `encode` property allows you to specify how to handle binary data in the extracted messages.
Expand Down Expand Up @@ -145,11 +148,11 @@ Record labels: {'@encoding': 'cdr', '@schema': 'std_msgs/String', '@topic': '/te

#### Explanation

- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore.
- The extension extracts ROS messages from an `.mcap` file stored in ReductStore.
- Only messages from the topic `/test` are selected using the `topic` filter in the `ros.extract` configuration.
- The content of each message is CDR-encoded and decoded by the extension.
- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata.
- The decoded message is returned as JSON with the field `data`, matching the `std_msgs/String` schema.
- Each record corresponds to one ROS 2 message and includes:
- Each record corresponds to one ROS message and includes:
- The decoded JSON payload, e.g., `{"data":"hello"}`
- Message metadata as labels, including:
- `topic`: `/test`
Expand Down Expand Up @@ -183,11 +186,11 @@ Image parameters: {'height': 720, 'width': 1280, 'is_bigendian': 0, 'encoding':

#### Explanation

- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore.
- The extension extracts ROS messages from an `.mcap` file stored in ReductStore.
- Only messages from the topic `/image_raw` are selected using the `topic` filter in the `ros.extract` configuration.
- The content of each message is CDR-encoded and decoded by the extension.
- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata.
- The decoded message is returned as JSON with the field `data`, which contains the image data encoded in JPEG format.
- Each record corresponds to one ROS 2 message and includes:
- Each record corresponds to one ROS message and includes:
- The decoded image parameters, such as height, width, encoding, and step.
- The timestamp from the message header.
- Message metadata as labels, including:
Expand Down
39 changes: 25 additions & 14 deletions docs/extensions/official/ros-ext/raw.mdx
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: Raw Messages
description: Extract raw ROS records as JSON or export them as MCAP using `$ros` entry attachments.
description: Extract raw ROS 1 and ROS 2 records as JSON or export them as MCAP using `$ros` entry attachments.
---

# Raw Messages

Raw Messages mode lets ReductROS decode ROS messages that are stored as individual records rather
than as MCAP or rosbag archives. It is intended for pipelines that store ROS 1 schema metadata in
the `$ros` [**attachment**](../../../glossary#attachment) of each entry.
than as MCAP or rosbag archives. It supports both ROS 1 and ROS 2 pipelines that store schema
metadata in the `$ros` [**attachment**](../../../glossary#attachment) of each entry.

import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote";
import CodeBlock from "@theme/CodeBlock";
Expand All @@ -20,7 +20,7 @@ import TabItem from "@theme/TabItem";

Raw Messages expects:

- one ROS 1 message per record
- one ROS 1 (`application/ros1`) or ROS 2 (`application/cdr`) message per record
- one or more queried entries
- a `$ros` attachment on each queried entry that provides the schema metadata

Expand All @@ -32,16 +32,16 @@ supports **extraction** to JSON and **export** to MCAP.
Raw Messages uses an attachment named `$ros` on each queried entry. The attachment must contain the
following JSON fields:

| Field | Type | Description |
| ------------- | -------- | ---------------------------------------------------------------- |
| `encoding` | `string` | Must be `ros1`. |
| `schema` | `string` | Full ROS 1 message definition used to decode the binary payload. |
| `topic` | `string` | Topic name associated with the stored records. |
| `schema_name` | `string` | ROS 1 message type, for example `std_msgs/String`. |
| Field | Type | Description |
| ------------- | -------- | ---------------------------------------------------------- |
| `encoding` | `string` | Must be `ros1` (ROS 1) or `cdr` (ROS 2). |
| `schema` | `string` | Full message definition used to decode the binary payload. |
| `topic` | `string` | Topic name associated with the stored records. |
| `schema_name` | `string` | ROS message type, for example `std_msgs/String`. |

If the records were ingested with `reduct-bridge`, this attachment is written automatically.

**Example `$ros` metadata**
**Example `$ros` metadata (ROS 1)**

```json
{
Expand All @@ -52,6 +52,17 @@ If the records were ingested with `reduct-bridge`, this attachment is written au
}
```

**Example `$ros` metadata (ROS 2)**

```json
{
"encoding": "cdr",
"schema": "string data",
"topic": "/chatter",
"schema_name": "std_msgs/msg/String"
}
```

## Query format

```python
Expand Down Expand Up @@ -81,7 +92,7 @@ from the `$ros` attachment of each queried entry.

## Data extraction

The `extract` property tells ReductROS how to decode each raw ROS 1 record and how to shape the
The `extract` property tells ReductROS how to decode each raw ROS record and how to shape the
resulting JSON output.

| Parameter | Type | Mandatory | Description |
Expand Down Expand Up @@ -115,7 +126,7 @@ timestamp.

## Data export

The `export` property converts raw ROS 1 records into MCAP episodes. It can aggregate data from
The `export` property converts raw ROS records into MCAP episodes. It can aggregate data from
multiple ReductStore entries into a single MCAP output stream, using the topic and schema from the
`$ros` attachment of each entry.

Expand All @@ -127,7 +138,7 @@ multiple ReductStore entries into a single MCAP output stream, using the topic a

When `export` is provided, the extension:

1. Reads raw ROS 1 records from the queried entries.
1. Reads raw ROS records from the queried entries.
2. Resolves schema and topic metadata for each entry from its `$ros` attachment.
3. Streams all matching messages into a new MCAP file.
4. Starts a new MCAP episode whenever `duration` or `size` is reached.
Expand Down
1 change: 0 additions & 1 deletion docs/integrations/grafana/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,6 @@ apiVersion: 1
datasources:
- name: reductstore
type: reductstore-datasource
url: http://localhost:8383
jsonData:
serverURL: http://localhost:8383
secureJsonData:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,20 @@ description: Extension for ReductStore to extract and transform data in ROS form
# ReductROS

The ReductROS [**extension**](../../glossary#extension) provides tools that can be used to extract and transform data stored in ReductStore into formats that are compatible with the Robot Operating System (ROS).
It allows users to query and convert historical data, such as sensor readings and camera images, into the JSON format, which can then be easily used for analysis, visualisation or integration with other systems.
It allows users to query and convert historical data, such as sensor readings and camera images, into JSON or filtered MCAP episodes for analysis, visualisation or integration with other systems across both ROS1 and ROS2 workflows.

import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote";

<DemoLicenseNote />

## Supported file and message formats
## Supported formats

The ReductROS extension supports parsing and extracting data from:
You can use the ReductROS extension to extract and transform data in the following formats:

- **[MCAP](./ros-ext/mcap)** files containing messages encoded in the **CDR (Common Data Representation)** format used by ROS 2.
- **[Rosbag](./ros-ext/rosbag)** archives from ROS 2 `rosbag2` in **SQLite3** or **MCAP** storage, uploaded as ZIP files containing `metadata.yaml`.
| Format | Typical input | What you can do |
| ------------------------------ | ----------------------- | -------------------------------------------------------------------- |
| **[MCAP](./ros-ext/mcap)** | MCAP files | Extract ROS1/ROS2 messages as JSON or create filtered MCAP episodes. |
| **[Rosbag](./ros-ext/rosbag)** | ROS2 `rosbag2` archives | Extract messages as JSON or create filtered rosbag episodes. |

import DocCardList from "@theme/DocCardList";

Expand Down
21 changes: 12 additions & 9 deletions versioned_docs/version-1.18.x/extensions/official/ros-ext/mcap.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Parameters and examples for working with MCAP files in ReductROS.

# MCAP

MCAP is a container format for recorded robotics data. In ROS 2, it commonly stores CDR-encoded messages across many topics with metadata like timestamps and schemas. ReductROS can extract and transform ROS 2 messages from MCAP files stored in ReductStore (content type `application/mcap`).
MCAP is a container format for recorded robotics data. It can store ROS1 and ROS2 messages across many topics together with timestamps, schemas, and channel metadata. ReductROS can extract and transform ROS1 and ROS2 messages from MCAP files stored in ReductStore (content type `application/mcap`).

import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote";

Expand Down Expand Up @@ -39,8 +39,7 @@ A user can use the `ext` query parameter to activate the `ros` extension and def
### Data extraction

The `extract` property allows you to specify the ROS topic from which to extract messages and convert them to JSON format.
The extension will read the MCAP file, decode the messages from the specified topic, and return each message as a JSON record with
the timestamp from the message header (when present).
The extension reads the MCAP file, selects the parser from the schema encoding of the requested channel, and returns each decoded message as a JSON record with the timestamp from the message header (when present).

| Parameter | Type | Mandatory | Description |
| ---------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
Expand All @@ -49,6 +48,10 @@ the timestamp from the message header (when present).
| `encode.<field>` | `string` | No | The encoding format for binary fields in the message. Supported values are `base64` and `jpeg`. If not specified, the field will be returned as a JSON list. |
| `as_label` | `object` | No | An object that specifies dynamic computed labels that will be delivered to the client. Can be a basis for output filtering. |

:::info
MCAP extraction supports both ROS1 (`ros1msg`) and ROS2 (`cdr`) schema encodings. Mixed MCAP files are supported, so ROS1 and ROS2 topics can coexist in the same input file and still be extracted correctly topic by topic.
:::

#### Encoding binary data

The `encode` property allows you to specify how to handle binary data in the extracted messages.
Expand Down Expand Up @@ -145,11 +148,11 @@ Record labels: {'@encoding': 'cdr', '@schema': 'std_msgs/String', '@topic': '/te

#### Explanation

- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore.
- The extension extracts ROS messages from an `.mcap` file stored in ReductStore.
- Only messages from the topic `/test` are selected using the `topic` filter in the `ros.extract` configuration.
- The content of each message is CDR-encoded and decoded by the extension.
- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata.
- The decoded message is returned as JSON with the field `data`, matching the `std_msgs/String` schema.
- Each record corresponds to one ROS 2 message and includes:
- Each record corresponds to one ROS message and includes:
- The decoded JSON payload, e.g., `{"data":"hello"}`
- Message metadata as labels, including:
- `topic`: `/test`
Expand Down Expand Up @@ -183,11 +186,11 @@ Image parameters: {'height': 720, 'width': 1280, 'is_bigendian': 0, 'encoding':

#### Explanation

- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore.
- The extension extracts ROS messages from an `.mcap` file stored in ReductStore.
- Only messages from the topic `/image_raw` are selected using the `topic` filter in the `ros.extract` configuration.
- The content of each message is CDR-encoded and decoded by the extension.
- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata.
- The decoded message is returned as JSON with the field `data`, which contains the image data encoded in JPEG format.
- Each record corresponds to one ROS 2 message and includes:
- Each record corresponds to one ROS message and includes:
- The decoded image parameters, such as height, width, encoding, and step.
- The timestamp from the message header.
- Message metadata as labels, including:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,6 @@ apiVersion: 1
datasources:
- name: reductstore
type: reductstore-datasource
url: http://localhost:8383
jsonData:
serverURL: http://localhost:8383
secureJsonData:
Expand Down
Loading