diff --git a/docs/extensions/official/ros-ext/index.mdx b/docs/extensions/official/ros-ext/index.mdx index 0d29da59c..5e64ffcfb 100644 --- a/docs/extensions/official/ros-ext/index.mdx +++ b/docs/extensions/official/ros-ext/index.mdx @@ -13,7 +13,7 @@ description: Extension for ReductStore to extract and transform data in ROS form # ReductROS The ReductROS [**extension**](../../glossary#extension) provides tools that can be used to extract and transform data stored in ReductStore into formats that are compatible with the Robot Operating System (ROS). -It allows users to query and convert historical data, such as sensor readings and camera images, into JSON or filtered MCAP episodes for analysis, visualisation or integration with other systems. +It allows users to query and convert historical data, such as sensor readings and camera images, into JSON or filtered MCAP episodes for analysis, visualisation or integration with other systems across both ROS1 and ROS2 workflows. import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote"; @@ -23,11 +23,11 @@ import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote"; You can use the ReductROS extension to extract and transform data in the following formats: -| Format | Typical input | What you can do | -| --------------------------------- | ------------------------ | ------------------------------------------------------------ | -| **[Raw Messages](./ros-ext/raw)** | Raw ROS records | Extract ROS messages as JSON or export them as MCAP. | -| **[MCAP](./ros-ext/mcap)** | MCAP files | Extract messages as JSON or create filtered MCAP episodes. | -| **[Rosbag](./ros-ext/rosbag)** | ROS 2 `rosbag2` archives | Extract messages as JSON or create filtered rosbag episodes. | +| Format | Typical input | What you can do | +| --------------------------------- | ----------------------- | -------------------------------------------------------------------- | +| **[Raw Messages](./ros-ext/raw)** | Raw ROS1/ROS2 records | Extract ROS1/ROS2 messages as JSON or export them as MCAP. | +| **[MCAP](./ros-ext/mcap)** | MCAP files | Extract ROS1/ROS2 messages as JSON or create filtered MCAP episodes. | +| **[Rosbag](./ros-ext/rosbag)** | ROS2 `rosbag2` archives | Extract messages as JSON or create filtered rosbag episodes. | import DocCardList from "@theme/DocCardList"; diff --git a/docs/extensions/official/ros-ext/mcap.mdx b/docs/extensions/official/ros-ext/mcap.mdx index d916749d3..1977c309a 100644 --- a/docs/extensions/official/ros-ext/mcap.mdx +++ b/docs/extensions/official/ros-ext/mcap.mdx @@ -5,7 +5,7 @@ description: Parameters and examples for working with MCAP files in ReductROS. # MCAP -MCAP is a container format for recorded robotics data. In ROS 2, it commonly stores CDR-encoded messages across many topics with metadata like timestamps and schemas. ReductROS can extract and transform ROS 2 messages from MCAP files stored in ReductStore (content type `application/mcap`). +MCAP is a container format for recorded robotics data. It can store ROS1 and ROS2 messages across many topics together with timestamps, schemas, and channel metadata. ReductROS can extract and transform ROS1 and ROS2 messages from MCAP files stored in ReductStore (content type `application/mcap`). import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote"; @@ -39,8 +39,7 @@ A user can use the `ext` query parameter to activate the `ros` extension and def ### Data extraction The `extract` property allows you to specify the ROS topic from which to extract messages and convert them to JSON format. -The extension will read the MCAP file, decode the messages from the specified topic, and return each message as a JSON record with -the timestamp from the message header (when present). +The extension reads the MCAP file, selects the parser from the schema encoding of the requested channel, and returns each decoded message as a JSON record with the timestamp from the message header (when present). | Parameter | Type | Mandatory | Description | | ---------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | @@ -49,6 +48,10 @@ the timestamp from the message header (when present). | `encode.` | `string` | No | The encoding format for binary fields in the message. Supported values are `base64` and `jpeg`. If not specified, the field will be returned as a JSON list. | | `as_label` | `object` | No | An object that specifies dynamic computed labels that will be delivered to the client. Can be a basis for output filtering. | +:::info +MCAP extraction supports both ROS1 (`ros1msg`) and ROS2 (`cdr`) schema encodings. Mixed MCAP files are supported, so ROS1 and ROS2 topics can coexist in the same input file and still be extracted correctly topic by topic. +::: + #### Encoding binary data The `encode` property allows you to specify how to handle binary data in the extracted messages. @@ -145,11 +148,11 @@ Record labels: {'@encoding': 'cdr', '@schema': 'std_msgs/String', '@topic': '/te #### Explanation -- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore. +- The extension extracts ROS messages from an `.mcap` file stored in ReductStore. - Only messages from the topic `/test` are selected using the `topic` filter in the `ros.extract` configuration. -- The content of each message is CDR-encoded and decoded by the extension. +- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata. - The decoded message is returned as JSON with the field `data`, matching the `std_msgs/String` schema. -- Each record corresponds to one ROS 2 message and includes: +- Each record corresponds to one ROS message and includes: - The decoded JSON payload, e.g., `{"data":"hello"}` - Message metadata as labels, including: - `topic`: `/test` @@ -183,11 +186,11 @@ Image parameters: {'height': 720, 'width': 1280, 'is_bigendian': 0, 'encoding': #### Explanation -- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore. +- The extension extracts ROS messages from an `.mcap` file stored in ReductStore. - Only messages from the topic `/image_raw` are selected using the `topic` filter in the `ros.extract` configuration. -- The content of each message is CDR-encoded and decoded by the extension. +- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata. - The decoded message is returned as JSON with the field `data`, which contains the image data encoded in JPEG format. -- Each record corresponds to one ROS 2 message and includes: +- Each record corresponds to one ROS message and includes: - The decoded image parameters, such as height, width, encoding, and step. - The timestamp from the message header. - Message metadata as labels, including: diff --git a/docs/extensions/official/ros-ext/raw.mdx b/docs/extensions/official/ros-ext/raw.mdx index 0db5bb6fa..dcf900610 100644 --- a/docs/extensions/official/ros-ext/raw.mdx +++ b/docs/extensions/official/ros-ext/raw.mdx @@ -1,13 +1,13 @@ --- title: Raw Messages -description: Extract raw ROS records as JSON or export them as MCAP using `$ros` entry attachments. +description: Extract raw ROS 1 and ROS 2 records as JSON or export them as MCAP using `$ros` entry attachments. --- # Raw Messages Raw Messages mode lets ReductROS decode ROS messages that are stored as individual records rather -than as MCAP or rosbag archives. It is intended for pipelines that store ROS 1 schema metadata in -the `$ros` [**attachment**](../../../glossary#attachment) of each entry. +than as MCAP or rosbag archives. It supports both ROS 1 and ROS 2 pipelines that store schema +metadata in the `$ros` [**attachment**](../../../glossary#attachment) of each entry. import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote"; import CodeBlock from "@theme/CodeBlock"; @@ -20,7 +20,7 @@ import TabItem from "@theme/TabItem"; Raw Messages expects: -- one ROS 1 message per record +- one ROS 1 (`application/ros1`) or ROS 2 (`application/cdr`) message per record - one or more queried entries - a `$ros` attachment on each queried entry that provides the schema metadata @@ -32,16 +32,16 @@ supports **extraction** to JSON and **export** to MCAP. Raw Messages uses an attachment named `$ros` on each queried entry. The attachment must contain the following JSON fields: -| Field | Type | Description | -| ------------- | -------- | ---------------------------------------------------------------- | -| `encoding` | `string` | Must be `ros1`. | -| `schema` | `string` | Full ROS 1 message definition used to decode the binary payload. | -| `topic` | `string` | Topic name associated with the stored records. | -| `schema_name` | `string` | ROS 1 message type, for example `std_msgs/String`. | +| Field | Type | Description | +| ------------- | -------- | ---------------------------------------------------------- | +| `encoding` | `string` | Must be `ros1` (ROS 1) or `cdr` (ROS 2). | +| `schema` | `string` | Full message definition used to decode the binary payload. | +| `topic` | `string` | Topic name associated with the stored records. | +| `schema_name` | `string` | ROS message type, for example `std_msgs/String`. | If the records were ingested with `reduct-bridge`, this attachment is written automatically. -**Example `$ros` metadata** +**Example `$ros` metadata (ROS 1)** ```json { @@ -52,6 +52,17 @@ If the records were ingested with `reduct-bridge`, this attachment is written au } ``` +**Example `$ros` metadata (ROS 2)** + +```json +{ + "encoding": "cdr", + "schema": "string data", + "topic": "/chatter", + "schema_name": "std_msgs/msg/String" +} +``` + ## Query format ```python @@ -81,7 +92,7 @@ from the `$ros` attachment of each queried entry. ## Data extraction -The `extract` property tells ReductROS how to decode each raw ROS 1 record and how to shape the +The `extract` property tells ReductROS how to decode each raw ROS record and how to shape the resulting JSON output. | Parameter | Type | Mandatory | Description | @@ -115,7 +126,7 @@ timestamp. ## Data export -The `export` property converts raw ROS 1 records into MCAP episodes. It can aggregate data from +The `export` property converts raw ROS records into MCAP episodes. It can aggregate data from multiple ReductStore entries into a single MCAP output stream, using the topic and schema from the `$ros` attachment of each entry. @@ -127,7 +138,7 @@ multiple ReductStore entries into a single MCAP output stream, using the topic a When `export` is provided, the extension: -1. Reads raw ROS 1 records from the queried entries. +1. Reads raw ROS records from the queried entries. 2. Resolves schema and topic metadata for each entry from its `$ros` attachment. 3. Streams all matching messages into a new MCAP file. 4. Starts a new MCAP episode whenever `duration` or `size` is reached. diff --git a/docs/integrations/grafana/index.mdx b/docs/integrations/grafana/index.mdx index ed44af81f..bf44909b6 100644 --- a/docs/integrations/grafana/index.mdx +++ b/docs/integrations/grafana/index.mdx @@ -174,7 +174,6 @@ apiVersion: 1 datasources: - name: reductstore type: reductstore-datasource - url: http://localhost:8383 jsonData: serverURL: http://localhost:8383 secureJsonData: diff --git a/versioned_docs/version-1.18.x/extensions/official/ros-ext/index.mdx b/versioned_docs/version-1.18.x/extensions/official/ros-ext/index.mdx index 0c79749f3..1839a0e10 100644 --- a/versioned_docs/version-1.18.x/extensions/official/ros-ext/index.mdx +++ b/versioned_docs/version-1.18.x/extensions/official/ros-ext/index.mdx @@ -13,18 +13,20 @@ description: Extension for ReductStore to extract and transform data in ROS form # ReductROS The ReductROS [**extension**](../../glossary#extension) provides tools that can be used to extract and transform data stored in ReductStore into formats that are compatible with the Robot Operating System (ROS). -It allows users to query and convert historical data, such as sensor readings and camera images, into the JSON format, which can then be easily used for analysis, visualisation or integration with other systems. +It allows users to query and convert historical data, such as sensor readings and camera images, into JSON or filtered MCAP episodes for analysis, visualisation or integration with other systems across both ROS1 and ROS2 workflows. import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote"; -## Supported file and message formats +## Supported formats -The ReductROS extension supports parsing and extracting data from: +You can use the ReductROS extension to extract and transform data in the following formats: -- **[MCAP](./ros-ext/mcap)** files containing messages encoded in the **CDR (Common Data Representation)** format used by ROS 2. -- **[Rosbag](./ros-ext/rosbag)** archives from ROS 2 `rosbag2` in **SQLite3** or **MCAP** storage, uploaded as ZIP files containing `metadata.yaml`. +| Format | Typical input | What you can do | +| ------------------------------ | ----------------------- | -------------------------------------------------------------------- | +| **[MCAP](./ros-ext/mcap)** | MCAP files | Extract ROS1/ROS2 messages as JSON or create filtered MCAP episodes. | +| **[Rosbag](./ros-ext/rosbag)** | ROS2 `rosbag2` archives | Extract messages as JSON or create filtered rosbag episodes. | import DocCardList from "@theme/DocCardList"; diff --git a/versioned_docs/version-1.18.x/extensions/official/ros-ext/mcap.mdx b/versioned_docs/version-1.18.x/extensions/official/ros-ext/mcap.mdx index d916749d3..1977c309a 100644 --- a/versioned_docs/version-1.18.x/extensions/official/ros-ext/mcap.mdx +++ b/versioned_docs/version-1.18.x/extensions/official/ros-ext/mcap.mdx @@ -5,7 +5,7 @@ description: Parameters and examples for working with MCAP files in ReductROS. # MCAP -MCAP is a container format for recorded robotics data. In ROS 2, it commonly stores CDR-encoded messages across many topics with metadata like timestamps and schemas. ReductROS can extract and transform ROS 2 messages from MCAP files stored in ReductStore (content type `application/mcap`). +MCAP is a container format for recorded robotics data. It can store ROS1 and ROS2 messages across many topics together with timestamps, schemas, and channel metadata. ReductROS can extract and transform ROS1 and ROS2 messages from MCAP files stored in ReductStore (content type `application/mcap`). import DemoLicenseNote from "@site/src/components/promotional/DemoLicenseNote"; @@ -39,8 +39,7 @@ A user can use the `ext` query parameter to activate the `ros` extension and def ### Data extraction The `extract` property allows you to specify the ROS topic from which to extract messages and convert them to JSON format. -The extension will read the MCAP file, decode the messages from the specified topic, and return each message as a JSON record with -the timestamp from the message header (when present). +The extension reads the MCAP file, selects the parser from the schema encoding of the requested channel, and returns each decoded message as a JSON record with the timestamp from the message header (when present). | Parameter | Type | Mandatory | Description | | ---------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | @@ -49,6 +48,10 @@ the timestamp from the message header (when present). | `encode.` | `string` | No | The encoding format for binary fields in the message. Supported values are `base64` and `jpeg`. If not specified, the field will be returned as a JSON list. | | `as_label` | `object` | No | An object that specifies dynamic computed labels that will be delivered to the client. Can be a basis for output filtering. | +:::info +MCAP extraction supports both ROS1 (`ros1msg`) and ROS2 (`cdr`) schema encodings. Mixed MCAP files are supported, so ROS1 and ROS2 topics can coexist in the same input file and still be extracted correctly topic by topic. +::: + #### Encoding binary data The `encode` property allows you to specify how to handle binary data in the extracted messages. @@ -145,11 +148,11 @@ Record labels: {'@encoding': 'cdr', '@schema': 'std_msgs/String', '@topic': '/te #### Explanation -- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore. +- The extension extracts ROS messages from an `.mcap` file stored in ReductStore. - Only messages from the topic `/test` are selected using the `topic` filter in the `ros.extract` configuration. -- The content of each message is CDR-encoded and decoded by the extension. +- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata. - The decoded message is returned as JSON with the field `data`, matching the `std_msgs/String` schema. -- Each record corresponds to one ROS 2 message and includes: +- Each record corresponds to one ROS message and includes: - The decoded JSON payload, e.g., `{"data":"hello"}` - Message metadata as labels, including: - `topic`: `/test` @@ -183,11 +186,11 @@ Image parameters: {'height': 720, 'width': 1280, 'is_bigendian': 0, 'encoding': #### Explanation -- The extension extracts ROS 2 messages from an `.mcap` file stored in ReductStore. +- The extension extracts ROS messages from an `.mcap` file stored in ReductStore. - Only messages from the topic `/image_raw` are selected using the `topic` filter in the `ros.extract` configuration. -- The content of each message is CDR-encoded and decoded by the extension. +- The content of each message is decoded according to the schema encoding stored in the MCAP channel metadata. - The decoded message is returned as JSON with the field `data`, which contains the image data encoded in JPEG format. -- Each record corresponds to one ROS 2 message and includes: +- Each record corresponds to one ROS message and includes: - The decoded image parameters, such as height, width, encoding, and step. - The timestamp from the message header. - Message metadata as labels, including: diff --git a/versioned_docs/version-1.18.x/integrations/grafana/index.mdx b/versioned_docs/version-1.18.x/integrations/grafana/index.mdx index ed44af81f..bf44909b6 100644 --- a/versioned_docs/version-1.18.x/integrations/grafana/index.mdx +++ b/versioned_docs/version-1.18.x/integrations/grafana/index.mdx @@ -174,7 +174,6 @@ apiVersion: 1 datasources: - name: reductstore type: reductstore-datasource - url: http://localhost:8383 jsonData: serverURL: http://localhost:8383 secureJsonData: