diff --git a/.claude/skills/plotjuggler-plugin/SKILL.md b/.claude/skills/plotjuggler-plugin/SKILL.md new file mode 100644 index 00000000..61ac33a3 --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/SKILL.md @@ -0,0 +1,229 @@ +--- +name: plotjuggler-plugin +description: >- + Write or modify a PlotJuggler plugin (DataSource, MessageParser, Toolbox, or + Dialog) using the plotjuggler_sdk C++20 SDK. Use this whenever the task is to + implement, extend, debug, or build a PlotJuggler plugin — importing a file + format, streaming live data, parsing message payloads, adding a data-processing + toolbox, or building a plugin configuration dialog — even if the user does not + say the word "plugin" (e.g. "add a CSV loader to PlotJuggler", "make PJ read my + MCAP", "parse my protobuf topic", "a dialog for my source"). It gives the fast + path (which base class, which header, which macro), the correct CMake for both + in-tree and installed-SDK builds, and the load-bearing ABI/lifetime rules that + are easy to get wrong. Do NOT use it for changing the SDK's own ABI/protocol + (that is a maintainer task with different rules — see the note below). +--- + +# Writing a PlotJuggler plugin + +PlotJuggler is extended by four **plugin families**, each a small C++20 class you +subclass and export from a shared library (`.so`/`.dll`/`.dylib`). The SDK +(`plotjuggler_sdk`) gives you a base class per family; a one-line macro emits all +the C-ABI plumbing (entry point, version symbol, exception trampolines, symbol +folding) so **you never touch the raw ABI**. Your job is to override a handful of +virtual methods and ship the library. + +This skill is the author-oriented fast path. The repo's `pj_plugins/docs/` guides +are the full reference; this steers you to the right one and front-loads the +things that silently break a plugin. + +## Are you writing a plugin, or changing the SDK? + +- **Writing a plugin *against* the SDK** → this skill. The macros handle ABI + versioning, tail slots, `struct_size`, weak linkage, exception safety. You do + **not** need to read the ABI-stability rules. +- **Changing the SDK's own ABI/protocol** (adding vtable slots, editing a struct, + bumping a `PJ_*_PROTOCOL_VERSION`, touching `abi/baseline.abi`) → this is a + **maintainer** task. Stop and follow `pj_plugins/docs/ARCHITECTURE.md` §0a and + the "Release Versioning" contract in the root `CLAUDE.md`. Different rules apply. + +## Step 0 — How do you get the SDK? + +A plugin author obtains `plotjuggler_sdk` one of three ways. **All three end at the +same link line** — `target_link_libraries(my_plugin PRIVATE +plotjuggler_sdk::plugin_sdk)` — only the acquisition step differs: + +- **Conan package** (`plotjuggler_sdk`) — the common external route; the official + plugins use it. Pin the compatibility range explicitly: + `plotjuggler_sdk/[>=X.Y.Z <(X+1).0.0]`, where `X.Y.Z` is the version that + introduced the newest SDK feature you actually use (pre-1.0 the upper bound is + `<1.0.0` — the SDK guarantees no breaks within `0.x`). Tip from the official + repo: keep the version in a single `SDK_VERSION` file your `conanfile.py` reads. + Then `find_package(plotjuggler_sdk REQUIRED COMPONENTS plugin_sdk)`. +- **pixi / conda package** (`plotjuggler_sdk`) — same `find_package` + target once + the package is in your environment. +- **git submodule / vendored source** — `add_subdirectory(plotjuggler_sdk)`; the + same `plotjuggler_sdk::plugin_sdk` alias exists in-tree, and + `pj_emit_plugin_manifest` becomes available too, so the CMake below is identical. + (Editing the SDK repo itself is this world: the raw umbrella target is + `pj_plugin_sdk`; bare `pj_base` lacks the MessageParser and dialog SDK headers, + so prefer the umbrella. The `pj_plugins/docs/*-guide.md` snippets show + per-piece in-tree targets.) + +## Step 1 — Pick the family + +| Your goal | Family | Read | +|---|---|---| +| Turn a **file** or a **live source** (socket, serial, hardware) into topics | **DataSource** | `references/data-source.md` | +| Decode a **byte payload** on a topic into named fields (JSON, protobuf, ROS, custom) | **MessageParser** | `references/message-parser.md` | +| A **tool** that reads existing data, transforms it, and writes new topics | **Toolbox** | `references/toolbox.md` | +| A **configuration UI** (for a source/parser/toolbox, or standalone) | **Dialog** | `references/dialog.md` | + +A single `.so` can host **more than one family** — the supported pattern is an +owner + its embedded Dialog (one `PJ_*_PLUGIN(...)` macro per family at file +scope). Don't ship *unrelated* families in one DSO: catalog discovery reads one +primary family descriptor per library. + +If your data is object-like (image, point cloud, occupancy grid, transforms, +markers…) rather than scalar time series, also read `references/builtin-objects.md` +**before** writing — it changes what you emit and how. + +## Step 2 — The fast path per family + +Subclass the base, override the minimum, register with the macro. The macro's +second argument is the **manifest JSON literal** (see Step 3). + +| Family | Subclass | `#include` | Override (minimum) | Register | +|---|---|---|---|---| +| DataSource — file import | `PJ::FileSourceBase` | `pj_base/sdk/data_source_patterns.hpp` | `extraCapabilities()`, `importData()` | `PJ_DATA_SOURCE_PLUGIN(C, m)` | +| DataSource — live stream | `PJ::StreamSourceBase` | `pj_base/sdk/data_source_patterns.hpp` | `extraCapabilities()`, `onStart/onPoll/onStop()` | `PJ_DATA_SOURCE_PLUGIN(C, m)` | +| DataSource — manual | `PJ::DataSourcePluginBase` | `pj_base/sdk/data_source_plugin_base.hpp` | `capabilities()`, `start/stop()`, `currentState()` | `PJ_DATA_SOURCE_PLUGIN(C, m)` | +| MessageParser | `PJ::MessageParserPluginBase` | ⚠ `pj_plugins/sdk/message_parser_plugin_base.hpp` | `registerSchemaHandler()` in ctor/`bindSchema()` | `PJ_MESSAGE_PARSER_PLUGIN(C, m)` | +| Toolbox | `PJ::ToolboxPluginBase` | `pj_base/sdk/toolbox_plugin_base.hpp` | `capabilities()` (+ data ops) | `PJ_TOOLBOX_PLUGIN(C, m)` | +| Dialog | `PJ::DialogPluginTyped` | `pj_plugins/sdk/dialog_plugin_typed.hpp` + `.../widget_data.hpp` | `manifest()`, `ui_content()`, `widget_data()`, event handlers | `PJ_DIALOG_PLUGIN(C, m)` | + +> ⚠ **Header-location trap.** Three base classes live under `pj_base/sdk/`, but +> `MessageParserPluginBase` lives under **`pj_plugins/sdk/`**, and the Dialog SDK +> lives under `pj_plugins/sdk/` too (installed there from the `dialog_protocol` +> module). Some in-repo docs still show the parser header under `pj_base/` — that +> is stale; use the paths in the table. + +The quickest correct start is to copy a working example from this repo and edit it: +`examples/sdk_consumer/` (minimal external DataSource with the full CMake), +`pj_plugins/examples/mock_*.cpp` (DataSource, parser, toolbox, source+dialog), and +`pj_plugins/dialog_protocol/examples/mock_dialog.cpp` (standalone dialog). Each +family reference embeds an API skeleton for when the repo isn't at hand (domain +placeholders like `openSocket()` are yours to fill in). + +## Step 3 — Build it + +One CMakeLists.txt serves every acquisition channel from Step 0: + +```cmake +cmake_minimum_required(VERSION 3.22) +project(my_plugin LANGUAGES CXX) +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +# Conan / pixi / installed package: +find_package(plotjuggler_sdk REQUIRED COMPONENTS plugin_sdk) +# Submodule / vendored source instead: add_subdirectory(plotjuggler_sdk) + +add_library(my_plugin SHARED my_plugin.cpp) +target_link_libraries(my_plugin PRIVATE plotjuggler_sdk::plugin_sdk) # same in all worlds + +pj_emit_plugin_manifest(my_plugin + FAMILY data_source # data_source | message_parser | toolbox | dialog + MANIFEST_FILE ${CMAKE_CURRENT_SOURCE_DIR}/manifest.json +) +``` + +The single `plotjuggler_sdk::plugin_sdk` component is the whole author surface — +base + parser SDK + dialog SDK. You do **not** link a separate dialog target +downstream (`pj_dialog_sdk` is an in-tree name). + +**Manifest — two things named "manifest", only one is authoritative:** + +1. **The JSON literal you pass to `PJ_*_PLUGIN(Class, "…")`** is embedded in the + library and is the **source of truth** the host reads at discovery/load. + Required keys: `id`, `name`, `version`. Family extras: MessageParser **must** + include `"encoding": ["json", …]` (the host routes payloads by these names, case- + sensitive); DataSource may add `"file_extensions": [".csv"]`. +2. **`manifest.json` + `pj_emit_plugin_manifest(...)`** is **optional but + recommended**. It writes a human-readable `.pjmanifest.json` sidecar + (for tooling/packaging — runtime discovery does *not* read it) **and**, more + importantly, applies the symbol-isolation settings that stop your plugin's + symbols from clashing with the host's (hidden visibility everywhere; + `-Wl,-Bsymbolic-functions` on Linux/ELF — macOS's two-level namespace and + Windows' export model give the equivalent natively). Keep its `id/name/version` in sync with the macro literal — best by + making `manifest.json` the single source: generate a constexpr header from it at + build time and pass that to the macro (the official plugins do this with a small + CMake embed helper; see `pj-official-plugins/cmake/EmbedManifest.cmake` for a + ready-made one), instead of maintaining two hand-written copies. + +## Step 4 — The rules that silently break a plugin + +These cut across all families. The family references add family-specific traps +(dialog `buttonBox` naming, parser topic-scoping, toolbox `notifyDataChanged` +coalescing, etc.) — read them. + +1. **Never let an exception cross the ABI boundary.** Return + `PJ::unexpected("clear reason")` / `PJ::okStatus()` from fallible methods. The + macro's trampolines are only a safety net: fallible entry points convert a + stray exception's `what()` into the ABI error, non-fallible ones can only + swallow it, and an exception escaping a `noexcept` slot terminates the + process. Explicit `Status` returns are the only predictable diagnostics. +2. **Keep the data plane on the host's callback thread.** `writeHost()`, + `toolboxHost()`, and `pushMessage()` are single-threaded — a background I/O + thread must buffer under a mutex and hand data off inside `onPoll()`/`poll()`. + The authoritative source is the thread tag on each slot in the protocol + headers: slots tagged `[thread-safe]` — `reportMessage()`, + `isStopRequested()`, `notifyState()`, `requestStop()`, and the toolbox's + `notifyDataChanged()` — may be called from any thread (e.g. reporting a + connection loss from an I/O callback). Everything else stays on the callback + thread. +3. **Timestamps are `int64` nanoseconds since the Unix epoch.** Not seconds, not + row indices. Extract or synthesize a real absolute nanosecond value. Use + `std::chrono::system_clock` for receive-time stamps — **never** + `high_resolution_clock`, which is since-boot on some platforms. (Only data with + genuinely no time axis — e.g. a CSV with no time column — may fall back to a + documented synthetic index-as-time mode.) +4. **`saveConfig()`/`loadConfig()` must be a complete, deterministic round-trip** + with no ambient state (no `QSettings`, no cwd assumptions). Persist every + field. Concretely: a DataSource must run `loadConfig(saved) → start()` + headless with no dialog; a parser must decode from its config alone; dialogs + and toolboxes restore from the layout *before data arrives* — so tolerate + configs from older plugin versions (migrate or default missing keys), don't + *fail* `loadConfig()` just because referenced data isn't loaded yet, and + re-resolve in `onDataChanged()`. +5. **String lifetimes across the ABI.** SDK overrides return `std::string` — the + base class buffers it for the ABI, one buffer per slot, invalidated at the + next call of the *same* method on the same instance. So never hold the host's + view of a previous return, and if you ever hand out a raw `const char*` + yourself, it must stay valid until your next call. +6. **Arrow buffers use RAII holders.** Wrap read out-params in + `PJ::sdk::ArrowSchemaHolder` / `ArrowArrayHolder`; wrap write streams in + `PJ::sdk::ArrowStreamHolder`. On a **successful** append the host takes + ownership (the holder disarms); on **failure** you still own it (the holder + releases). Never release twice. +7. **Guard host access.** Check `writeHostBound()` / `runtimeHostBound()` (or the + toolbox equivalents) before using a host view. +8. **Object-like data → a builtin type + its codec.** Images, point clouds, + grids, transforms, markers go through the builtin object vocabulary, not raw + scalar fields. See `references/builtin-objects.md`. + +## Verify + +- Builds clean and produces a shared library. +- `PJ_*_PLUGIN` macro present exactly once per family; manifest literal has the + required keys (+ `encoding` for a parser). +- Load it in PlotJuggler, or better: a unit test that `dlopen`s the **real built + `.so`** through the host loaders (`DataSourceLibrary::load(path)`, + `MessageParserLibrary::load(path)`, … — inject the path as a compile + definition), binds the SDK test helpers + (`pj_base/sdk/testing/parser_write_recorder.hpp`, + `pj_plugins/testing/toolbox_test_store.hpp`) as host services, and asserts on + what was written. This exercises the exact ABI surface the host uses. +- Round-trip `saveConfig()` → `loadConfig()` → run with no dialog. + +## Deep-dive routing + +| Question | Doc | +|---|---| +| What each family may do; capabilities; permission matrix; config contract | `pj_plugins/docs/REQUIREMENTS.md` | +| How the C ABI / loaders / host bridge work (mostly maintainer detail) | `pj_plugins/docs/ARCHITECTURE.md` | +| Writing each family, in depth | `pj_plugins/docs/{data-source,message-parser,toolbox,dialog-plugin}-guide.md` | +| Dialog `WidgetData` setters + event-handler signatures | `docs/dialog-sdk-reference.md` | +| Builtin object types + their wire codecs | `docs/builtin_type.md`, `pj_base/include/pj_base/builtin/` | +| Object store: publish/read objects, ownership, lazy fetch | `V4_STORE.md` | +| C++ style, error handling, `PJ::Expected`, naming | `docs/cpp_design_recommendations.md` | diff --git a/.claude/skills/plotjuggler-plugin/references/builtin-objects.md b/.claude/skills/plotjuggler-plugin/references/builtin-objects.md new file mode 100644 index 00000000..58aba2df --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/references/builtin-objects.md @@ -0,0 +1,127 @@ +# Builtin object types + +Most plugins emit **scalar time series** (a named field → a number over time). But +PlotJuggler also has a canonical vocabulary of **object-like** types — images, +point clouds, grids, transforms, markers, etc. If your data is one of these, emit +the matching builtin type (as serialized bytes on an *object topic*) instead of +faking it with scalar fields. This is the shim that keeps ROS/protobuf/vendor +specifics out of PlotJuggler's viewers and storage. + +Reference: `docs/builtin_type.md`, headers in `pj_base/include/pj_base/builtin/`, +object store ABI in `V4_STORE.md`. + +## Decision: builtin type or raw scalars? + +1. **Is the payload a number (or a few named numbers) over time?** → raw scalar + fields (`writeHost().appendRecord(...)`). No builtin. Done. +2. **Is it object-like?** Match it to a builtin: + - image (raw or compressed) → `Image`; depth + intrinsics → `DepthImage` + - 3D points → `PointCloud`, or `CompressedPointCloud` (opaque compressed) + - 2D overlays on an image → `ImageAnnotations` + - 3D frame graph → `FrameTransforms`; camera calibration → `CameraInfo` + - occupancy/cost/ESDF → `OccupancyGrid` (+`OccupancyGridUpdate`) or `VoxelGrid` + - marker-style 3D primitives → `SceneEntities`; pose arrays → `PosesInFrame` + - mesh asset → `Mesh3D`; inter-frame video → `VideoFrame` + - URDF/SDF/MJCF text → `RobotDescription`; log line → `Log` + - findings/regions/events overlaid on a time-series plot → `PlotMarkers` +3. **Object-like but no builtin matches?** Emit meaningful scalars, or — when a + consumer that understands your format exists — a clearly-identified *custom* + object topic (the ObjectStore is deliberately payload-opaque). Never mislabel a + custom payload as a builtin. `BuiltinObjectType::kNone` means + "scalar/unknown classification", not "storage forbidden". + +The full list (17 concrete types) and their exact fields are in +`docs/builtin_type.md` and one header each under +`pj_base/include/pj_base/builtin/` (the `builtin_object.hpp` enum is the +authoritative roster — trust it over prose that may lag behind newer additions). + +## How you emit one + +Object topics are separate from scalar topics and go through an object-write view. +You obtain that view from the base class — for a DataSource it is +`objectWriteHost()`, which returns a **nullable** `const SourceObjectWriteHostView*` +(it is null when the host has no ObjectStore; a MessageParser/Toolbox has its own +`ParserObjectWriteHostView` / `ToolboxHostView` object methods). Always null-check +it, then register a topic once and push per sample: + +```cpp +#include // the matching codec + +const auto* objectHost = objectWriteHost(); // DataSource accessor; nullable +if (!objectHost) return PJ::unexpected("object store unavailable"); + +PJ::sdk::ImageAnnotations anno = /* fill in */; +std::vector bytes = PJ::serializeImageAnnotations(anno); // PJ:: (not PJ::sdk::); infallible + +// metadata_json is opaque to the store but read by VIEWERS to pick a renderer — +// an empty "{}" makes the topic unrenderable. Build it with MediaMetadataBuilder +// (pj_base/sdk/media_metadata.hpp): +const std::string meta = PJ::sdk::MediaMetadataBuilder().mediaClass("image_annotations").build(); + +auto topic = objectHost->registerTopic("overlays", meta); +if (!topic) return PJ::unexpected(topic.error()); +auto st = objectHost->pushOwned(*topic, timestamp_ns, bytes); // host copies the bytes +if (!st) return PJ::unexpected(st.error()); +``` + +Each builtin has a `*_codec.hpp` under `PJ::` (not `PJ::sdk::`) with +`serializeXxx()` / `deserializeXxx()`. Note the asymmetry: `serializeXxx()` returns +a plain `std::vector` (serialization can't fail), while `deserializeXxx()` +returns `PJ::Expected<...>`. The `RobotDescription` type is the exception — it +carries its source text as-is and has no codec. + +**MessageParsers emit objects differently — by returning, not pushing.** A parser's +`SchemaHandler.parse_object` returns an `ObjectRecord{optional ts, +BuiltinObject object}` — the host handles storage. Declare the handler's +`object_type` (`BuiltinObjectType::kImage`, …) so the host picks its ingest policy +*before* decoding any bytes, and propagate the incoming payload's `BufferAnchor` +into the returned object so large payloads stay zero-copy. See +`references/message-parser.md`. + +## Two storage families → two byte strategies + +- **Byte-backed** (Image, DepthImage, PointCloud, CompressedPointCloud, + OccupancyGrid(+Update), VoxelGrid, Mesh3D, VideoFrame): potentially megabytes. + Keep the payload a zero-copy `Span` anchored by a `BufferAnchor` + (typically a `shared_ptr>`); only allocate new bytes if a + conversion is unavoidable. +- **Owned values** (ImageAnnotations, FrameTransforms, SceneEntities, + RobotDescription, CameraInfo, PosesInFrame, Log, PlotMarkers): small (hundreds of + bytes) — the struct owns its `std::vector`s/strings directly; eager copy is fine. + +## Owned vs lazy push + +- **`pushOwned(topic, ts, bytes)`** — the host copies immediately; you may free your + buffer as soon as the call returns. +- **`pushLazy(topic, ts, fetch)`** — the host keeps your closure and calls it only + when a consumer reads that entry. Capture heavy state **by value** (e.g. a + `shared_ptr`), so it stays alive as long as the entry does. Use this + for large payloads you would rather read from a file on demand than hold in RAM. + +```cpp +auto st = objectHost->pushLazy(*topic, ts, + [reader = std::make_shared(path)] { return reader->readBytes(); }); +if (!st) return PJ::unexpected(st.error()); +``` + +The store owns the closure until the entry is evicted/removed, then destroys it +exactly once. Do not keep a raw pointer into anything the closure captured. + +## Frames & correlation (gotcha) + +- `PointCloud`/`Image`/`CameraInfo` carry a `frame_id`. To place moving-frame data + in a fixed frame, also emit `FrameTransforms` at the same timestamp; a 3D viewer + TF-transforms via the frame graph. +- `ImageAnnotations.image_topic` explicitly names the base image topic; camera + calibration correlates by **name convention** (`/camera_info` ↔ + `/image_raw`), not by a reference. +- **Association fields ride *outside* the serialized bytes.** The annotations + codec deliberately does not encode `image_topic` (or a timestamp) into the + payload — the store keys entries by (topic, timestamp), and association is a + topic/metadata convention. Don't expect a codec round-trip to carry them. + +## Codec round-trips are not bit-exact + +Some codecs normalize (e.g. circle radius↔diameter, RGBA float↔`uint8`). Do not +assert byte-for-byte equality after serialize→deserialize. See +`docs/image_annotations_format.md` for a worked example. diff --git a/.claude/skills/plotjuggler-plugin/references/data-source.md b/.claude/skills/plotjuggler-plugin/references/data-source.md new file mode 100644 index 00000000..dd6b8cb5 --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/references/data-source.md @@ -0,0 +1,189 @@ +# DataSource plugin + +A DataSource turns a file or a live source into topics + fields in PlotJuggler. +It is **write-only** (it cannot read existing host data). Pick the base class that +matches how your data arrives; each is a thin specialization of +`DataSourcePluginBase` that pre-declares the right capabilities and lifecycle. + +Full reference: `pj_plugins/docs/data-source-guide.md`. + +## Pick the base class + +| Data arrives as | Base class | You override | +|---|---|---| +| A **file**, imported once (CSV, Parquet, MCAP) | `PJ::FileSourceBase` | `extraCapabilities()`, `importData()` | +| A **live stream** you decode yourself | `PJ::StreamSourceBase` + `kCapabilityDirectIngest` | `extraCapabilities()`, `onStart/onPoll/onStop()` | +| A **transport** whose payload a MessageParser should decode (MQTT/ZMQ/UDP) | `PJ::StreamSourceBase` + `kCapabilityDelegatedIngest` | same, plus bind a parser and `pushMessage()` | +| None of the above | `PJ::DataSourcePluginBase` | `capabilities()`, `start/stop()`, `currentState()`, own state machine | + +Header: `pj_base/sdk/data_source_patterns.hpp` (the `*SourceBase` helpers) or +`pj_base/sdk/data_source_plugin_base.hpp` (the raw base). + +## Minimal file source + +```cpp +#include + +namespace { +class MyCsvSource : public PJ::FileSourceBase { + public: + uint64_t extraCapabilities() const override { return PJ::kCapabilityDirectIngest; } + + std::string saveConfig() const override { return config_; } + PJ::Status loadConfig(std::string_view json) override { + config_ = std::string(json); // host injects {"filepath":"/path", ...} here + return PJ::okStatus(); + } + + PJ::Status importData() override { + if (!writeHostBound() || !runtimeHostBound()) return PJ::unexpected("hosts not bound"); + auto topic = writeHost().ensureTopic("my/topic"); + if (!topic) return PJ::unexpected(topic.error()); + + // progressStart() is [[nodiscard]]; failure just means "host can't show it". + const bool progress = bool(runtimeHost().progressStart("Importing", total_rows_, /*cancellable=*/true)); + for (uint64_t i = 0; i < total_rows_; ++i) { + const PJ::sdk::NamedFieldValue fields[] = {{.name = "value", .value = rows_[i].value}}; + auto st = writeHost().appendRecord(*topic, rows_[i].timestamp_ns, fields); + if (!st) return PJ::unexpected(st.error()); + if (progress && !runtimeHost().progressUpdate(i)) { + return PJ::unexpected("import canceled by user"); // cancel is a real outcome, not ok + } + } + return PJ::okStatus(); // FileSourceBase calls progressFinish() for you + } + + private: + std::string config_ = "{}"; + // ... total_rows_, rows_ populated from the file named in config_ ... +}; +} // namespace + +PJ_DATA_SOURCE_PLUGIN(MyCsvSource, + R"({"id":"my-csv","name":"My CSV","version":"1.0.0","file_extensions":[".csv"]})") +``` + +## Minimal live stream (self-decoding) + +```cpp +#include +#include +#include +#include +#include + +class MyUdpSource : public PJ::StreamSourceBase { + public: + uint64_t extraCapabilities() const override { return PJ::kCapabilityDirectIngest; } + + std::string saveConfig() const override { return config_; } + PJ::Status loadConfig(std::string_view j) override { config_ = std::string(j); return PJ::okStatus(); } + + PJ::Status onStart() override { + fd_ = openSocket(); // your I/O + if (fd_ < 0) return PJ::unexpected("failed to open socket"); + running_.store(true); + io_ = std::thread([this] { recvLoop(); }); // background thread ONLY buffers + return PJ::okStatus(); + } + + PJ::Status onPoll() override { // host thread — the only place you touch the host + std::vector batch; + { std::lock_guard lk(mu_); batch.swap(buffer_); } + auto topic = writeHost().ensureTopic("udp/data"); + if (!topic) return PJ::unexpected(topic.error()); + for (auto& s : batch) { + const PJ::sdk::NamedFieldValue f[] = {{.name = "value", .value = s.value}}; + auto st = writeHost().appendRecord(*topic, s.timestamp_ns, f); + if (!st) return PJ::unexpected(st.error()); + } + return PJ::okStatus(); + } + + void onStop() override { // must be idempotent + running_.store(false); + if (fd_ >= 0) shutdownSocket(fd_); // unblock a blocking recv BEFORE join, or join() hangs forever + if (io_.joinable()) io_.join(); + if (fd_ >= 0) { closeSocket(fd_); fd_ = -1; } + } + + private: + struct Sample { PJ::Timestamp timestamp_ns; double value; }; + void recvLoop() { + while (running_.load()) { + Sample s = receiveOne(fd_); // do NOT call writeHost() here + std::lock_guard lk(mu_); buffer_.push_back(s); + } + } + int fd_ = -1; std::string config_ = "{}"; + std::atomic running_{false}; std::thread io_; std::mutex mu_; + std::vector buffer_; +}; + +PJ_DATA_SOURCE_PLUGIN(MyUdpSource, R"({"id":"my-udp","name":"My UDP","version":"1.0.0"})") +``` + +For **delegated** ingest (let a MessageParser decode the bytes): declare +`kCapabilityDelegatedIngest`, in `onStart()` call +`runtimeHost().ensureParserBinding({.topic_name=..., .parser_encoding="json", ...})`, +and in `onPoll()` push raw payloads with +`runtimeHost().pushMessage(*binding, ts, [bytes = std::move(bytes)] { return bytes; })`. +The fetch callable must capture its payload **by value** and be idempotent — the +host may invoke it zero, one, or many times, possibly from consumer threads, and +releases it exactly once even when the push fails. The config-envelope that ties +source⇆parser is managed by the host; you never see the parser directly. +Two delegated-ingest traps: + +- **Forward `_parser_config`.** The host injects the parser's saved options into + *your* `loadConfig()` JSON under the key `"_parser_config"`. Extract that string + and pass it as `ParserBindingRequest.parser_config_json` — otherwise + schema-based parsers bind unconfigured and silently drop every message. +- **Binding-unavailable is not an error.** If `ensureParserBinding` fails (e.g. no + parser installed for that encoding yet), skip the message and retry on a later + poll; only a failed `pushMessage` on an established binding is a real error. + +**Alternative receive shape — no thread at all:** if your transport offers a +non-blocking read (e.g. ZMQ `dontwait`), you can skip the background thread and +drain directly in `onPoll()` with a bounded loop (cap it, e.g. ≤100 messages per +poll, so a burst can't starve the host thread). Use the background-thread+buffer +pattern above when reads block or arrive faster than the poll cadence. + +## Capabilities + +Return them from `capabilities()` (raw base) or `extraCapabilities()` (`*SourceBase` +OR-adds them to the family default). Common flags: +`kCapabilityFiniteImport`, `kCapabilityContinuousStream`, `kCapabilityDirectIngest`, +`kCapabilityDelegatedIngest`, `kCapabilitySupportsPause`, `kCapabilityHasDialog`, +`kCapabilityPerTopicPause`. Declare only what you implement — flags gate what the +host lets you do. `kCapabilityPerTopicPause` in particular is not just a flag: it +also requires advertising available topics and implementing the topic-subscription +extension (`pluginExtension(PJ_TOPIC_SUBSCRIPTION_EXTENSION_V1)` returning a live +`PJ_topic_subscription_v1_t`) — see the per-topic-pause section of +`pj_plugins/docs/data-source-guide.md`. + +## Traps specific to DataSource + +- **Background threads never touch the host.** Buffer in plugin memory under a + mutex; flush in `onPoll()`/`poll()`. Calling `writeHost()` from your I/O thread + races the host and crashes. +- **`onStop()` must be idempotent** — it can be called more than once. Null out + handles after closing. +- **Streaming loses data if you block in `onPoll()`.** `onPoll()` runs at the + host's cadence; do the receiving on your own thread and use `onPoll()` only as + the hand-off point. +- **`FileSourceBase` calls `progressFinish()` for you.** Do not call it yourself + from `importData()`; a manual `DataSourcePluginBase` must call it itself. +- **Notify state transitions.** With the raw `DataSourcePluginBase`, call + `runtimeHost().notifyState(state)` on every transition you make, and never + `resume()` from a terminal (`stopped`/`failed`) state — the host makes a new + instance instead. +- **Timestamps: ns since epoch** (see SKILL.md rule 3). A row counter is not a + timestamp. + +## Embedding a configuration dialog + +Make the dialog a member and expose it via `getDialog()` returning +`PJ::borrowDialog(dialog_)`; add `kCapabilityHasDialog`; emit both macros +(`PJ_DATA_SOURCE_PLUGIN` and `PJ_DIALOG_PLUGIN`) in the same file. The source reads +the dialog member's state directly. The borrowed dialog must not outlive the +source. See `references/dialog.md` and `pj_plugins/examples/mock_source_with_dialog.cpp`. diff --git a/.claude/skills/plotjuggler-plugin/references/dialog.md b/.claude/skills/plotjuggler-plugin/references/dialog.md new file mode 100644 index 00000000..b484f88a --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/references/dialog.md @@ -0,0 +1,178 @@ +# Dialog plugin + +A Dialog is a **toolkit-neutral** configuration UI: your plugin links **no Qt**. It +describes the UI as Qt Designer `.ui` XML and exchanges state as JSON; the host +renders the widgets and relays events back over the vtable. A dialog can be +**standalone**, or **embedded** in a DataSource/Toolbox. + +Full reference: `pj_plugins/docs/dialog-plugin-guide.md`. +Setter + event-handler cheat sheet: `docs/dialog-sdk-reference.md`. + +## The reactive loop (the mental model) + +``` +host reads widget_data() (JSON) → renders widgets + → user interacts → host calls your onXxx() event handler + → you mutate state, return TRUE iff state changed + → if TRUE, host re-reads widget_data() → repeat +``` + +You never hold widgets. You describe *desired state* in `widget_data()` and react +to events. Return `true` **iff a re-rendered `widget_data()` would differ** from +what the host is already showing. That's subtler than "state changed": when the +user edits a line-edit and you only store the text, the widget already shows it — +return `false`. Return `true` when the event changes *other* widgets (enabling OK, +updating a preview, recomputing items). `true` always = wasted re-renders; `false` +after a dependent change = stale UI. + +## Headers + skeleton + +```cpp +#include +#include +#include + +namespace { +constexpr const char* kManifest = + R"({"id":"my-dialog","name":"My Dialog","version":"1.0.0"})"; + +// buttonBox MUST be named exactly "buttonBox" AND set standardButtons, or the +// dialog renders with no OK/Cancel and no error. +constexpr const char* kUi = R"( +MyDialog + + + + + + QDialogButtonBox::Cancel|QDialogButtonBox::Ok + + )"; +} // namespace + +class MyDialog : public PJ::DialogPluginTyped { + using PJ::DialogPluginTyped::onValueChanged; // avoid overload name-hiding + public: + std::string manifest() const override { return kManifest; } + std::string ui_content() const override { return kUi; } + + std::string widget_data() override { // describe current state + PJ::WidgetData wd; + wd.setText("name_input", name_); + wd.setValue("count_input", count_); + wd.setRange("count_input", 0, 1000); + wd.setOkEnabled(!name_.empty()); + return wd.toJson(); + } + + bool onTextChanged(std::string_view w, std::string_view t) override { + if (w == "name_input") { name_ = std::string(t); return true; } + return false; + } + bool onValueChanged(std::string_view w, int v) override { + if (w == "count_input") { count_ = v; return true; } + return false; + } + + std::string saveConfig() const override { // complete round-trip: every UI field + nlohmann::json cfg{{"name", name_}, {"count", count_}}; + return cfg.dump(); + } + bool loadConfig(std::string_view j) override { + auto cfg = nlohmann::json::parse(j, nullptr, false); + if (cfg.is_discarded()) return false; + const auto old_name = name_; + const auto old_count = count_; + name_ = cfg.value("name", name_); // tolerate missing keys + count_ = cfg.value("count", count_); + return name_ != old_name || count_ != old_count; // true iff state changed + } + + private: + std::string name_; + int count_ = 10; +}; + +PJ_DIALOG_PLUGIN(MyDialog, kManifest) +``` + +Downstream CMake links `plotjuggler_sdk::plugin_sdk` (dialog SDK is part of it); +in-tree, link `pj_dialog_sdk`. + +## Traps specific to Dialog + +- **In a modal/standalone dialog, the `QDialogButtonBox` must be named exactly + `buttonBox` and set `standardButtons` in the XML.** Otherwise it renders with no + OK/Cancel — and there is **no compile or runtime error**. This is the #1 dialog + bug. (Non-modal toolbox *panels* and parser-slot option panes legitimately have + no buttonBox at all — the rule applies where OK/Cancel semantics exist.) +- **Every interactive widget needs a unique `objectName`**, and your setter/handler + strings must match it exactly. A mismatch is silent — the state just doesn't apply. +- **Use `QPlainTextEdit` and `QTableWidget`**, not `QTextEdit` or a model-based + `QTableView` — the latter two are not supported by the widget binding. +- **Overload name-hiding:** `onValueChanged` has an `int` and a `double` overload. + Overriding one hides the other unless you add + `using PJ::DialogPluginTyped::onValueChanged;` in the class body. Same caution for + any other overloaded handler. +- **Return `true` iff state changed** (see the reactive loop above). +- **Do no I/O or blocking work in `widget_data()` or event handlers** — they run on + the GUI thread and freeze the UI. `onTick()` runs on the GUI thread too: use it + only for non-blocking checks or to drain results a worker thread produced + (return `true` to trigger a refresh) — the actual I/O belongs on the worker. +- **Do not retain the `const char*`/string returned by `widget_data()`** past the + next call (SKILL.md rule 5). +- **Config round-trip:** every UI-relevant field must survive + `saveConfig()`→`loadConfig()`, and `loadConfig()` returns `true` only when it + actually changed state. +- **`manifest()` must return the same JSON literal you pass to + `PJ_DIALOG_PLUGIN`** (as the skeleton does) — hosts on the legacy discovery + path read it from an instance via `get_manifest`. + +## `widget_data()` setters (common) + +`setText`, `setPlaceholder`, `setValue`, `setRange`, `setChecked`, `setItems`, +`setCurrentIndex`, `setLabel`, `setEnabled`, `setVisible`, `setOkEnabled`, +`setFilePicker`. Full list + which widget class each targets: +`docs/dialog-sdk-reference.md`. + +## Event handlers (common) + +`onTextChanged`, `onValueChanged(int)`, `onValueChanged(double)`, `onIndexChanged`, +`onToggled`, `onClicked`, `onFileSelected`, `onSelectionChanged`, +`onItemDoubleClicked`, plus lifecycle `onTick`, `onAccepted(final_state_json)`, +`onRejected`. Typed handlers **and `onTick()`** return `bool` (refresh needed); +`onAccepted`/`onRejected` return `void`. + +## Embedded vs standalone + +- **Standalone** — the whole `.so` is just the dialog; one `PJ_DIALOG_PLUGIN` macro. +- **Embedded** (in a DataSource/Toolbox) — the dialog is a member of the + source/toolbox class, exposed via `getDialog()` → `PJ::borrowDialog(dialog_)`; the + owner reads the dialog's state directly. Emit **both** family macros in the file. + The borrowed dialog must **not outlive** its owner. Example: + `pj_plugins/examples/mock_source_with_dialog.cpp`. + +Two patterns the official plugins converge on for embedded dialogs: + +- **The dialog owns the config.** The owner's `saveConfig()`/`loadConfig()` simply + delegate to the dialog member — one config schema, no duplication, and the + dialog is always consistent with what runs headless. +- **Owner→dialog data flows through injected callbacks.** The owner installs + `std::function`s on the dialog (e.g. "list available topics", "apply") instead of + the dialog reaching into the owner. Note for **non-modal toolbox panels**: + `requestAccept()` is ignored there — a "Save/Apply" button must invoke an + owner-installed callback instead. + +For anything beyond a trivial UI, keep the `.ui` as a real Qt Designer file and +embed it at build time (`pj_embed_ui` — see `pj_plugins/docs/dialog-plugin-guide.md`) +rather than growing an inline XML string; the file stays editable in Designer and +the plugin still links no Qt. + +## Static builds + +For static/WASM builds with no `dlopen`, hosts consume plugins through +`DataSourceLibrary::loadStatic`, `MessageParserLibrary::loadStatic`, or +`ToolboxLibrary::loadStatic` — each accepts an optional companion dialog vtable +for the embedded dialog (there is no standalone `DialogLibrary::loadStatic`). +Relevant only if you are wiring a static host; a normal shared-library plugin +needs nothing extra. diff --git a/.claude/skills/plotjuggler-plugin/references/message-parser.md b/.claude/skills/plotjuggler-plugin/references/message-parser.md new file mode 100644 index 00000000..9084ce4f --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/references/message-parser.md @@ -0,0 +1,138 @@ +# MessageParser plugin + +A MessageParser decodes one **byte payload** at a time into named fields (and +optionally builtin objects). The host binds it to exactly one topic and feeds it +messages. It is **write-only and topic-scoped**: you never name the topic — every +field you write is automatically namespaced under the bound topic. + +Full reference: `pj_plugins/docs/message-parser-guide.md`. + +## When a MessageParser is the WRONG choice + +If your input is a *file* or a *stream* you open yourself, you want a **DataSource**, +not a parser. A parser only decodes payloads the host hands it. A parser is right +when many payloads on a topic share an encoding (JSON, protobuf, ROS, a custom +binary) and you decode each into fields. + +## Header (trap) + +```cpp +#include // pj_plugins, NOT pj_base +``` + +This base class lives under `pj_plugins/sdk/`, unlike the DataSource/Toolbox bases +which live under `pj_base/sdk/`. Some in-repo docs show it under `pj_base/` — that +is stale. + +## The current model: register SchemaHandlers, don't override parse() + +The base class keeps a **handler table** keyed by schema/type name. You register a +`PJ::sdk::SchemaHandler` per schema you understand; the base class's `parse()` +dispatches through the table, and newer hosts call the scalar/object routes +directly. Overriding `parse()` yourself is the **legacy v4 path** — it still works, +but the SDK header marks it for deprecation, and every official parser uses the +handler table. A handler *returns* records; it does not call the write host: + +```cpp +#include +#include +#include + +namespace { +class MyParser : public PJ::MessageParserPluginBase { + public: + MyParser() { + // Register under "" so parsing works even when bindSchema() is never + // called (schema-less encodings like JSON land on the default entry). + registerSchemaHandler("", makeHandler()); + } + + // Re-register under the real type name once the host binds a schema. + PJ::Status bindSchema(std::string_view type_name, PJ::Span schema) override { + if (auto st = MessageParserPluginBase::bindSchema(type_name, schema); !st) return st; + if (findSchemaHandler(type_name) == nullptr) { + registerSchemaHandler(std::string(type_name), makeHandler()); + } + return PJ::okStatus(); + } + + private: + PJ::sdk::SchemaHandler makeHandler() { + return { + .object_type = PJ::sdk::BuiltinObjectType::kNone, // declare what you emit + .parse_scalars = + [this](PJ::Timestamp ts, PJ::Span payload) + -> PJ::Expected { + std::string_view text(reinterpret_cast(payload.data()), payload.size()); + auto value = PJ::parseNumber(text); // locale-independent + if (!value) return PJ::unexpected("payload is not a number"); + PJ::sdk::ScalarRecord rec; + rec.ts = std::nullopt; // nullopt → host uses the transport timestamp + rec.fields.push_back({.name = "value", .value = *value}); + return rec; + }, + .parse_object = nullptr}; + } +}; +} // namespace + +PJ_MESSAGE_PARSER_PLUGIN(MyParser, + R"({"id":"my-parser","name":"My Parser","version":"1.0.0","encoding":["json"]})") +``` + +Key pieces: + +- **`ScalarRecord`** = `{optional ts, vector fields}`. + Leave `ts` empty to use the host's transport timestamp; set it to override with a + timestamp embedded in the payload (e.g. a sensor's own clock) — typically gated + by a `use_embedded_timestamp` flag you read in `loadConfig()`. +- **`parse_object`** is the builtin-object route: return an `ObjectRecord` + (`{optional ts, BuiltinObject object}`) for images, point clouds, etc. + Set the handler's `object_type` to the matching `BuiltinObjectType` so the host + can choose its ingest policy *before* decoding bytes. See + `references/builtin-objects.md`. +- **Schema-carrying encodings** (protobuf, ROS, IDL): compile descriptors in + `bindSchema()` and register one handler per schema/type name. A static catalog + mapping type names → handlers scales well (the official ROS parser maps 20+ + types this way). + +## Optional overrides + +- `loadConfig(json)` / `saveConfig()` — parser options (array-size limits, + embedded-timestamp flag and field name, …). Tolerate unknown/missing keys. +- `parse()` — legacy direct route; only touch it when porting old code. Plugins + that populate the handler table inherit a working `parse()` from the base. + +## Traps specific to MessageParser + +- **`encoding` in the manifest is mandatory and case-sensitive.** The host routes + payloads to your parser by matching these names. `"json"` ≠ `"JSON"`. If it is + missing or mismatched, your parser is never invoked and the plugin looks dead. +- **Never name a topic.** The write path is pre-scoped to the bound topic; your + records' field names become `/` automatically. +- **Do not require `bindSchema()`.** Schema-less encodings never receive it — + hence the `registerSchemaHandler("")` default-entry idiom above. +- **String field values are views.** `NamedFieldValue.value` holds a + `string_view` for strings — the backing `std::string` must stay alive until the + record is consumed (official parsers keep a `std::deque` for + address stability within a parse call). +- **Stay consistent after a failed parse.** The next message must still decode. + Build schema-derived caches in `bindSchema()`/`loadConfig()`, never half-way + through a parse; don't rebuild descriptor pools per message. +- **Return diagnostic-quality errors.** `PJ::unexpected("line 5: unexpected token + at offset 42")` beats `"parse failed"`. + +## Configuration dialog + +A parser's dialog is an **independent, host-owned** instance (unlike a DataSource's +embedded dialog): the host creates it and bridges its config to parser instances as +JSON. See `references/dialog.md`. + +## Testing + +Two proven layers: +- Unit-test the decode logic directly (keep it in a plugin-free static lib). +- Load the **real built `.so`** through the host loader + (`MessageParserLibrary::load(path)`), bind a + `pj_base/sdk/testing/parser_write_recorder.hpp` as the write host, and assert on + the recorded rows — this exercises the ABI surface exactly as the host does. diff --git a/.claude/skills/plotjuggler-plugin/references/toolbox.md b/.claude/skills/plotjuggler-plugin/references/toolbox.md new file mode 100644 index 00000000..5698a984 --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/references/toolbox.md @@ -0,0 +1,124 @@ +# Toolbox plugin + +A Toolbox is an interactive tool that, unlike the other families, can **read** +existing host data, **transform** it, and **write** new topics or whole new data +sources. It usually has a dialog for its UI. + +Full reference: `pj_plugins/docs/toolbox-guide.md`. + +## Header + skeleton + +```cpp +#include + +namespace { +class MyToolbox : public PJ::ToolboxPluginBase { + public: + uint64_t capabilities() const override { + return 0; // or PJ::kToolboxCapabilityHasDialog if you embed a dialog + } + + std::string saveConfig() const override { return config_; } + PJ::Status loadConfig(std::string_view json) override { + config_ = std::string(json); + return PJ::okStatus(); + } + + void onDataChanged() override { + // host tells you the dataset changed; re-read if you cache anything + } + + // Do work when the user acts (e.g. from the dialog). Take the timestamps from + // the input series you transform — do not invent them. + PJ::Status applyTransform(PJ::Timestamp input_ts) { + if (!toolboxHostBound() || !runtimeHostBound()) return PJ::unexpected("hosts not bound"); + auto host = toolboxHost(); + auto source = host.createDataSource("my_output"); + if (!source) return PJ::unexpected(source.error()); + auto topic = host.ensureTopic(*source, "result"); + if (!topic) return PJ::unexpected(topic.error()); + + const PJ::sdk::NamedFieldValue f[] = {{.name = "value", .value = 99.0}}; + auto st = host.appendRecord(*topic, input_ts, PJ::Span(f, 1)); + if (!st) return PJ::unexpected(st.error()); + + runtimeHost().notifyDataChanged(); // ONE call per logical operation, after success + return PJ::okStatus(); + } + + private: + std::string config_ = "{}"; +}; +} // namespace + +PJ_TOOLBOX_PLUGIN(MyToolbox, + R"({"id":"my-toolbox","name":"My Toolbox","version":"1.0.0"})") +``` + +## Host services + +- **`toolboxHost()`** — data plane: `catalogSnapshot()` (enumerate + sources/topics/fields), `readSeriesArrow(field, &schema, &array)` (read a series), + `createDataSource()`, `ensureTopic()`, `appendRecord()` / `appendArrowStream()` + (write), plus **object writes** (`registerObjectTopic()`, `pushOwnedObject()`). +- **`objectReadHost()`** — **object reads** live on a separate, *optional* service: + it returns a nullable `const ToolboxObjectReadHostView*`. Null-check it before + reading ObjectStore data. +- **`runtimeHost()`** — control plane: `notifyDataChanged()`, `reportMessage()`. + +## Reading a series (Arrow) + +```cpp +#include + +PJ::sdk::ArrowSchemaHolder schema; +PJ::sdk::ArrowArrayHolder array; +auto st = toolboxHost().readSeriesArrow(field, schema.out(), array.out()); +if (st) { + // Two columns: children[0] = int64 timestamp (ns); children[1] = the field's + // value column, typed per field (commonly float64). Read while the holders are + // alive; they release at scope exit. +} +``` + +For bulk output build an `ArrowArrayStream` (e.g. via nanoarrow) and write it once +with `appendArrowStream(topic, std::move(streamHolder), "timestamp")` — far faster +than row-by-row `appendRecord`. + +## Traps specific to Toolbox + +- **Coalesce `notifyDataChanged()` yourself** — the host does not promise to. + Call it **once per logical operation**, not per row. Forgetting it entirely + means your new series never appear in the UI; calling it per record floods the + host. +- **Host methods only from the host thread** (SKILL.md rule 2). Background work + must marshal back. +- **Arrow RAII, no dangling pointers** (SKILL.md rule 6). Never keep a raw + `ArrowSchema*`/`ArrowArray*` past its holder's scope. +- **A `catalogSnapshot()` is a point-in-time view.** After you write and call + `notifyDataChanged()`, re-acquire the snapshot if you need to see your own writes. +- **Object writes are tail-slot-gated.** `registerObjectTopic()` / + `pushOwnedObject()` may be absent on an older host; check the returned + `Expected`/`Status` and degrade gracefully. + +## Emitting objects (images, clouds, …) + +`registerObjectTopic(source, name, metadata_json)` then +`pushOwnedObject(topic, ts, serialized_bytes)` — serialize with the matching +builtin codec first. See `references/builtin-objects.md` and `V4_STORE.md`. + +## Embedding a dialog + +Return `PJ::kToolboxCapabilityHasDialog` from `capabilities()`, keep the dialog as a +member, and expose it via `getDialog()` returning `PJ::borrowDialog(dialog_)`. Emit +`PJ_TOOLBOX_PLUGIN` and `PJ_DIALOG_PLUGIN` in the same file. For a persistent +(non-modal) panel rather than a pop-up, also OR in +`PJ::kToolboxCapabilityNonModalDialog` — and note that non-modal panels ignore +`requestAccept()`; wire "Apply/Save" through an owner-installed callback instead. +See `references/dialog.md`. + +## Testing + +`pj_plugins/testing/toolbox_test_store.hpp` is an in-memory store speaking the +toolbox host ABI (including the Arrow read path) so you can unit-test reads/writes +without a host.