Skip to content
Draft

WIP #82

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/contents/usage/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,57 @@ class DeviceClient:

The callback runs on every scrape, so it should be cheap and side-effect free.

## Identity info metrics (robot model, firmware, fleet manager version)

To track *what* is deployed — robot models, firmware versions, fleet manager version — and detect when a customer upgrades, the framework ships two canonical info gauges (the Prometheus `kube_pod_info` idiom: constant value 1, payload in the labels). Register them during `_connect()` once your API client can answer the questions:

```python
from inorbit_connector.metrics import (
register_robot_info_gauge,
register_fleet_manager_info_gauge,
)

# robot.info — one series per robot
register_robot_info_gauge(
robot_ids=lambda: self.robot_ids,
robot_info=lambda rid: self._api.robot_identity(rid),
# must return {"model": "MiR250", "firmware_version": "2.13.1"} or None
)

# fleet_manager.info — one series per process
register_fleet_manager_info_gauge(
version=lambda: self._api.fleet_manager_version(), # "3.4.0" or None
)
```

Exports as:

```
inorbit_connector_robot_info{robot_id="r1", model="MiR250", firmware_version="2.13.1"} 1
inorbit_connector_fleet_manager_info{version="3.4.0"} 1
```

Both schemas are **frozen**: `robot.info` carries exactly `robot_id` / `model` / `firmware_version` (the last two optional), `fleet_manager.info` exactly `version`. Unknown keys are dropped with a warning. Return `None` while the info isn't known yet — no placeholder series.

Why a separate info metric instead of a `model` label on every metric: model is per-robot, so it can't be a Resource attribute (those are per-process, like `connector_type`), and the canonical/SDK instruments have frozen schemas. The info-join gives you the slice anyway, against **every** per-robot series the framework emits, with zero call-site changes:

```promql
# Publish rate by robot model
sum by (model) (
rate(calls_publish_pose_total[5m])
* on (robot_id) group_left(model)
max by (robot_id, model) (inorbit_connector_robot_info)
)

# Upgrade detection: a (robot, version) combination that didn't exist 1h ago
inorbit_connector_robot_info unless inorbit_connector_robot_info offset 1h

# Fleet manager upgraded
inorbit_connector_fleet_manager_info unless inorbit_connector_fleet_manager_info offset 1h
```

Label hygiene: values must come from a **normalized catalog** — `"MiR250"`, not the raw upstream string (`"MiR 250 (rev. 4)"` vs `"mir250"` across firmware versions splits the series). Strip build metadata from versions (`2.13.1`, not `2.13.1-build.20260604`). Never include per-device-unique values (serials) — that's what `robot_id` is for.

## Production deployment

For multi-container deployments, see [`examples/metrics/`](https://github.com/inorbit-ai/inorbit-connector-python/tree/main/examples/metrics) for a reference OTel collector compose stack that:
Expand Down
128 changes: 128 additions & 0 deletions inorbit_connector/metrics/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -355,3 +355,131 @@ def _session_callback(_options):
"process is alive but its MQTT link to InOrbit is down."
),
)


# --- Identity info gauges ---------------------------------------------------
#
# Prometheus "info metric" idiom (kube_pod_info-style): a gauge whose value is
# the constant 1 and whose payload lives in the labels. Joining any per-robot
# series against robot.info via `* on (robot_id) group_left(model) ...` lets
# dashboards slice every metric by robot model or firmware version without
# stamping those labels on each metric — the canonical and SDK instruments
# have frozen schemas and could not carry them anyway.
#
# Both schemas are frozen by the framework: robot.info carries exactly
# {robot_id, model, firmware_version} and fleet_manager.info exactly
# {version}. Unknown keys are dropped (with a warning) so a typo at a call
# site cannot permanently pollute the descriptor downstream.

_ROBOT_INFO_ALLOWED_KEYS = frozenset({"model", "firmware_version"})


def register_robot_info_gauge(robot_ids, robot_info) -> None:
"""Register the canonical per-robot identity info gauge (``robot.info``).

Emits one series per robot with the constant value 1 and identity
attributes as labels::

inorbit_connector_robot_info{robot_id="r1", model="MiR250",
firmware_version="2.13.1"} 1

A robot upgrading its firmware (or being swapped for a different model)
shows up as a label change: the old series goes stale and a new one
appears — which is the alertable event for "the customer changed
something".

All callbacks run on every Prometheus scrape, so they should be cheap
and side-effect free — return cached values, never make an upstream
call. No-op when the OTel API is not installed.

Args:
robot_ids: zero-arg callable returning the current list of robot ids
in the fleet.
robot_info: callable ``(robot_id: str) -> dict | None`` returning the
identity attributes for that robot. Supported keys: ``model``,
``firmware_version`` — both optional; unknown keys are dropped
with a warning. Values MUST come from a normalized catalog
(``"MiR250"``, not the raw upstream string) — see the metrics
guide on label normalization. Return ``None`` (or ``{}``) while
the info is not yet known to omit that robot from the scrape
instead of emitting placeholder values.
"""
if not OTEL_API_AVAILABLE:
return

def _robot_info_callback(_options):
observations = []
for rid in robot_ids():
info = robot_info(rid)
if not info:
continue
unknown = set(info) - _ROBOT_INFO_ALLOWED_KEYS
if unknown:
_logger.warning(
"robot.info attributes %s for robot %s are outside the "
"frozen schema %s and were dropped",
sorted(unknown),
rid,
sorted(_ROBOT_INFO_ALLOWED_KEYS),
)
attrs = {
key: value
for key, value in info.items()
if key in _ROBOT_INFO_ALLOWED_KEYS and value
}
if not attrs:
continue
observations.append(Observation(1, {"robot_id": rid, **attrs}))
return observations

meter.create_observable_gauge(
"robot.info",
callbacks=[_robot_info_callback],
unit="1",
description=(
"Constant 1; per-robot identity (model, firmware_version) rides "
"in the labels. Join other per-robot series against this metric "
"to slice them by model or firmware version."
),
)


def register_fleet_manager_info_gauge(version) -> None:
"""Register the fleet manager identity info gauge (``fleet_manager.info``).

Emits a single series with the constant value 1 and the upstream fleet
manager's software version as a label::

inorbit_connector_fleet_manager_info{version="3.4.0"} 1

An upstream upgrade shows up as a label change — the old series goes
stale and a new one appears.

The callback runs on every Prometheus scrape: return a cached value,
never make an upstream call. No-op when the OTel API is not installed.

Args:
version: zero-arg callable returning the fleet manager's version
string, normalized (e.g. ``"3.4.0"``, stripped of build
metadata). Return ``None`` while unknown to emit nothing
instead of a placeholder.
"""
if not OTEL_API_AVAILABLE:
return

def _fleet_manager_info_callback(_options):
value = version()
if not value:
return []
return [Observation(1, {"version": value})]

meter.create_observable_gauge(
"fleet_manager.info",
callbacks=[_fleet_manager_info_callback],
unit="1",
description=(
"Constant 1; the upstream fleet manager's software version rides "
"in the 'version' label. A version change indicates the customer "
"upgraded their fleet manager."
),
)
147 changes: 147 additions & 0 deletions tests/test_metrics_info_gauges.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# SPDX-FileCopyrightText: 2026 InOrbit, Inc.
# SPDX-License-Identifier: MIT

import logging
from unittest.mock import MagicMock

import pytest

from inorbit_connector import metrics as m


@pytest.fixture
def captured_meter(monkeypatch):
"""Replace the module-level meter so we can capture registered callbacks."""
fake_meter = MagicMock()
monkeypatch.setattr(m, "meter", fake_meter)
return fake_meter


def _registered_callback(fake_meter, name):
"""Return the single callback registered for the gauge `name`."""
for call in fake_meter.create_observable_gauge.call_args_list:
if call.args and call.args[0] == name:
return call.kwargs["callbacks"][0]
if call.kwargs.get("name") == name:
return call.kwargs["callbacks"][0]
raise AssertionError(f"no observable gauge registered with name {name!r}")


# --- robot.info -------------------------------------------------------------


class TestRobotInfoGauge:
def test_emits_one_observation_per_robot_with_identity_labels(
self, captured_meter
):
info = {
"r1": {"model": "MiR250", "firmware_version": "2.13.1"},
"r2": {"model": "MiR100", "firmware_version": "2.9.0"},
}
m.register_robot_info_gauge(lambda: ["r1", "r2"], lambda rid: info[rid])

observations = _registered_callback(captured_meter, "robot.info")(None)

assert len(observations) == 2
by_robot = {o.attributes["robot_id"]: o for o in observations}
assert by_robot["r1"].value == 1
assert by_robot["r1"].attributes["model"] == "MiR250"
assert by_robot["r1"].attributes["firmware_version"] == "2.13.1"
assert by_robot["r2"].attributes["model"] == "MiR100"

def test_robot_with_unknown_info_is_omitted(self, captured_meter):
info = {"r1": {"model": "MiR250"}, "r2": None}
m.register_robot_info_gauge(lambda: ["r1", "r2"], lambda rid: info[rid])

observations = _registered_callback(captured_meter, "robot.info")(None)

assert len(observations) == 1
assert observations[0].attributes["robot_id"] == "r1"

def test_partial_info_emits_only_known_keys(self, captured_meter):
m.register_robot_info_gauge(
lambda: ["r1"], lambda _rid: {"model": "Nipper"}
)

observations = _registered_callback(captured_meter, "robot.info")(None)

assert observations[0].attributes == {"robot_id": "r1", "model": "Nipper"}

def test_unknown_keys_are_dropped_with_warning(self, captured_meter, caplog):
caplog.set_level(logging.WARNING, logger=m.__name__)
m.register_robot_info_gauge(
lambda: ["r1"],
lambda _rid: {"model": "MiR250", "serial_number": "abc-123"},
)

observations = _registered_callback(captured_meter, "robot.info")(None)

assert observations[0].attributes == {"robot_id": "r1", "model": "MiR250"}
assert any("serial_number" in r.getMessage() for r in caplog.records)

def test_empty_values_are_dropped(self, captured_meter):
m.register_robot_info_gauge(
lambda: ["r1"],
lambda _rid: {"model": "", "firmware_version": "2.13.1"},
)

observations = _registered_callback(captured_meter, "robot.info")(None)

assert observations[0].attributes == {
"robot_id": "r1",
"firmware_version": "2.13.1",
}

def test_robot_with_only_unknown_or_empty_attrs_is_omitted(self, captured_meter):
m.register_robot_info_gauge(
lambda: ["r1"], lambda _rid: {"serial_number": "abc", "model": ""}
)

observations = _registered_callback(captured_meter, "robot.info")(None)

assert observations == []

def test_fleet_membership_is_read_per_scrape(self, captured_meter):
fleet = ["r1"]
m.register_robot_info_gauge(
lambda: fleet, lambda _rid: {"model": "MiR250"}
)
callback = _registered_callback(captured_meter, "robot.info")

assert len(callback(None)) == 1
fleet.append("r2") # robot added at runtime (fleet autodiscovery)
assert len(callback(None)) == 2


# --- fleet_manager.info -------------------------------------------------------


class TestFleetManagerInfoGauge:
def test_emits_single_observation_with_version_label(self, captured_meter):
m.register_fleet_manager_info_gauge(lambda: "3.4.0")

observations = _registered_callback(captured_meter, "fleet_manager.info")(
None
)

assert len(observations) == 1
assert observations[0].value == 1
assert observations[0].attributes == {"version": "3.4.0"}

def test_unknown_version_emits_nothing(self, captured_meter):
m.register_fleet_manager_info_gauge(lambda: None)

observations = _registered_callback(captured_meter, "fleet_manager.info")(
None
)

assert observations == []

def test_version_is_read_per_scrape(self, captured_meter):
versions = iter(["3.4.0", "3.5.0"])
m.register_fleet_manager_info_gauge(lambda: next(versions))
callback = _registered_callback(captured_meter, "fleet_manager.info")

assert callback(None)[0].attributes == {"version": "3.4.0"}
# Upstream upgraded between scrapes — the label value follows.
assert callback(None)[0].attributes == {"version": "3.5.0"}
Loading