Refactor package for unified data and device API

[bruno-f-cruz]

Refactor package for unified data and device API

Objective

Establish a clean, typed Python API for the Harp protocol that:

• works identically for live device communication and bulk file reads
• provides full static type support so IDEs and type-checkers give correct completions and error highlighting for register payloads
• exposes a declarative DSL (Field, BitFlag, GroupMask) that code generators for downstream device packages can emit directly from device.yml (the codegen is NOT part of this PR and likely will not belong in this repository)
• keeps each concern in its own installable package: wire protocol (harp-protocol), device/transport (harp-device + harp-serial), and tabular data (harp-data)

---

Packages

The repo is a uv workspace; every member lives under src/packages/* with a uniform src/-layout, and the top-level project is named harp. There are four packages:

harp-protocol   wire protocol, message parsing, payload/register DSL   (numpy only)
harp-device     transport-agnostic Device + ITransport, core registers, REGISTER_MAP
harp-serial     serial transport: SerialTransport + open_serial_device  (pulls in pyserial)
harp-data       load register data into pandas DataFrames               (pulls in pandas)

Separation of concerns

Each package owns exactly one concern, depends only on the layers below it, and isolates its heavy third-party dependency so consumers pay only for what they import:

Package	Owns (single responsibility)	Explicitly does NOT	Depends on	Heavy dep
harp-protocol	Wire protocol: message framing/parsing, the payload + register DSL (Field/BitFlag/GroupMask), and the pandas-free Column/to_columns() tabular view	No transport, no device lifecycle, no DataFrames	—	numpy only
harp-device	Transport-agnostic device: the Device base (framing, request/reply, reader thread, read/write), the ITransport protocol, core register definitions, REGISTER_MAP	No concrete transport (no serial/zmq), no DataFrames	harp-protocol	—
harp-serial	The serial transport: SerialTransport + the open_serial_device factory	No protocol/device logic — only moves bytes	harp-device, harp-protocol	pyserial
harp-data	Turning parsed register data into pandas DataFrames: read_dataframe / to_dataframe	No I/O, no protocol logic — consumes Columns	harp-protocol	pandas

Dependency edges:

• harp-device → harp-protocol
• harp-serial → harp-device, harp-protocol
• harp-data → harp-protocol

A downstream device package (e.g. harp-device-behavior) depends only on harp-protocol; it imports harp-device only if it also wants the live transport, harp-serial only if it wants serial, and harp-data only if it wants DataFrames. The heavy third-party dependencies (pandas, pyserial) live only in the leaf packages that need them, so harp-protocol and harp-device stay lightweight.

---

Major features/architecture

1. Register DSL (harp.protocol)

RegisterBase[U] is the schema object for a Harp register. Every register class carries address, payload_type, and payload_class as ClassVars (plus length for array registers) and exposes three uniform class-methods:

Method	Purpose
parse(value)	Decode a single HarpMessage (or raw bytes/bytearray/memoryview) → typed U
parse_bulk(source, *, parse_timestamp=True)	Decode a bulk binary buffer → (data, timestamps, message_types, payload) (numpy, zero-copy strided views)
format(value?, ...)	Build a Harp wire frame (Read if no value, Write otherwise)

Note: registers are numpy-only — they don't produce pd.DataFrames. Tabular output is a harp-data concern: harp.data.read_dataframe(register, source) (see §5).

The same register class works regardless of whether the source is a serial port reply, an in-memory buffer, or a .bin file on disk. No subclassing or reconfiguration is required.

Concrete register base-classes for every Harp scalar type are provided (RegisterU8, RegisterU16, …, RegisterFloat) as well as array variants (RegisterU16Array, …). These use a metaclass so that one-off anonymous registers can be created inline:

Reg = RegisterU16(address=0x20)
frame = Reg.format()

2. Payload DSL — anonymous (scalar / array) payloads

For scalar and array registers (no internal structure), PayloadU8, PayloadU16, …, PayloadFloatArray are provided. RegisterBase.parse unwraps these directly to a numpy scalar or ndarray — i.e. no .value accessor needed.

# ── Anonymous scalar register ─────────────────────────────────────────────
class WhoAmI(RegisterU16):
    address: ClassVar[int] = 0

who = WhoAmI.parse(msg)                # np.uint16 — no wrapper
frame = WhoAmI.format(np.uint16(1216)) # encode a write frame

NOTE I went with numpy types rather than native python types since these preserve bitdepth and add additional typing information. We should discuss. In general, while the type hinting may break, native python types will still get implicitly cast by numpy.

# ── Anonymous array register (e.g. AnalogData: 3 × S16) ──────────────────
# Array registers are sized by *calling* the base class with a length, which
# returns a concrete register type.
AnalogData = RegisterS16Array(44, length=3)

samples = AnalogData.parse(msg)        # NDArray[np.int16], shape (3,)

# Bulk parse — zero-copy strided views into the raw buffer
data, timestamps, msg_types, payload = AnalogData.parse_bulk(buf)

# DataFrame assembly lives in harp.data (see §5)
from harp.data import read_dataframe
df = read_dataframe(AnalogData, buf)   # columns: "timestamp", "0", "1", "2"

3. Payload DSL — structured payloads

StructPayload + Field / BitFlag / GroupMask descriptors declare the layout of a structured register payload. Under @dataclass_transform the class gets a fully type-checked keyword-only __init__ for free.

Field

Each member declares its byte position with offset= (in base-element units). offset defaults to 0, which suits a single-member payload; a payload with several members must give each an explicit offset= or the layout overlap-check rejects it.

from harp.protocol import StructPayload, Field, StringConverter, IdentityConverter

class FileSettings0Payload(StructPayload[np.uint8]):
    cycles:              np.uint32 = Field(converter=IdentityConverter(np.uint32), offset=0)
    duration_us:         np.uint32 = Field(converter=IdentityConverter(np.uint32), offset=4)
    update_frequency_hz: np.uint32 = Field(converter=IdentityConverter(np.uint32), offset=8)
    path:                str       = Field(converter=StringConverter(33),
                                           offset=12, default="220khzwaveform")

BitFlag + GroupMask — bit-packed payloads

Descriptor types for registers whose payload packs multiple boolean flags or enum values into a single byte (the OperationControl pattern common across all Harp devices). The right-shift for a GroupMask is derived automatically from the mask's trailing-zero count — there is no separate shift= argument.

Enum members are emitted in canonical UPPER_CASE (e.g. OperationMode.ACTIVE, EnableFlag.ENABLED), and the core registers use canonical names (HardwareVersionHigh, TimestampSeconds, ClockConfiguration, …) — both come straight from device.yml via the generator.

from harp.protocol import HarpMessage, MessageType
from harp.device import (
    OperationControl, OperationControlPayload, OperationMode, EnableFlag,
)

# ── Construct from kwargs — type-checked by IDE/type-checker ─────────────
p = OperationControlPayload(
    operation_mode=OperationMode.ACTIVE,
    heartbeat=EnableFlag.ENABLED,
)
print(p.operation_mode)   # OperationMode.ACTIVE   ← Python enum, no [0]
print(p.dump_registers)   # False                  ← Python bool

# ── Encode / decode a single frame ───────────────────────────────────────
frame  = OperationControl.format(p, message_type=MessageType.Write)
parsed = OperationControl.parse(HarpMessage.parse(frame))
assert parsed.heartbeat == EnableFlag.ENABLED

# ── Bulk read — same descriptors, array-shaped results ───────────────────
data, timestamps, _, payload = OperationControl.parse_bulk(buf)
print(payload.operation_mode)   # NDArray of raw enum codes, shape (N,)

from harp.data import read_dataframe
df = read_dataframe(OperationControl, buf, decode_enums=True)
# columns: "timestamp", "operation_mode" (Categorical), "dump_registers", …

The descriptors are shape-polymorphic: a 0-D (single) parse returns a Python scalar; a 1-D (bulk) parse returns a numpy array. The descriptor code is identical in both cases.

4. Device & transport layer (harp-device + harp-serial)

The device side uses composition over inheritance, split into two independent axes so they multiply without a diamond:

• what the device is — registers, __whoami__, device logic → the Device axis
• how bytes move — serial, zmq, file → the ITransport axis

Device  (protocol logic)  ──has-a──▶  ITransport  (byte channel)

Device (in harp.device) owns all protocol logic — framing, the request/reply correlation, the reader thread, and read/write — and drives an ITransport. ITransport is a typing.Protocol (structural — any object with open/write/read/close qualifies; failures surface as TransportError). A device must be opened (with / .open()); opening starts the reader thread and validates __whoami__ (a class attribute; 0x0 skips the check).

harp-serial provides SerialTransport (conforms structurally to ITransport) and an open_serial_device factory that — like the builtin open — returns an already-connected device:

from harp.device import Device
from harp.serial import open_serial_device

with open_serial_device(Device, port="COM3", baudrate=1_000_000) as dev:
    print(dev.read(dev.WhoAmI).parsed)   # core registers are exposed as Device attrs

A device definition is a pure subclass with no transport concern — exactly what a generator emits:

class Behavior(Device):
    __whoami__ = 1216
    # + generated device-specific register class-attrs

Because the transport is just an ITransport, new ones drop in with one implementation + a factory and zero changes to device classes:

• zeromq — talk to a device (or broker) over a ZMQ socket: open_zmq_device(MyDevice, endpoint=...).
• http — drive a device behind an HTTP/REST bridge.
• a file-system "device" — a transport (or a sibling factory) that "knows" how to open register datastreams from a standard harp folder structure (similar to what harp-python implements currently)

harp.device also exposes a REGISTER_MAP: dict[int, type[RegisterBase[Any]]] of the core registers, which a downstream (generated) package spreads into its own map:

from harp.device import REGISTER_MAP as _CORE_REGISTER_MAP

REGISTER_MAP = {**_CORE_REGISTER_MAP, 32: DigitalInputState, 34: OutputSet, ...}

5. Data layer (harp-data)

Tabular/DataFrame concerns live in harp-data. The payload layer (harp-protocol) exposes a pandas-free, numpy-only tabular view via to_columns(), returning list[Column]:

@dataclass(frozen=True, slots=True, eq=False)
class Column:
    name: str
    data: NDArray[Any]
    categories: Any | None = None   # labels if enum-backed (data holds codes), else None

harp-data turns those columns into a DataFrame — building pd.Categorical.from_codes for enum columns, which keeps the integer codes and copies nothing:

from harp.data import read_dataframe, to_dataframe

# register + source (path / bytes / open binary file) → DataFrame
df = read_dataframe(OperationControl, "OperationControl.bin", timestamp=True)

# or, from an already-parsed payload
_data, ts, _msg, payload = OperationControl.parse_bulk(buf)
df = to_dataframe(payload)

This keeps the heavy/optional pandas dependency out of the protocol and device layers, and gives harp-data room to grow into a home for further analysis utilities.

Also, this opens the door to future implementations (e.g: arrow), using the same protocol exposed interface.

6. Code-generation affordances

A downstream package for a specific device only needs to emit register/payload class declarations — the entire declaration is mechanical and driven directly by device.yml:

# Generated from device.yml — no boilerplate beyond this
class AnalogDataPayload(StructPayload[np.int16]):
    analog_input0 = Field(converter=IdentityConverter(np.int16), offset=0)
    encoder       = Field(converter=IdentityConverter(np.int16), offset=1)
    analog_input1 = Field(converter=IdentityConverter(np.int16), offset=2)

class AnalogData(RegisterBase[AnalogDataPayload]):
    address:       ClassVar[int]                     = 44
    payload_type:  ClassVar[PayloadType]             = PayloadType.S16
    payload_class: ClassVar[type[AnalogDataPayload]] = AnalogDataPayload

The generator emits one Field/BitFlag/GroupMask per payload member, one RegisterBase subclass per register, and (for a device package) a Device subclass with __whoami__ plus a spread REGISTER_MAP. No per-device framework extension is required; the output is pure application code that composes the DSL, and the constructor is type-hinted for free.

---

[MicBoucinha]

Hi @bruno-f-cruz,

I will check your latest changes as soon as possible.

In the meantime, I tried to update the generated-pyharp repo with some support for complex registers. If you have the opportunity to take a look at how are things right now, I appreciate your time.

Cheers!

[bruno-f-cruz]

Hi @bruno-f-cruz,

I will check your latest changes as soon as possible.

In the meantime, I tried to update the generated-pyharp repo with some support for complex registers. If you have the opportunity to take a look at how are things right now, I appreciate your time.

Cheers!

I took a quick look. I think this example summarizes my concerns:

class ComplexConfiguration(IRegister[ComplexConfigPayload]):
    spec = RegisterSpec[ComplexConfigPayload](
        address=34,
        payload_type=PayloadType.U8,
        decode=lambda p: ComplexConfigPayload(
            pwm_port=p[0],
            duty_cycle=struct.unpack_from("<f", bytes(p), 4)[0],
            frequency=struct.unpack_from("<f", bytes(p), 8)[0],
            events_enabled=p[12] != 0,
            delta=int.from_bytes(bytes(p[13:17]), "little"),
            name=bytes(p[17:50]).rstrip(b"\x00").decode("utf-8"),
        ),
        encode=lambda v: [
            v.pwm_port,
            0,
            0,
            0,
            *struct.pack("<f", v.duty_cycle),
            *struct.pack("<f", v.frequency),
            1 if v.events_enabled else 0,
            *v.delta.to_bytes(4, "little"),
            *v.name.encode("utf-8").ljust(33, b"\x00"),
        ],
        count=50,
        access=RegisterAccess.WRITABLE | RegisterAccess.EVENTFUL,
        payload_struct=(
            StructField("pwm_port", PayloadType.U8, offset=0),
            StructField("duty_cycle", PayloadType.FLOAT, offset=4),
            StructField("frequency", PayloadType.FLOAT, offset=8),
            StructField("events_enabled", PayloadType.U8, offset=12),
            StructField("delta", PayloadType.U32, offset=13),
            StructField("name", PayloadType.U8, offset=17, length=33, is_string=True),
        ),
    )

payload_struct essentially implements a subset of what this PR implements by using StructField to build numpy dtypes and back the bulk parsing (You should try to implement the full device.yml in the generators tests tho). We can discuss the differences in performance that i saw, but after ensuring parity in decisions across the two libraries (e.g: skip string conversion and do not copy pandas dataframes), the decoding time is virtually the same. This is not surprising since the backend is the same.

The points that worry me:

• You still need to maintain a parallel DSL that is struct-based (decode and encode methods)
• The argument of getting rid of numpy as a dependency is a bit weird since, when people do the code-generation, the device must still be dependent on the numpy that backs up the bulk parsing using the "StructField" API. It seems like the only thing we would win is the ability of not to use NumPy in the protocol at the cost of having to maintain two parsing code paths. I don't think this is worth it :P

[MicBoucinha]

Hi @bruno-f-cruz,

I tried to simplify and unify the registers and have a single point of encode/decode. I believe that this is somewhat similar to your solution. The resulting examples are much simpler and as far as I can see, it solves your main concern on the last point you mentioned regarding the two parsing code paths.

The new code in available in the generated-pyharp repo.

There are a few things I also tried to add from the new protocol definition from this PR (harp-tech/protocol#215) that weren't possible before. There is still somethings that I didn't start trying to implement (e.g. interfaceType). I wanted to get your feedback before continuing. Also, as a warning, there are most likely things I misinterpreted from the definition.

I've also updated the benchmark scripts, although some of the new comparisons aren't really fair with harp-python with the complex registers since it handles them as U8 as far as I could understand. But I think that we can get at least an idea of what it is possible. I updated the tests also, although I think that more tests are required. If you have suggestions of things I've missed, please let me know.

I wanted to compare performance with your implementation, especially regarding Complex registers for bulk reads but I couldn't find a benchmark on your side for that. I might have missed it though.

Thanks!

[glopesdev]

@MicBoucinha can you provide a link to the code you are mentioning, ideally with a pinned SHA? That would be really helpful (at least for me) to compare.

What is the goal here? @bruno-f-cruz already has a fully compliant implementation here, so I implemented the generators with CI/CD to track that.

Ideally I only want to do the full integration work for one implementation, so I'm a bit lost on which one we are going to go for. Bear in mind we need to support all register formats, including interface types which is why implementing all unit tests is important. But also if we are duplicating work it would probably be good to synchronise on why we are doing so, no?

[MicBoucinha]

Hi @glopesdev,

Here is the link: github.com/fchampalimaud/generated-pyharp/tree/58bdb5dd827ddb0a1e33eb890c6e900e663d87eb.

The goal here was, as talked before, to show a viable option of implementation that doesn't require numpy in the harp-protocol package and keep that dependency only on harp-data. I want to know if this is "comparable performance" as @bruno-f-cruz's implementation but I couldn't find exact benchmarks for complex registers here (at the time of posting my previous comment) so I could compare it.

[bruno-f-cruz]

@MicBoucinha @glopesdev I have added an internal package to this PR to run benchmarks. I think this is useful to compare notes but also as a regression test harness.

To run simply: uv run harp-benchmark

For comparison with your implementation, I would urge you to target the EXACTLY same set of registers I am targeting since those are the tests used for the generators package. If you have questions let me know!

[MicBoucinha]

Hi @bruno-f-cruz.

I ran your benchmark on my computer and I got the attached file as a result.

At the same time, I updated the generated-pyharp repo (https://github.com/fchampalimaud/generated-pyharp/tree/5f8577f2ebdba044f88401c5d322593e0a42c8c1) with the following:

• Add support for interface_type
• Replicated your harp-benchmarks package (and registers) so that we can have a similar point of comparison. I tried to implement what you are benchmarking (at the closest points you are testing) to the best of my ability but something might be missing or being interpreted wrongly on my end. I want to apologize beforehand and appreciate your feedback if there is something clearly wrong on my end. I wanted to have something that we could eventually discuss in the meeting.
• Added a compare script that receives two reports.md and creates a new .md file with the running values side by side with a delta

To run the compare script you can use something similar to:

uv run harp-benchmark-compare .\benchmark\report-generated-pyharp.md .\benchmark\report-pyharp-pr5.md -o .\benchmark\comparison.md --label-a generated-pyharp --label-b pyharp-pr5

While implementing HarpVersion, I had a question regarding how these types are represented in a single column on the DataFrame, but it seemed to me (please correct me if I'm wrong) that the Hash: list[int] is expanded. I don't think I understood fully when they should be on a single column or expanded so I would appreciate your help in explaining the rationale behind it.

Here are the 3 files:
report-generated-pyharp.md
report-pyharp-pr5.md
comparison.md

We can discuss it further in the meeting. I still feel that I am probably missing something in the interpretation of the interface specification.

Cheers!

[MicBoucinha]

Hi @bruno-f-cruz,

Regarding what we spoke last time on the meeting, here his the explanation on how things are working right now.

Everything is based on PayloadType. We use this as the source of truth as defined in the protocol.

Single-message parsing

Single-message parsing occurs in 4 steps:

• Step 1 - get PayloadTypefrom the message bytes, extracts its size, skipping timestamp if it exists and calculates the element count.

• Step 2 - Uses struct_charto get the structformat char directly from the PayloadType's bit flags

_UNSIGNED_CHARS = {1: "B", 2: "H", 4: "I", 8: "Q"}

def struct_char(self) -> str:
    if self.is_float():
        return "f"
    c = _UNSIGNED_CHARS[self.type_size()]
    return c.lower() if self.is_signed() else c

• Step 3 - Creates a struct.Struct(f"<{count}{char}") and calls .unpack(raw). From there it returns plain Python int, float or list.

• Step 4 - RegisterBase.parse(message) calls cls.decode(message.payload) to convert the raw scalar/list into the register's typed representation (an IntEnum, a StructPayload dataclass, etc.).

Bulk dump loading (harp-data)

Everything also occurs in 4 steps:

• Step 1 - load_register_dump()receives a file path and a RegisterBase subclass.

• Step 2 - Calls numpy_scalar_dtype(payload_type) where it will pick up the PayloadTypeand convert it to a corresponding np.dtype. This is done by reading the type_size(), is_float() and is_signed()methods from PayloadType.

def numpy_scalar_dtype(payload_type):
    itemsize = int(payload_type.type_size())
    if payload_type.is_float():
        return np.dtype("<f4")
    kind = "i" if payload_type.is_signed() else "u"
    return np.dtype(f"<{kind}{itemsize}")

• Step 3 - Constructs the complete structured frame's dtype that describes the entire frame layout, using build_frame_dtype()

• Step 4 - Reads the file's raw bytes with np.fromfile / np.frombuffer, then creates a zero-copy view: raw.view(frame_dtype).

Notes

It is important to note, that Complex registers also go through this. The PayloadTypeis there, and it is well defined.

Both paths derive from the same three PayloadType properties: type_size(), is_float(), is_signed(). One path produces struct format chars for single-message parsing, the other produces numpy dtype strings for bulk analysis. They cannot drift because neither maintains a separate mapping since both read the PayloadType's bit flags.

Sure, we have two parsing paths but the cost of maintaining this seems small since we have two small functions that base themselves on the same source. It is not like we have two parallel lookup tables or something more complex to maintain here. The big advantage is that we keep numpywhere it belongs, on the data analysis side of things, while keeping the protocol and transport numpyfree since it might not exactly be available in certain scenarios as discussed.

[bruno-f-cruz]

Can you point me to Step 4 for both code paths? This is the thing that I am struggling with being "low cost". I am probably missing something very obvious, but at some point you gotta have a signature:

Let T be the payload type for the register; you will need:

decode_single(struct_or_native_type: bytes) -> T
decode_from_numpy(massive_blob: bytes) -> nd.array[T]

I still don't see how these are 0-cost. But I may be missing a common data representation between the python native types and np.dtypes. For instance, if T is a datastruct, how do you vectorize its parsing?

--- edit

Ok I found it: https://github.com/fchampalimaud/generated-pyharp/blob/5f8577f2ebdba044f88401c5d322593e0a42c8c1/src/harp-data/harp/data/harp_data.py#L220

This is the code fork I was trying to avoid maintaining. I think your argument is that it is small enough that it won't be an issue (?). Still, you are pushing complexity into this RegisterDump class to know how to parse EVERY single type (string, interface_type is bool, masks, struct field, etc...). Maybe I should have made this point a bit more clear, but the architecture of the current PR is that this conversion should be instead handled by the underlying Converter (

pyharp/src/packages/harp-protocol/src/harp/protocol/_payload_converters.py

Lines 25 to 35 in d7ed4b7

	@abstractmethod
	def decode_scalar(self, view: np.generic) -> T:
	"""Decode a 0-D structured-array element into a Python value."""
	@abstractmethod
	def decode_batch(self, view: "NDArray[np.generic]") -> Any:
	"""Decode a 1-D structured-array column into an array-like."""
	@abstractmethod
	def encode_into(self, view: NDArray[np.generic], value: T) -> None:
	"""Write a Python value back into a structured-array element."""

) or Field-like attribute instead (

pyharp/src/packages/harp-protocol/src/harp/protocol/_payload.py

Lines 349 to 356 in d7ed4b7

	def _to_batch(self) -> "_BitMaskBatch[F]":
	"""Returns the metadata for the corresponding batch type"""
	return _BitMaskBatch(
	self._mask,
	self._enum,
	slot=self._slot,
	dtype=self._dtype,
	)

). This way adding a new type does not require changing an internal function that must be aware of all other types.

So I think there are a few issues here:

1. Where does the parsing logic live (in the converter vs in the "reader-like" object)
2. You are still duplicating struct/np.dtype logic for the attributes. If we consider this to be a finite number, sure, I think what you are saying may make sense: we pay the cost once, and we don't need to "maintain it". It kinda worries me how the same code base is already forked in some places.
3. What happens if people want to add custom types on their side? How can they change this parser directly? In the current case, the converters own the conversion logic; people can override it as they wish. Here, they are stuck with what we decide to implement.

Anyway, for the sake of moving this forward, I think we should just make the decision of what we are willing to support with the current team/effort and just go with it. As @glopesdev mentioned, we can hopefully always go back and remake it if necessary in a backwards compatible way with the public DSL.