Specify data guarantees for live data streaming

[llucax]

What's needed?

The live data streaming RPCs are using the frequenz.api.common.v1.microgrid.components.ComponentData message as the response, but this message is very flexible to allow streaming data efficiently in many different use cases (like streaming historical data).

Supporting all the provided flexibility is difficult for clients, so if there are some data restriction guarantees, these should be properly documented in the RPC response documentation so clients don't need to include unnecessary extra complexity.

Proposed solution

Document the ReceiveComponentDataStreamResponse.data field to mention that for a single response for a live data message:

• All the present data.metric_samples and data.states will have the same sampled_at timestamp.
• Duplicate metrics can only appear in the data.metric_samples if every metric have a different connection.name (so, connection should be present, and the name of the connection should not appear duplicated for a particular Metric).
• When a metric appears only once, the connection information MAY be present. When a metric appears multiple times, all instances MUST have the connection information filled in.
• There will be at most one data.states.¹

The same restriction should documented for ReceiveSensorDataStreamResponse.data, adapting to the particular fields of that response.

Use cases

Live data streaming client implementation.

Footnotes

1. @tiyash-basu-frequenz could there be none? Or is it guaranteed to have at least one? In any case I don't think it makes a huge difference for the client implementation if it can be 0 or 1 or if is guaranteed to have 1, so we can leave it open. ↩

[shsms]

• There will be at most one data.states.

But there will also be a relay state for batteries, right? Or would that come in a separate message?

[llucax]

My understanding is no. All states come in one ComponentState message, relay states are also ComponentStateCode which comes as a repeated field. But @tiyash-basu-frequenz should probably confirm or deny.

[shsms]

Yes, so there won't be "at most one" state, right? There will be two ComponentStateCode values in the repeated field.

[llucax]

Yeah, ComponentState != ComponentStateCode, so one ComponentState that contains many ComponentStateCodes.

But, I just talked to @tiyash-basu-frequenz and apparently it is not true, as relays and batteries can report data separate so have different timestamps, so maybe we'll have 2 ComponentState per streaming message for batteries, as the sampled_at timestamp is attached to one ComponentState, we would need one for the battery state and one for the relay state.

Anyway Tiyash said he'll clarify.

[shsms]

Oh I see, there's two levels of nesting.

[tiyash-basu-frequenz]

At the server side, we will try our best to have each data sample as "complete" as possible, and return data for a single timestamp (with a slight catch, read below).

It means:

• All available metrics and states will be available in each data sample, provided we received them from the component hardware. E.g., battery data will contain all DC electricity samples, SoC, temperature, and all its states (overall component state and the relay state). On a side note, for all downstream client activities, component state should be all that is needed, since it also encapsulates the effect of the relay state.
• Each data sample will have metrics from a single timestamp. Again in a side note, since this results in a lot of closely spaced duplicate samples in the data stream, in future we may change this slightly (on the server implementation side) to have less duplicates. This might result in a delta of 10s of milliseconds between individual metrics. The delta might arise depending upon how the data is received from the underlying component hardware. We will still continue having all available metrics in a single message.

Hope this addresses your questions. If you have further questions, please feel free to add them below :)

[llucax]

There will be no duplicate metrics in the data.metric_samples.

This is false, we can have the same metric as long as we have multiple connections. But we should add a guarantee that given multiple connections, the connection.name should be unique for a particular metric.