CRT topology change

[denizergonul]

20.03 Updates:

CRTReader is not a DataReader. It's not an interface to read DAQ frames from a front-end (like DPDKReader) and send them to data handlers. Instead, it builds DAQ frames from raw electronics output. It's similar to Hermes FW, a functional block that prepares data and passes it on.
The new name, CRTFrameBuilder, eliminates confusion with standard DataReader modules and accurately reflects its role as a builder of DAQ frames rather than a reader.

• CRT frame builders communicate with SocketWriter via IOM.
• For each connection, there are, per DataSender:
  • 1 dedicated D2D connection (with this sender only)
  • 1 CRT frame builder
  • 1 SocketWriter
  • 1 IOM

---

appmodel PR dte/crt_topology_change
daqsystemtest PR dte/crt_topology_change
crtmodules PR dte/crt_topology_change
asiolibs PR dte/crt_topology_change
daqconf PR dte/crt_cb_update
datahandlinglibs PR dte/cb_acquire
fdreadoutmodules PR dte/crt_rates

Prior to this development, socket connection info resided in reader and writer configurations. Now, there is SocketDetectorToDaqConnection which has SocketDataSenders and SocketReceiver; SocketDataSender has the info socket type (TCP/UDP), local port and remote port; plus we make use of their NetworkInterface.ip_address to interpret them as local IP and remote IP.

CRT readers communicate with socket writers via IOM.
Socket readers communicate with DHLs via callbacks.

To try out:

• drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-socket-1x1-config ${USER}-local-test
• generate_modules_test local-socket-1x1-config crt-data-source-01 config/daqsystemtest/example-configs.data.xml
• generate_modules_test local-socket-1x1-config socket-ru-01 config/daqsystemtest/example-configs.data.xml

---

Type of change

• New feature or enhancement (non-breaking change which adds functionality)

Testing checklist

• Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas
• Code style is correct (dbt-build --lint, and/or see https://dune-daq-sw.readthedocs.io/en/latest/packages/styleguide/)

[alessandrothea]

• Move Socket* classes outside wiec schema file.
• Motivate the use of callbacks outside readout.

[alessandrothea]

Why is the SocketDataSender in the wiec.schema file?
Am I wrong or the sender It has nothing to do with wiec and TPC cold electronics?

[alessandrothea]

Callbacks, at this stage, are an internal Readout interface. Its use at the interface with detector code needs to be motivated, since performace is typically not an issue at the transmitter.
Switch back to queues if not soundly motivated.

[eflumerf]

The model used when separating the callback configuration from the iomanager configuration was that DataReaderModules should use callbacks to communicate with DataHandlerModules. CRTReaderModule is a DataReaderModule, and the symmetry argument is that the SocketWriterModule is taking the place of the DataHandlerModule in the CRTReaderApplication. Using a callback mechanism here also makes sense for the many-to-one communication case, and follows the same pattern as the DRM->DHM communication in the ReadoutApplication.

[roland-sipos]

I'm not sure if all of you are aware, but it was discussed at many occasions in the past:
Callbacks don't work on fan-in mode! That's why we don't use them for the TPs. In the context of the callee every members should be locked and protected against concurrent access.

Was there a demonstrator or simple test app available to demonstrate that callbacks will work for the CRTReaderModule between the data producers and the SocketWriter?

Seems like this slipped under the radar...

[roland-sipos]

I'd also argue, that CRTReaderModule is not a DataReaderModule. The SocketReaderModule is, to push data towards the DLHs through sockets (fan-out).

We don't have an appropriate model or classes to represent the CRTReaderModule. Again, these are a new type. The CRTReaderModule is something similar that the Hermes firmware block which is a hardware implementation of a principle or concept, that we do not have in the current configuration model. But for sure, it's not a DataReaderModule. I'm not arguing here about the chosen -unfortunate- name of "CRTReaderModule".

[denizergonul]

Summary of where we are at:
We'll follow up with a new issue for the next release to switch to IOMs for the communication between CRT readers and socket writers. For this release, we'll stick to callbacks. We'll switch to IOMs for the communication between CRT readers and socket writers. Keeping the text below for reference.

I still want to share the current flow (that I already shared with some of you in another chat) because it may be very similar to what we would also do with IOMs/consumer threads. (This is to be confirmed, maybe we'll follow a different approach instead, like using MPMC queues.)

Here's the example flow.

1. SocketWriter creates 2 Writers, registers 4 Callbacks (Callback1007, Callback1008, Callback1009, Callback1010).
    Callback1007 will call Writer1.enqueue()
    Callback1008 will call Writer1.enqueue()
    Callback1009 will call Writer2.enqueue()
    Callback1010 will call Writer2.enqueue()
2. CRTReaderModule acquires Callback1007, Callback1008, Callback1009, Callback1010. It creates packets for all.
    2.1. Callback1007 (calls Writer1.enqueue())
    2.2. Callback1008 (calls Writer1.enqueue())
    2.3. Callback1009 (calls Writer2.enqueue())
    2.4. Callback1010 (calls Writer2.enqueue())

enqueue() pushes a packet to the internal std::queue of Writer. if the queue was empty, it spawns a coroutine (start()).
until the queue is empty, start() fires async_write with the front element; after the write is completed, it pops the element.

as there's only 1 thread, the only synchronization needed here is due to async_write. for a case like this:

1. CRTReader calls Callback1007 (Writer1.enqueue()).
2. enqueue() pushes the packet to the internal queue. the queue was empty, so it spawns a coroutine (start()).
3. Coroutine: the queue is not empty, it fires async_write with the front element.
4. while async_write is ongoing, CRTReader calls Callback1008 (Writer1.enqueue()).
5. enqueue() pushes the packet to the internal queue. the queue was not empty, so it doesn't spawn a new coroutine. it returns.
6. Coroutine: the write is completed, it pops the element.
7. Coroutine: the queue is still not empty (a new element was pushed), so it wants to fire an async_write with the front element.
8. let's say, this time, CRTReader calls Callback1007 (Writer1.enqueue()) before the coroutine fires the async write.
9. this enqueue() will have to wait until the coroutine releases the strand (which happens either when it suspends/async_write or completes). this is how strand works, it synchronizes the coroutine and the lambda function inside enqueue().

• strand acts like a mutex for the std::queue.
• std::queue is needed to prevent firing simultaneous async_writes on the same socket.

[alessandrothea]

See previous comment about callbacks.

[alessandrothea]

LGTM

[bieryAtFnal]

Closing this PR since the branch has been merged to the prep-release/fddaq-v5.6.0 branch in a different PR.