Currently, the original ecal code for handling UDP streams is very brittle, and there's no packet recovery or handling at all. It completely depends on the UDP to transmit packets in the correct order (rightfully so, though we can do some recovery to improve drops)
Currently, we are dropping close to 50-80% of packets on the publisher side due to how UDP publishing is handled in the original ecal code. TLDR: Sending is synchronous, as they want to ensure that images are published on all layers before returning, this breaks some of our driver implementation because the Send takes too long. To circumvent this, our current solution is to send the image on another thread, effectively making it async. (Improvements: The current implementation starts a new thread for each send, which i can forsee become expensive eventually, so preferably maintain a single thread. Threadpool cannot work since we want the images to be sent in order, and we are okay with dropping images)
On the receiver end, we are dropping another 50%-ish of the remainder over UDP, so original 20hz becomes 1-5hz depending on how good the wifi is. These are for full images, not compressed or thumbnail.
Benchmarking
For benchmarking use the profiling branch, which intercepts the packets and counts losses and other stuff. Build this with cpack and install, and if remember to install the python wheel as well.
Current problems
There's a current issue with ecal original code handles the packets, since there's zero tolerance for out of order packets, a ReceiveSlot can remain in rcm_reading mode for some reason. Take a look at the image above for example. My guess is that there are some scenarios that we get the content before the header itself, and even if the we get all the packets, only the packets received after the header are used, so it might not go into the rcm_aborted branch. The root cause of this is unknown as of now.
Currently, the original ecal code for handling UDP streams is very brittle, and there's no packet recovery or handling at all. It completely depends on the UDP to transmit packets in the correct order (rightfully so, though we can do some recovery to improve drops)
Currently, we are dropping close to 50-80% of packets on the publisher side due to how UDP publishing is handled in the original ecal code. TLDR: Sending is synchronous, as they want to ensure that images are published on all layers before returning, this breaks some of our driver implementation because the Send takes too long. To circumvent this, our current solution is to send the image on another thread, effectively making it async. (Improvements: The current implementation starts a new thread for each send, which i can forsee become expensive eventually, so preferably maintain a single thread. Threadpool cannot work since we want the images to be sent in order, and we are okay with dropping images)
On the receiver end, we are dropping another 50%-ish of the remainder over UDP, so original 20hz becomes 1-5hz depending on how good the wifi is. These are for full images, not compressed or thumbnail.
Benchmarking
For benchmarking use the profiling branch, which intercepts the packets and counts losses and other stuff. Build this with cpack and install, and if remember to install the python wheel as well.
Current problems
There's a current issue with ecal original code handles the packets, since there's zero tolerance for out of order packets, a ReceiveSlot can remain in rcm_reading mode for some reason. Take a look at the image above for example. My guess is that there are some scenarios that we get the content before the header itself, and even if the we get all the packets, only the packets received after the header are used, so it might not go into the rcm_aborted branch. The root cause of this is unknown as of now.