Part of the initial idea behind Ether was to dynamically communicate between instances and process for multiple forms of data. So far we have implemented small message passing/function calling with ZMQ, but we also want to handle larger data in an efficient way. I'm pretty sure the path to doing this is by using Pyarrow/Apache Arrow and no-copy IPC between processes.
BONUS: This could help unlock integration with Hugginface datasets and other users of Arrow.
Part of the initial idea behind Ether was to dynamically communicate between instances and process for multiple forms of data. So far we have implemented small message passing/function calling with ZMQ, but we also want to handle larger data in an efficient way. I'm pretty sure the path to doing this is by using Pyarrow/Apache Arrow and no-copy IPC between processes.
BONUS: This could help unlock integration with Hugginface
datasetsand other users of Arrow.