From Kevin, relevant to using their EDD vizualization tool during automatic updating:
I did some testing on HDF5 readers while there are writers. There is a "supported" paradigm using Single-Writer-Multiple-Reader (SWMR). In this mode, both the writer and the reader(s) have to know thats what they are doing. The writer isn't allowed to create new groups / datasets, but they can modify existing ones.
In the case where the writer is just opening with "w" or "r+" and doing no synchronization management, then we are a bit out of luck in terms of data-integrity. The reader can never really assume that the data is in a valid state. In many cases, the reader will attempt to read and run into some flavor of PermissionError.
In my case, I am just going to retry on these kinds of expected errors and eventually I will be able to read some probably-not-corrupt data. To do this without interrupting the writer, however, you have to set an OS flag: "HDF5_USE_FILE_LOCKING = false".
From Kevin, relevant to using their EDD vizualization tool during automatic updating:
I did some testing on HDF5 readers while there are writers. There is a "supported" paradigm using Single-Writer-Multiple-Reader (SWMR). In this mode, both the writer and the reader(s) have to know thats what they are doing. The writer isn't allowed to create new groups / datasets, but they can modify existing ones.
In the case where the writer is just opening with "w" or "r+" and doing no synchronization management, then we are a bit out of luck in terms of data-integrity. The reader can never really assume that the data is in a valid state. In many cases, the reader will attempt to read and run into some flavor of PermissionError.
In my case, I am just going to retry on these kinds of expected errors and eventually I will be able to read some probably-not-corrupt data. To do this without interrupting the writer, however, you have to set an OS flag: "HDF5_USE_FILE_LOCKING = false".