ndtiff Datasets cannot be pickled

[ieivanov]

Dask arrays created by the ndtiff library cannot be pickled. Pickling is important for multiprocessing applications.

For example,

dataset = Dataset(\path\to\ndtiff\dataset)
data_dask_array = dataset.as_array()
pickle.dumps(data_dask_array)

first throws a AttributeError: Can't pickle local object 'NDTiffDataset.as_array.<locals>.read_one_image', which is fixed by #108. It then throws TypeError: cannot pickle '_thread.RLock' object, which I don't think I'll be able to fix myself.

A workaround may be to open a new dask array in every worker rather than pickling the opened dask array for distribution to workers.

[henrypinkard]

The lock is meant to handle synchronization on the the datasets index, so that you cant read data if the index is currently being updated. This is specifically for image saved callbacks. I forget what specific problem/bug it was solving, but there are probably better alternatives. In any case, are you trying to do this on an in progress dataset? If not, you could safely ignore that lock because the dataset is read only

[ieivanov]

No, here I am trying to work with a dataset that has finished acquiring. Could we encode a check if the acquisition is in progress and only use the lock then? I am not sure if pickle will be OK with that logic, we could test on a simpler example first

[henrypinkard]

Yes, if it works with pickle that should work