Write DynaCLR predictions before PHATE computations

[Soorya19Pradeep]

The most time-consuming part of DynaCLR prediction is the PHATE computation for very large datasets, like 20X multiwell data. The prediction runs in half the time required for PHATE computation. In case the job is interrupted, no output is received, as with the current implementation, predictions are written after finishing the PHATE computations. Should we start writing the output zar before the PHATE computation so that it doesn't interfere with result generation, and then add the PHATE and other observations later, after the PHATE computation step completes? Also, is there any way to reduce PHATE computation time for large datasets?
@edyoshikun , @srivarra , thoughts?

[edyoshikun]

Yeah we can compute this as a separate step.

The priority actually is to write efficiently. Currently, we hold all the predictions in memory and then write. This does not work with the OPS datasets, so I had to do something similar to create_empty_plate(), to create an empty thing and then store directly by slicing.

[edyoshikun]

this is flagged in #362

[srivarra]

@Soorya19Pradeep

Also, is there any way to reduce PHATE computation time for large datasets?

Besides throwing more cores at it probably not, unless we implement it with CUDA support via Nvidia Rapids / CuML.