[SYNPY-1809] Added generate_sync_manifest method

[andrewelamb]

Problem:

The generate_sync_manifest method in sync.py worked with the old-style manifest.

Solution:

A new version was created as a StorableContainer method that creates a new-style manifest.
The old version was deprecated.

Testing:

Integration tests for the new method
Unit tests for the various helper functions

[BryanFauble]

I have a concern around concurrency, otherwise the rest looks good to me

[linglp]

Hi @andrewelamb ! I think overall it looks good.

While testing this function, I noticed that local files are always re-uploaded to Synapse even when unchanged based on the log. This may not be ideal for users with large manifests who only want to re-upload a subset. For example, users who have problems with only a subset of files might only want to reupload those and ignore all the others. I am curious if the current implementation would let us skip files that have the same MD5 checksums? If not, is it possible to implement a different solution that let us pull the files from Synapse first -> compare MD5 checksums -> upload if MD5 checksums are different

[andrewelamb]

Hi @andrewelamb ! I think overall it looks good.

While testing this function, I noticed that local files are always re-uploaded to Synapse even when unchanged based on the log. This may not be ideal for users with large manifests who only want to re-upload a subset. For example, users who have problems with only a subset of files might only want to reupload those and ignore all the others. I am curious if the current implementation would let us skip files that have the same MD5 checksums? If not, is it possible to implement a different solution that let us pull the files from Synapse first -> compare MD5 checksums -> upload if MD5 checksums are different

Thanks for the suggestion! I think this is out of scope for this ticket as I'm just replicating the old function, but using the new manifest style. I'll create a ticket though!