[Question] faster file transfer?

[zatkins2]

Thoughts on speeding up file transfer? I observed around 35MB/s from the tiger headnode when uploading files.

I guess a brute-force thing would be to break up a batch of files over different processes (embarrassing parallelization) but idk if either the cluster or hippo bandwidth would eventually become saturated.

I wonder if there is some way to “stage” a batch of products and such and then use gridftp like globus under the hood? I usually find globus is about 10x faster than this was. For large file batches (TBs) I think that speed is necessary.

[JBorrow]

We could parallelize the upload as it is split into chunks. At some level the speed of uploads is limited by ingress into NERSC via s3, which should improve over time as we move to N10. We will not use something like gridftp or globus under the hood, that would cause unnecessary complexity.

The function that actually uploads files is here:

hippo/hippoclient/product.py

Line 24 in b0f24d6

def __upload_sources(

. Either we could try parallelizing this, or upload multiple products in parallel which may be easier.

[zatkins2]

Ok! What do you mean by parallelizing the function vs. uploading in parallel?

[JBorrow]

The way that upload function works is by uploading chunks of up to 50 MB of the file individually before asking s3 to 'stitch' them together. So you could upload more than one chunk at once, for instance.

The alternative would be to parallelize over products when you upload an entire collection at once.