Create an architecture for open source data exchange

[Deena-B]

Aim

Design a straight forward, low barrier to entry, solution for the movement of data between open source collaborators of the DCL project.

Background

We have a number of different systems that we are trying to make work together

• Individual's laptops
• UCI hpc
• Serge's GPU
• Google Drive

Data sizes vary. We want to be able to access and share data from 10s, 100s, and 1000s of files that can be as small as 20KB to as large as 10GB. Larger files can be processed on the HPC. We want a system to share small and medium sized files ASAP, which we can build on as our analyses get more data hungry.

UCI HPC related docs

1. How to transfer large amounts of data
2. OIT Transfering data
   1. see examples
3. UCI Cloud File Storage Options
   1. Google drive
      1. setup for UCI staff
      2. make folders public
4. UCI Webfiles
   1. not really what I want
   2. How to set up Private and Public Directories in WebFiles v7: UCI tutorial
   3. upload files UCI tutorial

Questions?

Please put questions related to this issue in this issue thread.

If you want a quick response, post a link to your comment in this thread to Slack #deepcelllineage and tag @deena or others. You can also DM @deena.

To join Slack enter your email address here. For questions NOT specifically related to this issue, get in touch through any of the communication methods listed in DCL's overview README.