Skip to content

Add download_dataset function (and one do Do It Live!)#49

Open
jashapiro wants to merge 16 commits into
jashapiro/process-and-statusfrom
jashapiro/download-dataset
Open

Add download_dataset function (and one do Do It Live!)#49
jashapiro wants to merge 16 commits into
jashapiro/process-and-statusfrom
jashapiro/download-dataset

Conversation

@jashapiro
Copy link
Copy Markdown
Member

Closes #35

The main addition here is the download_dataset() function, which really just does what it says and should behave more or less like the download project function, but for taking a dataset... The main extra is that it has a check to be sure the dataset is actually ready before trying to download it.

I also added handling of expired datasets, since that is a thing that can happen.

Finally, I also made a function that waits until a dataset is ready and then downloads. I wasn't sure if this should really be a separate function, but the additional option made me think that it probably should be... But I could reconsider and roll them together, with a separate internal function that just does the status polling. Let me know what you think!

The biggest challenge here was deciding how often to poll and how long to wait: For now I set polling to every 30 seconds: In my testing, processing for small datasets takes somewhere between 30 sec and 1 min, so faster polling didn't seem worth doing. But I did leave it as an option in case people really want to poll at a different rate. I also got to play around a bit with cli to make a spinner while waiting. The elapsed time you get with cli is a bit disappointing in that you can't control rounding, unless you want to do all the seconds to minutes translations yourself, so I just left it as the default.

I'm also now allowing downloads without automatic unzipping, when ends up in a bit a strange place for the arg because I didn't want to break position-based calls any more than already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant