feat: add flag for pending uploads#12
Conversation
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
| return aborted | ||
|
|
||
|
|
||
| def clean_multipart(context, data_dict): |
There was a problem hiding this comment.
This wasn't really in the scope of this change, but I think it may be worth taking a look and addressing:
clean_multipart (multipart.py:364) deletes expired uploads but never calls resource_patch to clear the pending flag. A resource whose multipart upload expires silently will be stuck with cloudstorage_multipart_pending="True" indefinitely — effectively blocking xloader forever. This needs the same flag-clearing logic added to clean_multipart.
There was a problem hiding this comment.
I’m not a fan of calling resource_patch in the clean_multipart API as it may operate across multiple resources and unnecessarily update their “last modified” timestamps. It may trigger side effects for something that’s really for cleanup.
Instead I added an ignore_pending argument in xloader, similar to the existing ignore_hash option.
5305e42 to
0cd79bf
Compare
Description
Set
cloudstorage_multipart_pendingwhenever a multipart upload starts so downstream hooks know the file is not committed yet. Clear the flag withresource_patchafterfinish_multipartandabort_multipartso cancelled or finished uploads do not leave resources stranded.