Skip to content

feat: add flag for pending uploads#12

Merged
jguo144 merged 2 commits into
masterfrom
jguo144/2026-04-17/add-pending-flag
Apr 28, 2026
Merged

feat: add flag for pending uploads#12
jguo144 merged 2 commits into
masterfrom
jguo144/2026-04-17/add-pending-flag

Conversation

@jguo144
Copy link
Copy Markdown
Collaborator

@jguo144 jguo144 commented Apr 27, 2026

Description

Set cloudstorage_multipart_pending whenever a multipart upload starts so downstream hooks know the file is not committed yet. Clear the flag with resource_patch after finish_multipart and abort_multipart so cancelled or finished uploads do not leave resources stranded.

Copy link
Copy Markdown

@orca-security-us orca-security-us Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

return aborted


def clean_multipart(context, data_dict):
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't really in the scope of this change, but I think it may be worth taking a look and addressing:

clean_multipart (multipart.py:364) deletes expired uploads but never calls resource_patch to clear the pending flag. A resource whose multipart upload expires silently will be stuck with cloudstorage_multipart_pending="True" indefinitely — effectively blocking xloader forever. This needs the same flag-clearing logic added to clean_multipart.

Copy link
Copy Markdown
Collaborator Author

@jguo144 jguo144 Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not a fan of calling resource_patch in the clean_multipart API as it may operate across multiple resources and unnecessarily update their “last modified” timestamps. It may trigger side effects for something that’s really for cleanup.
Instead I added an ignore_pending argument in xloader, similar to the existing ignore_hash option.

Comment thread ckanext/cloudstorage/logic/action/multipart.py Outdated
Comment thread ckanext/cloudstorage/logic/action/multipart.py
@jguo144 jguo144 force-pushed the jguo144/2026-04-17/add-pending-flag branch from 5305e42 to 0cd79bf Compare April 28, 2026 20:21
@jguo144 jguo144 merged commit ea663af into master Apr 28, 2026
11 checks passed
@jguo144 jguo144 deleted the jguo144/2026-04-17/add-pending-flag branch April 28, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants