buildpack s3 cleanup #1403
buildpack s3 cleanup #1403
Conversation
stephanme
left a comment
stephanme
left a comment
There was a problem hiding this comment.
Will older buildpack releases be deployable after the restructuring?
E.g. when I want to deploy cf-deployment v46.3.0 from Jan 2025 using javabuildpack v4.71.0 - will this work after the old blobs have been deleted (after phase 4 : cleanup)?
| 2. **Orphan Detection:** No automated way to identify unused blobs when `blobs.yml` diverges from S3 | ||
| 3. **Audit Challenges:** Cannot identify file types, owners, or purposes without external mappings | ||
| 4. **Cost Inefficiency:** Potentially storing obsolete blobs (estimated 30-40% orphan rate in some analyses) | ||
| 5. **Multi-Team Collision:** Multiple buildpack teams share the same flat namespace, increasing collision risk and confusion |
There was a problem hiding this comment.
The technical separation makes sense. But I don't think that there is a multi-team collision. All buildpacks are handled by the same team / WG area "Buildpacks and Stacks": https://github.com/cloudfoundry/community/blob/main/toc/working-groups/app-runtime-interfaces.md?plain=1#L75
There was a problem hiding this comment.
only creating a bosh release from a tag will create a issue. but still possible with minimal change.
when checking out a tag. the user should only add a folder_name: xxx in the config/final.yml
There was a problem hiding this comment.
Just adding more context. The final release tarballs are available on bosh.io (e.g. https://bosh.io/releases/github.com/cloudfoundry/java-buildpack-release?all=1) uploaded by this task, that is why they are not impacted. If you want to build from source for an older tag will require the workaround @ramonskie mentioned above.
|
As long as the older releases are still supported, I like option 1. |
beyhan
left a comment
beyhan
left a comment
There was a problem hiding this comment.
During the TOC meeting we discussed this and the feedback is
- Define a retention period for the orphaned blobs
- Option 1 has been selected
|
Given that there don't seem to be any open issues with this RFC and work for this is progressing at great speed, the TOC decided to move this to the final comment period. You now have one week to leave any comments or forever hold your peace. |
beyhan
left a comment
beyhan
left a comment
There was a problem hiding this comment.
Hi @ramonskie,
we looked into this during the TOC and would like to have:
- the option 2 removed completely
- the open question at the end don't belong to the RFC
- the ARI WG is owning the migration
- the cost savings are also not needed
- Release candidates bucket separation should be included in the RFC and implemented
Co-authored-by: Beyhan Veli <beyhan.veli@sap.com>
4 participants
This PR adds the RFC "buildpacks s3 bucket cleanup".
For easier viewing, you can see the full RFC as preview.