Skip to content

chore: adding new migrations (and s3 copy helper function)#177

Merged
ealexa05 merged 2 commits into
mainfrom
move-migration-1
Jun 9, 2026
Merged

chore: adding new migrations (and s3 copy helper function)#177
ealexa05 merged 2 commits into
mainfrom
move-migration-1

Conversation

@ealexa05

@ealexa05 ealexa05 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This starts our convoluted recovery scheme for Masabi data with minimal disruption for analysts:

  • run this migration, to copy 1-year data over to a new MASABI_TEMP location in case the new fix doesn't work as intended
  • apply the fix in Fix masabi parquet rollover error #175
  • wait to verify behavior
  • run another move migration to copy completed 1-year data over to copy over to MASABI_TEMP
  • put in the schema change PR (Add masabi temporary schema #176; beta instance will be saturated, but duck_db.py will run at some point because it is running on alpha)
  • Put in a new delete migration to delete 1-year data in MASABI_DATA and kick off ingestion of 5 year data

@ealexa05 ealexa05 merged commit 8c0379d into main Jun 9, 2026
5 checks passed
@ealexa05 ealexa05 deleted the move-migration-1 branch June 9, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants