Skip to content

rust(feat): parquet scpr/mcpr imports#584

Open
Brandon-Shippy wants to merge 21 commits into
mainfrom
bs/rust/parquet-scpr-imports
Open

rust(feat): parquet scpr/mcpr imports#584
Brandon-Shippy wants to merge 21 commits into
mainfrom
bs/rust/parquet-scpr-imports

Conversation

@Brandon-Shippy
Copy link
Copy Markdown
Contributor

@Brandon-Shippy Brandon-Shippy commented May 22, 2026

Description

🎫 ENG-11477 🎫

  • Adding parquet cpr import functionality, both single and multi.
  • moved arrow schema to dependencies from dev dependencies in order to show preview of multi schema we have to actually read the columns and the footer metadata isnt enough
  • Currently requires --time-path and --time-format, next PR will go through parquet imports to allow the cli to infer these
  • Condenses both flat dataset args and scpr args to use the common import clap args
  • Single cpr imports has --data-type, can use --data-type to set specific data type if need be, otherwise omitting it just uses the inferred type via metadata. Note: can also explicitly define --data-type infer. We need this because we need some value for ChannelDataType.
  • multi requires user to define the time path and data path and name path so we can properly map
  • single requires user to define the time path and data path as well as input a value for channel name, so we can fill in the name of the channel in sift

cpr single import
Screenshot 2026-05-22 at 2 52 20 PM

cpr multi import
Screenshot 2026-05-22 at 2 52 34 PM

Verification

cargo r -p sift_cli -- import parquet cpr -h yields helpful help page

cargo r -p sift_cli -- import parquet cpr --asset <> --mode <multi or single> --time-format <> --time-path <> --data-path <> --name-path <> PATH --preview properly shows preview of both multi and single.

Unit tests for cpr import

Manual imports of cpr parquet files

@Brandon-Shippy Brandon-Shippy changed the title Bs/rust/parquet scpr imports rust(feat): parquet scpr imports May 22, 2026
@Brandon-Shippy Brandon-Shippy marked this pull request as ready for review May 22, 2026 21:31
@Brandon-Shippy Brandon-Shippy changed the title rust(feat): parquet scpr imports rust(feat): parquet scpr/mcpr imports May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant