File index tables

[ialarmedalien]

Several importers generate file manifests or lists of files with file location and metadata. When data is promoted into the ref data client, these file lists should be imported and appended to an existing index table so that there's a central location for tracking where files are.

There are currently a couple of different sources and formats for these file listings:

• frictionless data package / data resource manifests, generated by DTS and pipelines like the NCBI FTP updater;
• JSON or parquet tables, as generated by the AllTheBacteria importers

Additionally, there are several pipelines that break up large files into small chunks for ease of parsing (e.g. xml_file_splitter with the uniprot and uniref XML files), and having an index of where these files are would be handy.

To do:

• ensure all tables use the same schema
• add in credit metadata if it does not exist
• update file paths when files are moved to the RefData tenant
• write file mover script to be run on jupyterhub

File mover script:

• if no file index exists, create one using the frictionless package
• move files from CTS bucket to appropriate path in the RefData tenant
• verify file checksum after move (could be done with frictionless or using s3.list_objects for the directory)
• update file index to reflect new path
• append file index to existing index table (name / schema TBD) -- write as Iceberg table, update Hive metastore