Description
Once MARCXML files and matching log files are generated, they need to be uploaded back to CERNBox. This issue covers implementing an upload step that takes the output directory (or temp directory from the XML generation step) and pushes all relevant files to a given CERNBox URL using credentials from environment variables (if needed).
Work involved
- Implement an
upload_to_cernbox(local_path: str, cernbox_url: str) function (reusing the same CERNBox access method established in the fetch issue)
- Upload all generated XML files (per-Boite + combined) and the matching log file
- Read credentials from environment variables (same pattern as the fetch step)
- Log each uploaded file and raise a clear error on failure
Acceptance criteria
- All XML files and log files produced by the pipeline are uploaded to the specified CERNBox URL
- Credentials are read from environment variables
- Failed uploads raise a clear error with the affected file name
- End-to-end pipeline (fetch Boite files → match → generate XML → upload)
Screenshots(Optional)
Description
Once MARCXML files and matching log files are generated, they need to be uploaded back to CERNBox. This issue covers implementing an upload step that takes the output directory (or temp directory from the XML generation step) and pushes all relevant files to a given CERNBox URL using credentials from environment variables (if needed).
Work involved
upload_to_cernbox(local_path: str, cernbox_url: str)function (reusing the same CERNBox access method established in the fetch issue)Acceptance criteria
Screenshots(Optional)