Skip to content

fix download mirrors#6

Draft
mtran-code wants to merge 1 commit into
mainfrom
fix/download-mirror
Draft

fix download mirrors#6
mtran-code wants to merge 1 commit into
mainfrom
fix/download-mirror

Conversation

@mtran-code

Copy link
Copy Markdown
Collaborator

Summary

  • Replace legacy CCLE 2019 DepMap portal download links with reachable Broad static mirror URLs.
  • Add a shared workflow/scripts/download_resource.py helper that downloads or copies resources atomically, rejects HTML verification/error pages, and validates gzip/zip content where expected.
  • Route download rules through the helper so gated DepMap pages fail loudly instead of being saved as raw CSV/GCT inputs.

Root cause

The DepMap portal download endpoints can return an HTML verification page with HTTP 200. Rules using raw curl -L could either fail later with gunzip: unknown compression format or silently save HTML into raw data outputs.

Validation

  • uvx ruff format workflow/scripts/download_resource.py
  • uv run python -m py_compile workflow/scripts/download_resource.py
  • Validated 17 updated https://data.broadinstitute.org/ccle/... mirror URLs with curl -fsSI; all returned reachable non-HTML responses.
  • Real helper smoke tests:
    • downloaded Cell_lines_annotations_20181226.txt through the helper.
    • downloaded and decompressed gencode.v19.genes.v7_model.patched_contigs.gtf.gz through the helper.
    • confirmed a DepMap verification-page URL is rejected by the helper.
  • uvx --python 3.11 --from snakemake==7.32.4 --with pulp==2.7.0 snakemake --dryrun --cores 1
  • uvx --python 3.11 --from snakemake==7.32.4 --with pulp==2.7.0 snakemake --cores 1 -F metadata/Cell_lines_annotations_20181226.txt rawdata/epigenetics/methylation/enhancer_cpg_clusters.tsv
  • git diff --check

Notes

The remaining DepMap portal URLs are current/public-release files without confirmed static mirrors in this pass: OmicsSomaticMutations.csv, OmicsCNGeneWGS.csv, OmicsCNSegmentsWGS.csv, and CCLE_miRNA_MIMAT.csv. These now fail with an explicit validation error if the portal returns a verification page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant