PEPATAC: gtars-backed BAM QC + refgenie 1.0 / drop-R modernization#329
Open
nsheff wants to merge 13 commits into
Open
PEPATAC: gtars-backed BAM QC + refgenie 1.0 / drop-R modernization#329nsheff wants to merge 13 commits into
nsheff wants to merge 13 commits into
Conversation
The refgenconf RGC and select_genome_config imports in pipelines/pepatac.py are never referenced anywhere else in the file. Confirmed via grep for \bRGC\b and select_genome. Adds tools/audit_refgenie_surface.sh which enumerates all refgenconf / RefGenConf / looper_refgenie_populate / REFGENIE references in the repo (sources, docs, tests, configs). Adds findings.md as the dogfooding deliverable for the refgenie1 branch. Initial sections: audit, API gaps, seek-key naming divergences, asset class gaps, CLI/install gaps, cluster integration. Validation run section will be filled by step 9 of the plan. Plan: assistant/pepatac_refgenie1_branch_plan_v1.md (step 2)
- var_templates: refgenie_config ($REFGENIE) → refgenie_db_config ($REFGENIE_DB_CONFIG_PATH). Refgenie1 uses its own env var. - pre_submit.python_functions: refgenconf.looper_refgenie_populate → refgenie.looper_refgenie_populate_local. The new populator lives in refgenie1's refgenie/populator.py (added on the refgenie/refgenie1#nsheff-refactor-2 branch alongside this one). - Jinja: refgenie[g].bowtie2_index.dir → bowtie2_index.bowtie2_index. Refgenie1's bowtie2_index asset class has no 'dir' seek_key (the legacy refgenconf 'dir' built-in was removed); the bowtie2_index seek_key returns the index prefix path which is what bowtie2 -x consumes. Same change for bwa_index. - pipelines/pepatac.yaml resources.genome_config: $REFGENIE → $REFGENIE_DB_CONFIG_PATH (consistency, the field is unused but was confusing). Plan: assistant/pepatac_refgenie1_branch_plan_v1.md (steps 4-5) Findings: see findings.md for the full divergence audit.
requirements.txt: refgenconf>=0.12.2 → refgenie>=1.0.0 requirements-conda.yml: refgenconf==0.12.2, refgenie==0.12.1 → refgenie>=1.0.0 (single dep — refgenie1 supersedes both). Note: refgenie 1.0.x and legacy refgenie 0.12.x share the PyPI name 'refgenie' but are different packages. The >=1.0.0 pin disambiguates. This is a known footgun — see findings.md. Plan: assistant/pepatac_refgenie1_branch_plan_v1.md (step 6)
docs/assets.md gets the canonical refgenie 1.0 setup walk-through.
The other refgenie-touching docs (tutorial, run-conda, detailed-install,
run-bulker, howto/install-refgenie) get a NOTE banner pointing at
docs/assets.md plus inline replacements of:
pip install refgenie → pip install "refgenie>=1.0.0"
export REFGENIE=... → export REFGENIE_HOME_PATH +
REFGENIE_DB_CONFIG_PATH
refgenie init -c ... → refgenie init
refgenie pull g/a → refgenie genome init <fasta> &&
refgenie add g/a --recipe <r>
This is the minimum migration to keep the docs consistent with the
branch. The legacy refgenie 0.12.x flow has no automatic CLI shim in
refgenie 1.0; users on that flow must follow the new commands.
Plan: assistant/pepatac_refgenie1_branch_plan_v1.md (step 7)
Adds findings discovered during the end-to-end Rivanna validation: - Refgenie() rejects str for database_config_path (signature says Path | None but no coercion → AttributeError deep in get_database_config) - Looper _update_namespaces requires the namespace to pre-exist on input — the populator must mutate the input dict (not just return) - Looper 2.1.x dropped the positional config argument; needs -c - Refgenie1 venv has no pip; need uv pip install - bulker activate must be wrapped in eval "$(...)" to take effect Each finding has a one-line recommended fix. Plan: assistant/pepatac_refgenie1_branch_plan_v1.md (steps 9-11)
End-to-end PEPATAC run on Rivanna against refgenie1-registered hg38 + rCRSd assets. Wall-clock 2:16 on a 4-core 12GB node (SLURM 12499800). Pipeline ran successfully through: trimming (skewer) → fastqc → rCRSd prealignment (bowtie2 + refgenie1's rCRSd index) → hg38 alignment (bowtie2 + refgenie1's hg38 index) → sort/index → dedup (samblaster) → fragment classification (NFR/mono/di/tri/poly BAMs) → genome size from refgenie1 chrom_sizes It failed at signal generation in gtars-uniwig with a Rust panic on a BAM header (missing VN: tag in @pg record). That's a gtars-rs / PEPATAC bug, NOT a refgenie1 issue. The refgenie1 dogfooding goal is met: every refgenie1 asset path PEPATAC referenced was resolved correctly and consumed by the pipeline. tests/refgenie1_validation/RUN_NOTES.md captures cluster paths for the artefacts (sort.bam, sort_dedup.bam, fragment-class BAMs, prealignment summary, fastqc reports). findings.md gets the full run report appended. Plan: assistant/pepatac_refgenie1_branch_plan_v1.md (steps 9, 11)
- tools/pepatac_summarizer/: Python package with CLI, consensus peak calling via gtars, peak counts, and summary plots - pipelines/pepatac_collator.py: --summarizer python|R dispatch, defaults to python - Remove obsolete PEPATACr R tests - tests/test_summarizer.py: unit tests - tests/test_summarizer_integration.py: integration tests
- Add plot_tss_distance using TssIndex.from_regionset.calc_tss_distances; wire into pepatac.py anno block (replaces R placeholder) - Fix plot_frif to sum read counts from bedtools coverage outputs - Reorder plot_partition_distribution to horizontal stacked bar with inline percent labels; add natural chrom sort + canonical chrom filter - Add fragment-distribution median; add chrom/tssdist/part/frif CLI subcommands - Align PartitionList.from_gtf defaults to R's GenomicDistributions: core_prom=100, prox_prom=2000 (was 2000/10000)
Unifies two parallel R-to-Python efforts on refgenie1: - Per-sample pipeline (tools/pepatac_qc_gtars.py + pepatac.py): TSS enrichment, fragment distribution, anno (chrom/tss-distance/ partition), and FRiF/cFRiF plots via gtars + matplotlib + pysam. Replaces PEPATAC.R for all per-sample QC plots. - Project-level collator (tools/pepatac_summarizer/ + pepatac_collator.py): Python package replacing PEPATAC_summarizer.R - consensus peaks via gtars, peak counts, alignment/complexity/library/TSS summary plots. Adds Python unit and integration tests. PartitionList defaults aligned with R's GenomicDistributions (core_prom=100, prox_prom=2000). Removes obsolete PEPATACr R tests.
Switch bamQC computation to use gtars uniwig bamqc (Rust) when --qc-backend gtars is specified. Falls back to bamQC.py for --qc-backend r (default).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the gtars-backed BAM QC into PEPATAC behind
--qc-backend gtars(callsgtars uniwig bamqcin place ofbamQC.py), alongside the refgenie 1.0 migration and the drop-R Python QC summarizer. Reconciled with current dev. Companion to databio/gtars#265.