Skip to content

Command to generate pipeline container config files#3955

Merged
mashehu merged 40 commits intonf-core:devfrom
mirpedrol:pipeline-container-config-builder
Apr 24, 2026
Merged

Command to generate pipeline container config files#3955
mashehu merged 40 commits intonf-core:devfrom
mirpedrol:pipeline-container-config-builder

Conversation

@mirpedrol
Copy link
Copy Markdown
Member

@mirpedrol mirpedrol commented Dec 2, 2025 •
edited
Loading

close #3953

Done:

  • New function to run nextflow inspect and obtain docker config file
  • New function to (re)generate all container config files
  • Container config files are regenerated with nf-core modules install|update|remove

TODO:

  • Add tests
  • Test a lot
  • What happens if the module is not found?
    • Modules form multiple repos
    • Modules included as an alias
  • Is there a case where two modules have the same name but different container? how to solve this? We couldn't find an existing case of this
  • Could the code be used to reduced duplication with download command?

@mirpedrol mirpedrol force-pushed the pipeline-container-config-builder branch from 6341cd3 to b8004aa Compare December 2, 2025 14:43
mashehu
mashehu reviewed Dec 3, 2025
View reviewed changes
Comment thread nf_core/pipelines/containers_utils.py Outdated
mashehu
mashehu reviewed Dec 3, 2025
View reviewed changes
Comment thread nf_core/pipelines/containers_utils.py Outdated
@mirpedrol mirpedrol marked this pull request as ready for review December 4, 2025 14:39
mashehu
mashehu reviewed Dec 4, 2025
View reviewed changes
Comment thread nf_core/pipelines/containers_utils.py Outdated
Comment thread nf_core/pipelines/containers_utils.py Outdated
Comment thread nf_core/components/components_utils.py Outdated
Comment thread nf_core/pipelines/containers_utils.py
Comment thread nf_core/pipelines/containers_utils.py Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 15, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 93.71069% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.33%. Comparing base (be0a1b2) to head (200571d).
⚠️ Report is 1 commits behind head on dev.

Files with missing lines Patch % Lines
nf_core/pipelines/containers_utils.py 91.17% 9 Missing ⚠️
nf_core/components/patch.py 85.71% 1 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mirpedrol mirpedrol changed the title [WIP] Command to generate pipeline container config files Command to generate pipeline container config files Dec 16, 2025
mashehu
mashehu approved these changes Dec 16, 2025
View reviewed changes
Comment thread tests/pipelines/test_container_configs.py Outdated
@mashehu mashehu mentioned this pull request Dec 16, 2025
Merged
11 tasks
mashehu
mashehu approved these changes Jan 13, 2026
View reviewed changes
Comment thread nf_core/components/components_utils.py Outdated
Comment thread nf_core/pipelines/containers_utils.py Outdated
Comment thread nf_core/pipeline-template/workflows/pipeline.nf Outdated
@mashehu mashehu force-pushed the pipeline-container-config-builder branch from f115b17 to bfc916b Compare January 22, 2026 16:05
@mashehu mashehu mentioned this pull request Jan 23, 2026
Draft
@mashehu mashehu added this to the 3.6.0 milestone Mar 3, 2026
@mashehu mashehu force-pushed the pipeline-container-config-builder branch 2 times, most recently from ddd4269 to e05a068 Compare March 31, 2026 09:37
@mashehu mashehu force-pushed the pipeline-container-config-builder branch 3 times, most recently from 531c509 to 7cf72c1 Compare April 23, 2026 09:19
@mashehu mashehu mentioned this pull request Apr 23, 2026
Open
4 tasks
@mashehu mashehu force-pushed the pipeline-container-config-builder branch from 75f204c to e4ccb25 Compare April 23, 2026 10:59
mirpedrol and others added 27 commits April 24, 2026 13:00
@mirpedrol @mashehu
Update tests/pipelines/test_container_configs.py
342c90a
@mirpedrol @mashehu
remove too many warnings and add debug messages and one info summary …
c1ccc2e 
…instead
@mashehu
update arch names to make it consistent with modules container command
eaeedcb
@mashehu
parse the include statements to get modules paths from process names
f28b861
@mashehu
sort generated config files
9c39d10
@mashehu
drop linux_ prefix of arch in config file names
f9e507e
@mashehu
Apply suggestions from code review
7693a23 
Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>

# Conflicts:
#	nf_core/pipeline-template/workflows/pipeline.nf
@mashehu
add container configs to linting
862e82e 
# Conflicts:
#	nf_core/pipelines/containers_utils.py
@mashehu
remove unused argument
085099c
@mashehu
remove org from containerconfigs
e9e6311
@mashehu
update template modules to latest version
aead8e7
@mashehu
fix container config test
cb53b36
@mashehu
bump nextflow version to 25.10.4
6c08b88
@mashehu
fix type warning
b55504a
@mashehu
fix ruff linting errors
b675d24
@mashehu
update modules.json in tempalte
50e58b0
@mashehu
fix todo hint
e449914
@mashehu
add container config creation step to template rendering
2d96acf
@mashehu
fix path to mock
39018d2
@mashehu
fix tests by actually removing file that is now generated with pipeli…
ee84386 
…nes create
@mashehu
more test fixes
817ea79
@mashehu
move try_generate_container_configs to container_utils
713e7b4
@mashehu
use same PLATFORMS constant in containers_utils and tests
ea65a14
@mashehu
also regenerate container configs for patch commands
831cdb2
@mashehu
clean up container configs before writing new ones, to keep state in …
ef2edb2 
…sync
@mashehu
manually add newly installed module to container config generator
5e9ae89
@mashehu
update fastqc and multiqc to latest version
200571d 
(cherry picked from commit 3e0d9ad)

# Conflicts:
#	nf_core/pipeline-template/modules.json
#	nf_core/pipeline-template/modules/nf-core/fastqc/main.nf
#	nf_core/pipeline-template/modules/nf-core/multiqc/meta.yml
#	nf_core/pipeline-template/modules/nf-core/multiqc/tests/main.nf.test.snap
@mashehu mashehu force-pushed the pipeline-container-config-builder branch from f092068 to 200571d Compare April 24, 2026 11:01
@mashehu mashehu merged commit 7744f96 into nf-core:dev Apr 24, 2026
118 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mashehu mashehu mashehu approved these changes

@JulianFlesch JulianFlesch JulianFlesch approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

4.0.0

Development

Successfully merging this pull request may close these issues.

3 participants

@mirpedrol @mashehu @JulianFlesch