Develop => main by ialarmedalien · Pull Request #186 · kbase/cdm-data-loaders

ialarmedalien · 2026-06-16T16:00:50Z

Description of PR purpose/changes

Merging develop => main to prep for next release

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Add qsv to docker image for xSV linting/validation/etc.

…tainer

Replace MinIO with CEPH for testing

Use xml_file_splitter binary instead of docker container

Add decompression of gz file util

* relabel ceph tests; let fail if ceph not reachable * address copilot comments * update test script to exclude ceph tests * update test script to exclude ceph tests * fail fast when ceph unreachable * clean up test imports * formatting --------- Co-authored-by: i alarmed alien <ialarmedalien@users.noreply.github.com>

…oints to be run separately

NCBI pipeline updates

* use path types for s3, ftp, local paths for ncbi ftp * Potential fix for pull request finding 'Statement has no effect' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Potential fix for pull request finding 'Unused import' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * address copilot comments * Update src/cdm_data_loaders/ncbi_ftp/manifest.py Co-authored-by: i alarmed alien <ialarmedalien@users.noreply.github.com> * address review comments * address review comments * address review comments * clean up settings tests * clean up settings tests more --------- Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: i alarmed alien <ialarmedalien@users.noreply.github.com>

Dependency updates for 06-2026

codecov · 2026-06-16T16:17:08Z

Codecov Report

❌ Patch coverage is 87.48906% with 143 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.06%. Comparing base (21d3a6f) to head (103b824).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/cdm_data_loaders/utils/s3.py	57.51%	65 Missing ⚠️
src/cdm_data_loaders/ncbi_ftp/promote.py	89.95%	23 Missing ⚠️
src/cdm_data_loaders/ncbi_ftp/manifest.py	92.98%	19 Missing ⚠️
...rc/cdm_data_loaders/pipelines/ncbi_ftp_download.py	91.92%	13 Missing ⚠️
src/cdm_data_loaders/ncbi_ftp/assembly.py	90.36%	8 Missing ⚠️
src/cdm_data_loaders/utils/ftp_client.py	90.24%	8 Missing ⚠️
src/cdm_data_loaders/ncbi_ftp/metadata.py	91.56%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #186      +/-   ##
==========================================
+ Coverage   60.13%   67.06%   +6.92%     
==========================================
  Files          64       72       +8     
  Lines        3339     4424    +1085     
==========================================
+ Hits         2008     2967     +959     
- Misses       1331     1457     +126

Files with missing lines	Coverage Δ
src/cdm_data_loaders/ncbi_ftp/constants.py	`100.00% <100.00%> (ø)`
src/cdm_data_loaders/pipelines/all_the_bacteria.py	`97.16% <ø> (ø)`
src/cdm_data_loaders/pipelines/ncbi_rest_api.py	`90.47% <100.00%> (+2.24%)`	⬆️
src/cdm_data_loaders/utils/checksums.py	`100.00% <100.00%> (ø)`
src/cdm_data_loaders/utils/gz.py	`92.85% <100.00%> (+3.11%)`	⬆️
src/cdm_data_loaders/utils/spark_delta.py	`82.41% <100.00%> (ø)`
src/cdm_data_loaders/ncbi_ftp/metadata.py	`91.56% <91.56%> (ø)`
src/cdm_data_loaders/ncbi_ftp/assembly.py	`90.36% <90.36%> (ø)`
src/cdm_data_loaders/utils/ftp_client.py	`90.24% <90.24%> (ø)`
...rc/cdm_data_loaders/pipelines/ncbi_ftp_download.py	`91.92% <91.92%> (ø)`
... and 3 more

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78ce964...103b824. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mattldawson and others added 30 commits April 16, 2026 15:18

draft ncbi scripts

f7fc9d5

add local minio testing

fa5fa97

add NCBI end-to-end testing instructions

f4d838a

debug and cleanup

5c6c547

cleanup and formatting

e5bc2f6

formatting

de8d325

Potential fix for pull request finding 'Unused import'

3f0e604

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Potential fix for pull request finding 'Unused import'

4f93d5f

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Potential fix for pull request finding

399a00d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

fee2ab4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

5233d9d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

e5b58f2

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

64ead4c

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

9fba2c5

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

address copilot comments

c89ad37

increased timeout for trivy action

9a5e2fe

Potential fix for pull request finding 'Unused import'

de77b14

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Potential fix for pull request finding 'Unused import'

7d9ced1

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Potential fix for pull request finding

16dfda3

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

521978d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

8f73674

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

add progress bar to manifest verification

1e8f363

add synthetic assembly summary generation option

7326641

debug synthetic manifest

4cf60d6

add custom release date to synthetic manifest generation

9a1d1e5

fix style

a4ee186

Merge branch 'main' into develop-add-ncbi-ftp

e351288

update docs and progress bar

6e45879

add checks for synthetic summary creation

adfda18

add connection check to S3 store

fdd67c4

mattldawson and others added 25 commits June 3, 2026 13:57

replace minio with ceph for testing

e45d433

add diagnostics and simplify integration action

38e549d

Potential fix for pull request finding

abbcf37

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

bfecd45

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Add qsv to docker image for xSV linting/validation/etc.

c5cf36a

address copilot comments

252c6df

Merge branch 'develop' into develop-170-ceph-testing

ed6cc21

update s3 credential test conditions

203f6c7

Merge pull request #172 from kbase/add_csv_lint

ae626c0

Add qsv to docker image for xSV linting/validation/etc.

Merge branch 'develop' into develop-170-ceph-testing

78d8072

reset s3 creds for testing in actions

2837bb0

update s3 test conditions

84d7492

reset s3 creds in test

80faf18

Just download binaries of xml_file_splitter and qsv to install in con…

aa85b92

…tainer

Merge pull request #171 from kbase/develop-170-ceph-testing

26ce76e

Replace MinIO with CEPH for testing

Merge branch 'develop' into xml_split_binary_qsv

9fc8eff

Merge pull request #174 from kbase/xml_split_binary_qsv

671e3bf

Use xml_file_splitter binary instead of docker container

Add decompression of gz file util

9ba7ab6

Merge pull request #178 from kbase/add_decompress_file

7d2ccdb

Add decompression of gz file util

Add option to NCBI REST pipeline to allow dataset and annotation endp…

62f0a1d

…oints to be run separately

Merge pull request #183 from kbase/separate_ncbi_requests

c5d758c

NCBI pipeline updates

Updating vcrpy to version compatible with newest AOIHTTP

cbfb350

Merge pull request #164 from kbase/update_deps

eaba059

Dependency updates for 06-2026

ialarmedalien requested a review from mattldawson June 16, 2026 16:00

ialarmedalien self-assigned this Jun 16, 2026

Merge branch 'main' into develop

103b824

ialarmedalien merged commit 4df47cd into main Jun 16, 2026
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Develop => main#186

Develop => main#186
ialarmedalien merged 171 commits into
mainfrom
develop

ialarmedalien commented Jun 16, 2026

Uh oh!

codecov Bot commented Jun 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ialarmedalien commented Jun 16, 2026

Description of PR purpose/changes

Uh oh!

codecov Bot commented Jun 16, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants