Add tests/mri fmriprep for fmriprep version 1.2.1

tests/data/mri/golden_master/mock/.gitignore

@@ -0,0 +1,4 @@
+
+*
+!*.h5
+!.gitignore

tests/data/mri/golden_master/real/.gitignore

@@ -0,0 +1,2 @@
+
+*.h5

tests/mri/fmriprep/README.md

@@ -0,0 +1,157 @@
+# Test code for `bdpy.mri.fmriprep`
+
+> **Path note:** The canonical location is `tests/mri/fmriprep/` (not `tests/bdpy/mri`).
+
+## Overview
+
+This directory contains tests for `bdpy.mri.fmriprep`.
+
+The current layout is split by data source:
+
+- `test_fmriprep_mock.py`: mock-data tests that run without external datasets
+- `test_fmriprep_real.py`: real-data golden-master test
+- `test_fmriprep_utils.py`: shared helpers for loading `fmriprep.py` and for real-data tests
+- `test_fmriprep_utils_mock.py`: mock dataset builder and expected-data helpers
+- `scripts/mock/`: helper scripts for mock golden-master preparation and test execution
+- `scripts/real/`: helper scripts for real-data download, preprocessing, golden-master preparation, and test execution
+
+For test coverage and maintenance notes, see `TEST_COVERAGE.md`.
+
+## Dependencies
+
+- Run these tests in the lab base environment where `bdpy`, `nipy`, `nibabel`, and related dependencies are already installed.
+- These tests assume the directory layout of `fmriprep` version `1.2`.
+- `bdpy/mri/fmriprep.py` still depends on the older `nipy` API (`get_data()`), so newer environments may fail even if the test code itself is correct.
+- Real-data tests additionally require `datalad` for dataset download and `docker` for `poldracklab/fmriprep:1.2.1`.
+
+## Working Directory
+
+Run all commands below from the project root, which is the directory containing `tests/` and `bdpy/`.
+
+Set the Python interpreter explicitly when needed:
+
+```bash
+export PYTHON_BIN=/path/to/python
+```
+
+If `PYTHON_BIN` is unset, the helper scripts default to `python`.
+
+## Mock Tests
+
+Run the mock-only suite directly:
+
+```bash
+"${PYTHON_BIN:-python}" -m pytest ./tests/mri/fmriprep/test_fmriprep_mock.py
+```
+
+Or use the helper scripts:
+
+```bash
+bash ./tests/mri/fmriprep/scripts/mock/step_1_prepare_gm.sh
+bash ./tests/mri/fmriprep/scripts/mock/step_2_run_test.sh
+```
+
+Script roles:
+
+- `step_1_prepare_gm.sh`: creates missing mock golden-master files if needed
+- `step_2_run_test.sh`: runs `test_fmriprep_mock.py`
+
+Mock golden-master files:
+
+- `./tests/data/mri/golden_master/mock/test_output_fmriprep_subject.h5`
+- `./tests/data/mri/golden_master/mock/test_output_fmriprep_subject_exclude.h5`
+
+If you need to regenerate them manually:
+
+```bash
+TEST_FMRIPREP_CREATE_GOLDEN_MASTER=1 "${PYTHON_BIN:-python}" -m pytest ./tests/mri/fmriprep/test_fmriprep_mock.py
+```
+
+## Real-Data Tests
+
+The real-data workflow is staged because it depends on external data and preprocessing outputs.
+
+### Environment
+
+For preprocessing scripts, set the environment variables below explicitly:
+
+```bash
+export BIDS_DIR=./tests/data/mri/ds006319
+export FREESURFER_HOME=/path/to/freesurfer
+export FS_LICENSE=/path/to/license.txt
+export FS_OUTPUT_DIR="${BIDS_DIR}/derivatives/freesurfer"
+```
+
+To avoid committing machine-specific paths, copy the local template below and edit it on your machine only:
+
+```bash
+cp ./tests/mri/fmriprep/scripts/real/_fs_env.sh.example ./tests/mri/fmriprep/scripts/real/_fs_env.sh
+```
+
+`step_2_run_freesurfer.sh` and `step_3_run_fmriprep.sh` read `./tests/mri/fmriprep/scripts/real/_fs_env.sh` if it exists, then fall back to the current shell environment.
+Keep `_fs_env.sh` local-only; the repository only tracks `_fs_env.sh.example`.
+
+### Run The Full Pipeline
+
+```bash
+bash ./tests/mri/fmriprep/scripts/real/step_1_download.sh
+bash ./tests/mri/fmriprep/scripts/real/step_2_run_freesurfer.sh
+bash ./tests/mri/fmriprep/scripts/real/step_3_run_fmriprep.sh
+bash ./tests/mri/fmriprep/scripts/real/step_4_prepare_gm.sh
+bash ./tests/mri/fmriprep/scripts/real/step_5_run_test.sh
+```
+
+Script roles:
+
+- `step_1_download.sh`: downloads the raw real dataset
+- `step_2_run_freesurfer.sh`: runs FreeSurfer
+- `step_3_run_fmriprep.sh`: runs fMRIPrep using the two functional sessions plus `ses-anatomy`
+- `step_4_prepare_gm.sh`: creates the real-data golden master if it is missing
+- `step_5_run_test.sh`: runs `test_fmriprep_real.py`
+
+### Run The Test Only
+
+If the dataset, FreeSurfer output, and fMRIPrep output are already prepared, run the test directly:
+
+```bash
+"${PYTHON_BIN:-python}" -m pytest ./tests/mri/fmriprep/test_fmriprep_real.py
+```
+
+Real-data golden-master file:
+
+- `./tests/data/mri/golden_master/real/test_output_fmriprep_real_exclude.h5`
+
+If you need to regenerate it manually:
+
+```bash
+TEST_FMRIPREP_CREATE_GOLDEN_MASTER=1 "${PYTHON_BIN:-python}" -m pytest ./tests/mri/fmriprep/test_fmriprep_real.py
+```
+
+## Notes
+
+- The split `mock` / `real` structure replaced the older `USE_REAL`-based mixed test module.
+- The current recommended entry points are `test_fmriprep_mock.py` and `test_fmriprep_real.py`.
+- The real-data helper scripts and tests prefer `./tests/data/mri/ds006319`. The test helpers also accept the older `./tests/data/ds006319` layout for compatibility.
+
+## Submission Checklist
+
+Before submitting, run this minimal checklist from the project root:
+
+```bash
+# 1) Mock tests
+"${PYTHON_BIN:-python}" -m pytest ./tests/mri/fmriprep/test_fmriprep_mock.py
+
+# 2) Real-data tests (only when dataset + preprocessing outputs are ready)
+"${PYTHON_BIN:-python}" -m pytest ./tests/mri/fmriprep/test_fmriprep_real.py
+
+# 3) Verify golden-master files exist
+test -f ./tests/data/mri/golden_master/mock/test_output_fmriprep_subject.h5
+test -f ./tests/data/mri/golden_master/mock/test_output_fmriprep_subject_exclude.h5
+test -f ./tests/data/mri/golden_master/real/test_output_fmriprep_real_exclude.h5
+```
+
+Recommended for reproducibility:
+
+- Keep machine-specific FreeSurfer settings in `scripts/real/_fs_env.sh` (local-only).
+- Do not commit `_fs_env.sh`; only `_fs_env.sh.example` should be tracked.
+- If golden masters were regenerated intentionally, mention that explicitly in your submission notes.

tests/mri/fmriprep/TEST_COVERAGE.md

@@ -0,0 +1,125 @@
+# Test Coverage and Maintenance Notes for `bdpy.mri.fmriprep`
+
+This document summarizes what the current test suite covers, what remains untested, and known implementation issues observed during test reorganization.
+
+_Last updated: 2026-04-06_
+
+## Test Coverage Summary
+
+`test_fmriprep_mock.py` covers:
+
+- `FmriprepData` parsing and dataset traversal
+- failure cases for missing or malformed inputs
+- golden-master validation for `create_bdata_fmriprep`
+- surface loading through `BrainData`
+- `LabelMapper`
+- subject-level BData creation
+- `cut_run` behavior
+- `get_xyz`
+- `load_mri`
+
+`test_fmriprep_real.py` covers:
+
+- golden-master validation of `create_bdata_fmriprep` using real fMRIPrep output with exclusions
+
+## Functions and Classes in `bdpy/mri/fmriprep.py`
+
+### FmriprepData (class)
+
+Parses a BIDS/fMRIPrep directory and stores subject/session/run structures in an `OrderedDict`.
+Internally calls `__parse_data` (subject/session/run discovery) and `__get_task_event_files` (events.tsv + bold.json pairing).
+
+| Item | Details |
+| --- | --- |
+| Test status | Covered |
+| Real-data dependency | Replaced by mock directory structure |
+| Existing tests | `TestFmriprepDataMock`: private vars, get_subjects, get_sessions, parse_session, parse_data, get_task_event_files / `TestFmriprepDataFailures` (parsing failures): missing events, missing bold.json |
+| Not yet tested | File-pattern branch for fMRIPrep versions `1.0`/`1.1` |
+
+Note: `TestFmriprepDataFailures` is a single test class containing 5 methods. Two (missing events, missing bold.json) test `FmriprepData` parsing failures; the remaining three (missing confounds, missing motion columns, unknown labels) test failures during `create_bdata_fmriprep` execution. They are listed under the respective sections below.
+
+### create_bdata_fmriprep (function)
+
+Top-level orchestrator. Parses with `FmriprepData`, applies exclusions (`subject`/`session`/`run`/`session-run`), loads `label_mapper` files (csv/tsv), and calls `__create_bdata_fmriprep_subject` per subject. Can also split outputs by task label when `split_task_label=True`.
+
+| Item | Details |
+| --- | --- |
+| Test status | Partially covered; known issues remain |
+| Real-data dependency | Replaced by mock data |
+| Existing tests | `TestCreateBdataFmriprepMock`: golden masters (with/without exclusion), surface modes, empty list when subject excluded / `TestFmriprepDataFailures` (execution failures): missing confounds, missing motion columns, unknown labels |
+| Gaps / issues | (1) Potential mutation-while-iterating issue: deleting from `fmriprep.data` (`OrderedDict`) during iteration may raise `RuntimeError: dictionary changed size during iteration` on Python 3.8. (2) No direct test for `split_task_label=True`. (3) No focused unit test for csv/tsv `label_mapper` loading logic. (4) No explicit test for `return_list=False` single-BData return path. |
+
+### BrainData (class)
+
+Loads fMRI data (volume or surface).
+For volume: reads NIfTI via `nipy.load_image`, reshapes 4D to 2D (sample x voxel), and computes xyz/ijk from affine.
+For surface: loads left/right GIFTI via nibabel and concatenates.
+
+| Item | Details |
+| --- | --- |
+| Test status | Surface path covered; volume path mostly indirect |
+| Real-data dependency | Replaced by mock data |
+| Existing tests | `TestBrainDataMock`: paired surface load, one-side missing, empty darray, vertex-count mismatch, invalid dtype |
+| Gaps / issues | (1) No direct unit test for volume loader (`__load_volume`), only indirect coverage through `__create_bdata_fmriprep_subject`. (2) String identity checks (`is` / `is not`) are used for dtype and should be equality checks (`==` / `!=`). (3) No direct test for xyz/ijk extraction on 3D volume input. |
+
+### LabelMapper (class)
+
+Maps string labels to numeric values. Converts `'n/a'` to `NaN` and validates reverse-mapping uniqueness.
+
+| Item | Details |
+| --- | --- |
+| Test status | Mostly covered |
+| Real-data dependency | None |
+| Existing tests | `TestLabelMapperMock` (3 methods): normal mapping including n/a→NaN (`test_get_value_returns_expected_numeric_mapping`), same-value mapping (`test_get_value_returns_existing_value_for_duplicate_mapping`), unknown key (`test_get_value_raises_on_missing_key`) |
+| Not yet tested | (1) Non-unique reverse mapping case that raises `RuntimeError('Invalid label-value mapping')` (mapping different labels to the same value in separate calls, as opposed to the tested case where multiple labels share the same value in one dict). (2) Direct assertion of `dump()` output. |
+
+### create_bdata_singlesubject (function)
+
+Thin public wrapper for `__create_bdata_fmriprep_subject`; forwards arguments directly.
+
+| Item | Details |
+| --- | --- |
+| Test status | Covered |
+| Real-data dependency | Replaced by mock data |
+| Existing tests | `TestCreateBdataFmriprepSubjectMock.test_create_bdata_singlesubject` |
+| Not yet tested | Calling this public API with `cut_run=True` (the private function path is tested directly). |
+
+### __create_bdata_fmriprep_subject (private function)
+
+Core worker. For each run: loads with `BrainData`, reads confounds/motion, builds block/label arrays from task events, and writes to BData. Supports cropping when `cut_run=True` and event length exceeds data length.
+
+| Item | Details |
+| --- | --- |
+| Test status | Covered |
+| Real-data dependency | Replaced by mock data |
+| Existing tests | golden master, direct call path, `cut_run` (True/False), confounds with NaN |
+| Not yet tested | `NotImplementedError` path when `cut_duration < 0` |
+
+### __get_xyz (private function)
+
+Computes voxel coordinates with `itertools.product` and `xrange`; supports 3D/4D images.
+
+| Item | Details |
+| --- | --- |
+| Test status | Covered (with patching) |
+| Real-data dependency | None (uses fake image objects) |
+| Existing tests | `TestGetXyzMock`: 4D/3D fake images |
+| Notes | Python 3 does not provide `xrange`, so tests patch `fmriprep.xrange = range`. The function is not part of the current main execution path, so maintenance priority is low. |
+
+### __load_mri (private function)
+
+Loads NIfTI via `nipy.load_image` and returns `(data, xyz, ijk)` for 3D/4D input. Its behavior is largely duplicated by `BrainData.__load_volume`.
+
+| Item | Details |
+| --- | --- |
+| Test status | Covered |
+| Real-data dependency | Replaced by mock data |
+| Existing tests | `TestLoadMriMock`: 4D mock, 3D temp file, invalid dimensions |
+| Notes | Code is largely duplicated with `BrainData.__load_volume`; retained for compatibility checks but a candidate for future cleanup. |
+
+## Main Findings
+
+- `create_bdata_fmriprep` has a potential dictionary-mutation bug while iterating over `OrderedDict`.
+- `BrainData` contains `is`-based string comparisons that should be equality comparisons.
+- `__get_xyz` and `__load_mri` appear to be outside the current main path.
+- `split_task_label=True` still has no direct test coverage.

tests/mri/fmriprep/scripts/mock/step_1_prepare_gm.sh

@@ -0,0 +1,23 @@
+PYTHON_BIN="${PYTHON_BIN:-python}"
+
+mock_gm_h5_path_list=(
+  ./tests/data/mri/golden_master/mock/test_output_fmriprep_subject_exclude.h5
+  ./tests/data/mri/golden_master/mock/test_output_fmriprep_subject.h5
+)
+
+need_mock_gm=0
+
+for gm_h5_path in "${mock_gm_h5_path_list[@]}"; do
+  if [ ! -f "${gm_h5_path}" ]; then
+    need_mock_gm=1
+    echo "Missing golden master file: ${gm_h5_path}"
+  fi
+done
+
+if [ "${need_mock_gm}" -eq 0 ]; then
+  echo "All mock golden master files already exist."
+  echo "Skipping GM creation step. If you want to recreate the GM, please delete the existing file and run this script again."
+  exit 0
+fi
+
+TEST_FMRIPREP_CREATE_GOLDEN_MASTER=1 "${PYTHON_BIN}" -m pytest ./tests/mri/fmriprep/test_fmriprep_mock.py

tests/mri/fmriprep/scripts/mock/step_2_run_test.sh

@@ -0,0 +1,4 @@
+
+PYTHON_BIN="${PYTHON_BIN:-python}"
+
+"${PYTHON_BIN}" -m pytest ./tests/mri/fmriprep/test_fmriprep_mock.py

tests/mri/fmriprep/scripts/real/.gitignore

@@ -0,0 +1 @@
+_fs_env.sh

tests/mri/fmriprep/scripts/real/_fs_env.sh.example

@@ -0,0 +1,4 @@
+#!/bin/sh
+
+export FREESURFER_HOME="/path/to/freesurfer"
+export FS_LICENSE="/path/to/license.txt"

tests/mri/fmriprep/scripts/real/step_1_download.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+set -euo pipefail
+
+data_root="./tests/data/mri"
+dataset_name="ds006319"
+dataset_dir="${data_root}/${dataset_name}"
+
+mkdir -p "${data_root}"
+cd "${data_root}"
+
+if [ ! -d "${dataset_name}/.git" ]; then
+  datalad install "https://github.com/OpenNeuroDatasets/${dataset_name}.git"
+fi
+
+cd "${dataset_name}"
+
+# Materialize the exact tree consumed by the FreeSurfer/fMRIPrep helper scripts.
+datalad get -r \
+  dataset_description.json \
+  participants.tsv \
+  participants.json \
+  README \
+  CHANGES \
+  sub-S1/ses-SoundTest01 \
+  sub-S1/ses-SoundTest02 \
+  sub-S1/ses-anatomy
+
+echo "Download completed for ${dataset_dir}"