Our solution to the Inria "Cross-modal Content-based Retrieval for 3D Medical Images" challenge (EHL Paris 2026). For each query brain MRI (T1 post-contrast), we rank a gallery of candidate scans so the same patient's other scan (T2, or an intra-operative scan) comes out on top. The score is Mean Reciprocal Rank (MRR), averaged over three datasets.
Result: perfect MRR = 1.000 on all three validation pools — training-free. We match patients by aligning anatomy and measuring structural similarity, not by learned embeddings.
| Dataset | Geometry | Our method | Val MRR |
|---|---|---|---|
| 1 | already on a common grid | direct NMI | 1.000 |
| 2 | independently warped | N4 + brain mask → GPU registration (affine + deformable) + MIND | 1.000 |
| 3 | intra-op, anatomy changed | shape prior + (masked) NMI | 1.000 |
The final submission is submission_n4final.csv. Full write-up: APPROACH.md and
report/report.pdf.
The query and target are different MRI modalities, so their raw brightnesses are unrelated — you can't compare pixels directly. The whole solution is one idea applied with per-dataset geometry handling:
- Measure structural similarity, not brightness. Mutual Information (MI/NMI) rewards a consistent statistical correspondence between two images — high when the same anatomy lines up, regardless of modality. MIND descriptors do the same at the local-texture level and drive the registration.
- Align first when needed. Dataset 1 is already aligned (direct NMI). Dataset 2 is warped, so we register (GPU affine + deformable) before scoring. Dataset 3's anatomy genuinely changed (surgery), so registration is rejected; instead we use a shape cue + MI on the shared region.
- Clean the signal. N4 bias-field correction + brain masking removes scanner shading and non-brain voxels — the lever that took Dataset 2 from 0.976 to 1.000.
We also built three deep-learning alternatives as a research comparison (contrastive
embeddings, learned dense registration / VoxelMorph, and a hybrid learned-affine + per-pair
refinement). All lost to the training-free method — the takeaway being that per-instance
optimisation beats the model here. See report/report.pdf and the slide decks.
retrieval.py Core: IO, preprocessing, NMI, ranking, MRR scorer
run.py Direct-NMI ranking (dataset 1) + shape-filtered (dataset 3) + local validation
gpu_reg/ GPU registration (the dataset-2 engine)
gpu_reg.py Batched coarse-to-fine affine + deformable registration with MIND
gpu_rank.py GPU ranking -> submission CSV (flags: --deformable --n4 --fast --shard ...)
robustness/ Robust dataset-3 variant, N4 cache, RRF ensemble, A/B + benchmarks
register.py / reg_rank.py CPU SimpleITK registration (earlier baseline)
assemble.py Merge per-dataset CSVs into the 377-row submission
run_2gpu.sh Dual-GPU sharded driver (~2x, identical results)
APPROACH.md Final method, validation, and the fast production pipeline
report/ Full write-up (report.pdf) + transcript
presentation/ Pitch decks (general + technical) and the cheat sheet
pitch_general.pdf Plain-language walkthrough (for the whole hackathon)
pitch_technical.pdf Technical deck (for organizers / participants)
cheatsheet.pdf Slide-ordered glossary of every term, model, and what we trained
submission_n4final.csv Final 1.000 submission
Runs in a CUDA container with the Kaggle data under ./data.
# Dataset 1 — direct NMI (no registration, no training)
python run.py rank --dataset dataset1 --split test --out ds1_test.csv
# Dataset 2 — GPU registration (affine + deformable) + N4 bias correction
PYTHONPATH=. CUDA_VISIBLE_DEVICES=0 python gpu_reg/gpu_rank.py \
--dataset dataset2 --split test --out ds2_test.csv --batch 100 --deformable --n4
# Dataset 3 — shape-filtered NMI
python run.py rank --dataset dataset3 --split test --out ds3_test.csv
# Merge into one valid 377-row Kaggle submission
python assemble.py --out submission.csv ds1_*.csv ds2_*.csv ds3_*.csvFast production pipeline (validated 1.000, ~13 min on 2 Blackwell GPUs — parallel N4 cache +
dual-GPU sharding + two-tier registration) is documented in APPROACH.md.
- Held-out labelled data (dataset 1 + synthetic deformation): MRR 0.99, independent of the leaderboard. Raw NMI on the same data scores ~0.30, so registration is doing real work.
- Dataset-3 leak test: shape-only = 0.82 vs shape + NMI = 1.0 → NMI genuinely resolves same-shape candidates (not a shape leak).
Kaggle: https://www.kaggle.com/t/b33ec3e76c3d4e16a6b56852470b3ebf
- Modalities: query = T1 post-contrast; target = T2 (or intra-operative for dataset 3). All volumes are 3D NIfTI in RAS orientation, 1mm isotropic.
- Three independent retrieval pools. Always rank a query only against its own dataset's, same-split gallery. Counts: dataset1 40/100, dataset2 40/100, dataset3 20/77 (val/test); dataset1 also ships 350 labelled training pairs (the only labels in the challenge).
- Metric:
score = (MRR_d1 + MRR_d2 + MRR_d3) / 3. Reciprocal rank =1 / rankof the true target; missing/omitted query → 0. Validation rows = public LB; test rows = private LB. - Submission: one combined CSV, columns
query_id,target_id_ranking, one row per query with a full space-separated ranking of that pool's gallery (377 rows total). Submitting one dataset only → multiply the displayed score by 3 for that dataset's MRR. Limit: 100 submissions/team/day.
Full data layout, manifests, and submission spec
dataset1/ train_pairs.csv val_queries.csv val_gallery.csv test_queries.csv test_gallery.csv images/{train,val,test}/
dataset2/ val_queries.csv val_gallery.csv test_queries.csv test_gallery.csv images/{val,test}/
dataset3/ val_queries.csv val_gallery.csv test_queries.csv test_gallery.csv images/{val,test}/
sample_submission.csv
dataset1 is registered to a common grid and carries the labelled pairs. dataset2 applies
independent rigid + non-linear deformation to each image. dataset3 resamples each intra-op target
into its query's coordinate space, but anatomy can genuinely differ (resection, brain shift). Code
must not assume one fixed image shape; matching query/target may differ in shape.
train_pairs.csv : pair_id,query_id,target_id,query_image,target_image,query_modality,target_modality,dataset
queries.csv : query_id,query_image,query_modality,dataset
gallery.csv : target_id,target_image,target_modality,dataset
query_id,target_id_ranking
q_...,g_... g_... g_... ...
Each row ranks every target ID from that query's same-dataset, same-split gallery, most likely first. All IDs are globally unique, so the combined file needs no dataset column. Expected ranking lengths: d1 val 40 / test 100, d2 val 40 / test 100, d3 val 20 / test 77.
Provided baseline (slice_clip_baseline.py)
A deliberately simple MONAI + PyTorch 2D-slice CLIP baseline ships with the challenge to demonstrate the data format and submission generation. It is not a strong solution and we do not build on it (it discards the 3D geometry that actually solves the task).
DATA_ROOT=/path/to/kaggle_dataset
uv run slice_clip_baseline.py \
--data-root "$DATA_ROOT" \
--train-pair-csv "$DATA_ROOT/dataset1/train_pairs.csv" \
--query-csv "$DATA_ROOT/dataset1/val_queries.csv" --gallery-csv "$DATA_ROOT/dataset1/val_gallery.csv" \
--query-csv "$DATA_ROOT/dataset1/test_queries.csv" --gallery-csv "$DATA_ROOT/dataset1/test_gallery.csv" \
--query-csv "$DATA_ROOT/dataset2/val_queries.csv" --gallery-csv "$DATA_ROOT/dataset2/val_gallery.csv" \
--query-csv "$DATA_ROOT/dataset2/test_queries.csv" --gallery-csv "$DATA_ROOT/dataset2/test_gallery.csv" \
--query-csv "$DATA_ROOT/dataset3/val_queries.csv" --gallery-csv "$DATA_ROOT/dataset3/val_gallery.csv" \
--query-csv "$DATA_ROOT/dataset3/test_queries.csv" --gallery-csv "$DATA_ROOT/dataset3/test_gallery.csv" \
--out slice_clip_submission.csv