PerturbingGenesProject

This repository contains our project for INFO8010 Deep Learning.

• Course repository: https://github.com/glouppe/info8010-deep-learning
• Challenge: https://docs.crunchdao.com/competitions/competitions/broad-obesity/crunch-2

The goal of this version is to keep only the code that matters for our final submission:

• the preprocessing actually used downstream
• the retained final models
• the retained final samplers
• the scripts needed to retrain the poster setup and regenerate the poster figures

Data

To run the project, first download the challenge data from CrunchDAO:

https://docs.crunchdao.com/competitions/competitions/broad-obesity/crunch-2

You must sign in to access the files.

The code expects a data/ layout like this:

data/
├── default/
│   ├── obesity_challenge_2.h5ad
│   ├── obesity_challenge_2_local_gtruth.h5ad
│   ├── predict_genes_2.txt
│   ├── predict_perturbations_2.txt
│   ├── program_proportion.csv
│   ├── program_proportion_local_gtruth.csv
│   └── signature_genes.csv
└── small/
    ├── obesity_challenge_2.h5ad
    ├── obesity_challenge_2_local_gtruth.h5ad
    ├── predict_genes_2.txt
    ├── predict_perturbations_2.txt
    ├── program_proportion.csv
    ├── program_proportion_local_gtruth.csv
    └── signature_genes.csv

Typical options are:

• put the full CrunchDAO dataset in PerturbingGenesProject/data/default/
• or keep it outside the repository and pass its location with --data-dir
• if you want the reduced version, place it in data/small/

The code maps size=big to the same layout as size=default.

Environment

This project uses pdm and targets Python 3.12.

Install pdm:

python3.12 -m pip install pdm

From the PerturbingGenesProject/ root, create the environment:

pdm use -f 3.12
pdm install

Activate it:

eval "$(pdm venv activate)"

Or run commands directly through pdm:

pdm run python --version

Project structure

• src/perturbinggenes/utils/data/: perturbation parsing, metadata construction, and the fair perturbation-level split.
• src/perturbinggenes/models/: retained final models and samplers.
• src/perturbinggenes/poster_training.py: orchestrates the reduced poster retraining workflow.
• src/perturbinggenes/poster_results.py: builds the poster figures from generated artifacts.
• scripts/train_poster_models.py: command-line entry point for poster retraining.
• scripts/reproduce_poster_results.py: command-line entry point for figure generation.

Brief preprocessing overview

The preprocessing kept in this reduced project is intentionally small:

1. Perturbation labels from adata.obs["gene"] are parsed into a canonical representation.
2. The downstream code keeps only the useful features: (gene1, gene2, guide_count) and perturbation_type.
3. Cell metadata is built once from those parsed perturbation labels.
4. The train/validation split is done at the perturbation level, not at the cell level.
5. Controls stay in training, and higher-order perturbations remain train-only.

Final retained paths

The reduced project keeps only the directions used in the final report and poster:

• additive compositional baseline
• flat VAE center
• hierarchical VAE center
• state-aware MFA sampler
• hierarchical HVAE residual sampler

Reproducing the results from the poster

The poster workflow has two steps:

1. retrain the reduced final models and samplers
2. generate the poster figures from those artifacts

1. Train the poster models

From the PerturbingGenesProject/ root:

pdm run python scripts/train_poster_models.py \
  --project-root . \
  --data-dir data/default \
  --size default \
  --device cuda \
  --output-dir artifacts/poster_training

If your dataset is stored outside the repository, point --data-dir to that location instead, for example:

pdm run python scripts/train_poster_models.py \
  --project-root . \
  --data-dir ../data/default \
  --size default \
  --device cuda \
  --output-dir artifacts/poster_training

This retrains only the retained final path:

• additive baseline
• flat VAE center
• HVAE center
• identity state-aware MFA sampler
• hierarchical HVAE residual sampler

The expensive step is this training script, especially on the default dataset.

2. Render the poster figures

After training finishes:

pdm run python scripts/reproduce_poster_results.py \
  --training-dir artifacts/poster_training \
  --output-dir artifacts/poster_results

If cuda is not available, replace --device cuda above with --device mps or --device cpu.

This writes:

• artifacts/poster_results/quantitative/poster_quantitative_bar_chart.png
• artifacts/poster_results/quantitative/poster_quantitative_bar_chart.pdf
• artifacts/poster_results/quantitative/poster_quantitative_bar_chart.svg
• artifacts/poster_results/quantitative/selected_models_metrics.csv
• artifacts/poster_results/qualitative/panel_A_real_vs_generated_umap__single_guide.png
• artifacts/poster_results/qualitative/panel_A_real_vs_generated_umap__monogenic_double_guide.png
• artifacts/poster_results/qualitative/panel_A_real_vs_generated_umap__heterotypic_double_guide.png
• artifacts/poster_results/qualitative/panel_B_program_state_umap.png
• artifacts/poster_results/poster_results_manifest.json

Resource note

A full rerun on the default dataset retrains:

• the additive mean model
• the flat VAE and its additive mean model
• the HVAE and its additive mean model
• the retained samplers

On our final cluster run, this completed successfully with the following setup:

• GPU requested through Slurm: gpu:a6000ada
• GPU type used: NVIDIA RTX 6000 Ada / A6000 Ada
• GPU VRAM: about 48 GB
• System RAM requested: 128 GB
• Maximum RAM reported by Slurm: about 113.5 GiB (119052516K)
• Node memory available: 128 GiB

The corresponding seff summary was:

Job ID: 1096347
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 5
CPU Utilized: 00:42:06
CPU Efficiency: 20.02% of 03:30:20 core-walltime
Job Wall-clock time: 00:42:04
Memory Utilized: 113.54 GB
Memory Efficiency: 88.70% of 128.00 GB

So, to safely reproduce the full poster training workflow, a good practical target is:

• 1 GPU with about 48 GB VRAM
• about 128 GB of system RAM
• about 45 minutes of wall-clock time on hardware comparable to the run above