Inter-individual reconstruction of subjective experience from brain activity

Haibao Wang, Fan L. Cheng, Shuntaro C. Aoki, Misato Tanaka, Yoshihiro Nagano, Hideki Izumi,
Yukiyasu Kamitani

Getting Started

Installation

To begin, clone the repository on your local machine, using git clone and pasting the url of this project:

git clone https://github.com/KamitaniLab/InterIndividualSubjectiveExperienceReconstruction.git

Build Environment

Step1: Navigate to the base directory and create the Conda environment:

conda env create -f env.yaml

Step2: Activate the environment:

conda activate NCC

Download Data

To use this project, you'll need to download and organize the required data:

• Download the training brain data for veridical perception from Figshare.
• Download the test brain data for visual illusion from Figshare.
• Download the test brain data for visual attention from Figshare.
• Download the test brain data for visual imagery from Figshare.
• Download the DNN features of stimuli from Figshare

Alternatively, you can use the following commands to download specific data (The data will be automatically extracted and organized into the designated directory, in progress):

# In "data" directory:
# To download the training fMRI data:
python download.py fmri_training

# Or to download the test fMRI data:
python download.py fmri_test

# download the DNN features of training images:
python download.py stimulus_feature

Download Pre-trained Decoders

To use this project, you'll need to download the required pre-trained decoders from Figshare with the following command:

python download.py pre-trained-decoders

If you prefer to train the decoders yourself (approximately 2 days per subject), detailed instructions and scripts are available in the feature-decoding directory.

Usage

Train Neural Code Converters

To train the neural code converters using content loss for subject pairs, navigate to the corresponding subdirectory under neural_code_conversion.

For example, to train the converter for visual illusion, navigate to neural_code_conversion/illusion and run:

python NCC_train.py --cuda

• Note: Use the --cuda flag when running on a GPU server. Omit --cuda if training on a CPU server.

Training one subject pair usually takes about 15 hours due to the large computational requirements. You can also download the pre-trained converters from Figshare with the following command:

python download.py pre-trained-converters

Test Neural Code Converters

DNN Feature Decoding

To decode DNN features from converted brain activities, navigate to the corresponding subdirectory under neural_code_conversion.

For example, for visual illusion, navigate to neural_code_conversion/illusion and run:

python NCC_test.py

Image Reconstruction

To reconstruct images from the decoded features:

1. Navigate to the reconstruction directory.
2. Follow the provided README and reconstruction demo for detailed instructions on setting up the environment and usage.
3. Modify the directory of the decoded features in the script as needed to reconstruct images.

Quantitative Evaluation

The quantitative evaluations are presented in terms of conversion accuracy and reconstruction quality.

Conversion Accuracy

To calculate raw correlations for conversion accuracy, navigate to the evaluation/conversion_accuracy directory and run:

# pattern correlation
python fmri_pattern_corr_content_loss.py

# profile correlation
python fmri_profile_corr_content_loss.py

Evaluation of reconstruction

To quantitatively evaluate the reconstructed images, navigate to the evaluation/reconstruction_evaluation directory.

For example, to evaluate illusion reconstruction, navigate to the illusion directory and run:

python Eval_color_illusion_vs_control.py
python Eval_line_global.py
python Eval_line_local.py

To evaluate attention reconstruction, navigate to the attention directory and run:

python recon_image_eval.py
python recon_image_eval_dnn.py

Imagery reconstruction can be evaluated in the same way by navigating to the corresponding imagery subdirectory.

Due to licensing restrictions, the ground truth/test images for attention and imagery reconstruction evaluation are not included in this repository. Please request and download them using this link, then organize the downloaded images under data/test_image/attention and data/test_image/imagery.

(back to top)