Feature decoding

This repository provides scripts of deep neural network (DNN) feature decoding from fMRI brain activities, originally proposed by Horikawa & Kamitani (2017) and employed in DNN-based image reconstruction methods of Shen et al. (2019) as well as recent studies in Kamitani lab.

Usage

Environment setup

Please setup Python environment where packages in requirements.txt are installed.

# Using venv
$ python -m venv .venv
$ . .venv/bin/activate
$ pip install -r requirements.txt

Data setup

Run the following commands in data directory to download required fMRI and DNN features.

# In "./data" directory:

# fMRI data (collected by Shen et al., 2019)
python download.py fmri_deeprecon_fmriprep_vc 

# DNN features (VGG-19)
python download.py features_imagenet_training_vgg19
python download.py features_imagenet_test_vgg19

Decoding with PyFastL2LiR

• Training: train_decoder_fastl2lir.py
• Test (prediction): predict_feature_fastl2lir.py
• Evaluation: evaluation.py
• Example config file: deeprecon_pyfastl2lir_alpha100_vgg19_allunits.yaml

# Training of decoding models
$ python train_decoder_fastl2lir.py config/deeprecon_pyfastl2lir_alpha100_vgg19_allunits.yaml

# Prediction of DNN features
$ python predict_feature_fastl2lir.py config/deeprecon_pyfastl2lir_alpha100_vgg19_allunits.yaml

# Evaluation
$ python evaluation.py config/deeprecon_pyfastl2lir_alpha100_vgg19_allunits.yaml

Decoding with generic regression models

• Training: train_decoder_sklearn_ridge.py (example for scikit-learn Ridge regression)
• Test (prediction): predict_feature.py
• Evaluation: evaluation.py
• Example config file: deeprecon_sklearn_ridge_alpha100_vgg19_allunits

# Training of decoding models
$ python train_decoder_sklearn_ridge.py config/deeprecon_pyfastl2lir_alpha100_vgg19_allunits.yaml

# Prediction of DNN features
$ python preeict_feature.py config/deeprecon_pyfastl2lir_alpha100_vgg19_allunits.yaml

# Evaluation
$ python evaluation.py config/deeprecon_pyfastl2lir_alpha100_vgg19_allunits.yaml

Cross-validation feature decoding

• Training: cv_train_decoder_fastl2lir.py (example for scikit-learn Ridge regression)
• Test (prediction): cv_predict_feature_fastl2lir.py
• Evaluation: cv_evaluation.py
• Example config file: deeprecon_cv_pyfastl2lir_alpha100_vgg19_allunits

# Training of decoding models
$ python cv_train_decoder_fastl2lir.py config/deeprecon_cv_pyfastl2lir_alpha100_vgg19_allunits.yaml

# Prediction of DNN features
$ python cv_predict_feature_fastl2lir.py config/deeprecon_cv_pyfastl2lir_alpha100_vgg19_allunits.yaml

# Evaluation
$ python cv_evaluation.py config/deeprecon_cv_pyfastl2lir_alpha100_vgg19_allunits.yaml

References

• Horikawa and Kamitani (2017) Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications 8:15037. https://www.nature.com/articles/ncomms15037
• Shen, Horikawa, Majima, and Kamitani (2019) Deep image reconstruction from human brain activity. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1006633