This repository provides the code to predict human fMRI responses for videos from features of a baseline deep neural network (DNN) model (AlexNet). We further provide scripts to prepare the submission for the Algonauts 2021 challenge. The terms features and activations are used interchangeably in the code.

IMPORTANT: The Alexnet example is provided to explain how to prepare submissions using a baseline model. Please prepare your results in same format to participate in the challenge.!

• Install anaconda
• Clone the repository git clone https://github.com/Neural-Dynamics-of-Visual-Cognition-FUB/Algonauts2021_devkit.git
• Change working directory cd Algonauts2021_devkit
• Run conda create -n Algonauts2021 python=3.7 anaconda to setup a new conda environment with required libraries
• Activate environment conda activate Algonauts2021
• Install nilearn , pytorch , decord and opencv
• Download the data here (if not already downloaded)and unzip in the working directory. Data is organized in two directories
  • AlgonautsVideos268_All_30fpsmax : contains 1102 videos: training (first 1000) and test (last 102) videos.
  • participants_data_v2021 : contains fMRI responses to training videos for both the challenge tracks.

• Run python feature_extraction/generate_features_alexnet.py to extract pretrained AlexNet features from the videos for all layers of AlexNet.
• With the default arguments, the script expects a directory ./AlgonautsVideos268_All_30fpsmax/ with the video sequences and saves the features in a directory called ./alexnet
• This code saves Alexnet features for every frame of every video, as well as a PCA transformation of these features to get top-100 components. These activations are split in train and test data

• Run python perform_encoding.py to create predicted fMRI responses for test videos based on AlexNet features or custom Neural Network Layers
• With the default arguments, the script expects a directory ./participants_data_v2021 with real fMRI data and ./alexnet/ with extracted NN features. It will generate predicted features using Alexnet (--model) layer_5 (--layer) for EBA (--roi) of subject 4 (--sub) in validation mode (--mode). The results will be stored in a directory called ./results. Running the script in default mode should return a mean correlation of 0.219
• Example command for whole brain data (will take several minutes): python perform_encoding.py -rd ./results -ad ./alexnet/ -model alexnet_devkit -l layer_5 -sub sub01 -r WB -m val -fd ./participants_data_v2021 -v True -b 1000

• Note: Predicted results should be generated for all combinations of ROIs and subjects. We have given an example file to generate results for all ROIs and all subjects in a given track. Please run python generate_all_results.py to generate predicted results using default model and layer.
  • For mini_track please run python generate_all_results.py -t mini_track
  • For full_track please run python generate_all_results.py -t full_track

• After generating predicted fMRI activity for all the ROIs and all the subjects in a given track (mini_track and/or full_track) using generate_all_results.py, run python prepare_submission.py in order to prepare the results for submission. All results will be combined into a single file.
  • For mini_track please run python prepare_submission.py -t mini_track
  • For full_track please run python prepare_submission.py -t full_track
• With the default arguments, the script expects the results from step 2 in a directory ./results/alexnet_devkit/layer_5. It prepares the submission for all 9 ROIs (mini_track) . To generate results for full_track change the arguments as mentioned above.
• The script creates a Pickle and a zip file (containing the Pickle file) for the corresponding track that can then be submitted for participation in the challenge.
• Submit the mini_track results here and full_track results here

If you use our code, partly or as is, please cite the paper below

@misc{cichy2021algonauts,
      title={The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion}, 
      author={R. M. Cichy and K. Dwivedi and B. Lahner and A. Lascelles and P. Iamshchinina and M. Graumann and A. Andonian and N. A. R. Murty and K. Kay and G. Roig and A. Oliva},
      year={2021},
      eprint={2104.13714},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}