This repository contains an implementation of our ICLR 2019 paper;

• Seil Na, Yo Joong Choe, Dong-Hyun Lee and Gunhee Kim. Discovery of Natural Language Concepts in Individual Units of CNNs

TL;DR: Individual units of deep CNNs learned in NLP tasks (e.g. translation, classification) could act as a natural language concept detector.

This work covers the interpretability of Deep Neural Network. We expect that it sheds useful light on how the representation of Deep CNNs learned in language tasks represents the given text.

We show that several information in the given text are not distributed across all units of representation. We observe AND quantify that even a single unit can act as a natural language concept (e.g. morpheme, word, phrase) detector.

In this work, we align three natural language concepts per unit. Most units are selectively responsive to the concepts we align. If you want to see the full results, see Optional-Full Visualization Results.

We also discovered that several units tend to capture the concepts that go beyond natural langauge form. Although it is relatively hard to quantify it, we belive that further investigation would be one of interesting future direction. We visualize some units that capture abstract form concepts as follows:

If you want to get the results without running the code, skip these parts and see Optional-Full Visualization Results.

• Python 2.7
• anaconda (Python 2.7 version, latest version recommended)

• Clone the code from GitHub.

git clone https://github.com/seilna/CNN-Units-in-NLP.git

• Create environment via conda & downlaod spacy (english) model

conda env create -f environment.yml
conda activate iclr_19_na
python -m spacy download en

• Download training data & pretrained models (~160GB space)

cd script
bash setup.sh 

cd script
bash run.sh 

will save visualization results at visualization/.

or skip to Optional-Full Visualization Results.

cd script
bash download_visualization.sh

or google drive link

If you find the code useful, please cite the following paper.

@inproceedings{
  Na:ICLR:2019,
  title = "{Discovery of Natural Language Concepts in Individual Units of CNNs}",
  author = {Seil Na and Yo Joong Choe and Dong-Hyun Lee and Gunhee Kim},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=S1EERs09YQ},
}

Each model used in our experiments is implemented based on this and this repository. We thank the authors.

We also appreciate Insu Jeon, Jaemin Cho, Sewon Min, Yunseok Jang and the anonymous reviewers for their helpful comments and discussions. This work was supported by Kakao and Kakao Brain corporations, IITP grant funded by the Korea government (MSIT) (No. 2017-0-01772) and Creative Pioneering Researchers Program through Seoul National University.

Have any question? Please contact:

[email protected]