UAI2022: Addressing Token Uniformity in Transformers via Singular Value Transformation

Hanqi Yan, Lin Gui, Wenjie Li, Yulan He.

In this work, we characterise the token uniformity problem (See in the Figure below) commonly observed in the output of transformer-based architectures by the degree of skewness in singular value distributions and propose a singular value transformation function (SoftDecay) to address the problem.

Our project is based on Huggingface Transformers, so basically you can refer the requirements from their repository.

1. Main Functions

Our method is evaluated on text-classification tasks (GLUE Datasets), so we define the main functions in run_glue_no_trainer.py. It should be noted that if you have already downloaded the official Transformer packgae, you may need to specify the directory for our updated version.

# involve our updated SoftDecay Transofmer Directory /tokenUni/src/

sys.path.insert(0,"/YourDir/tokenUni/src/")

2. Pretrained Language Models + SoftDecay

Our SoftDecay Transofmer code is in this repo, including the AlBERT, BERT, DistilBERT, Roberta. Take BERT as an example, we modify the configuration file configuration_bert.py and the modeling_bert.py to insert the SoftDecay function. And the SoftDecay function is defined in soft_decay.py

3. Visualization

This code visualization.py is used to visualize the singular value distributions, i.e., the CDF and histgrams, as well as calculate the representation metrics.

Since SoftDecay is directly applied on output representation, and we can fix the function parameter $\alpha$ to define a decay function and get the transorformed representations for evaluation without training. (In supervised setting, $\alpha$ is trained under the task supervision to get better downstream task results.) This demo is mainly based on WhiteningBERT repo. We have added the softDecay post-process method in the unsupervisedSTS directory

#specify the PLTM/pooling method/layer_index/post_process(soft_decay or whitening)
python evaluation_stsbenchmark.py --pooling aver --encoder_name bert-base-cased --last2avg --post_process soft_decay

By changing the different PLTMs, you are suppposed to get the results below:

If you find our work useful, please cite as:

@inproceedings{
yan2022addressing,
title={Addressing Token Uniformity in Transformers via Singular Value Transformation},
author={Hanqi Yan and Lin Gui and Wenjie Li and Yulan He},
booktitle={The 38th Conference on Uncertainty in Artificial Intelligence},
year={2022},
url={https://openreview.net/forum?id=BtUxE_8i5l5}
}