中文说明 | English

Google and Stanford University released a new pre-trained model called ELECTRA, which has a much compact model size and relatively competitive performance compared to BERT and its variants. For further accelerating the research of the Chinese pre-trained model, the Joint Laboratory of HIT and iFLYTEK Research (HFL) has released the Chinese ELECTRA models based on the official code of ELECTRA. ELECTRA-small could reach similar or even higher scores on several NLP tasks with only 1/10 parameters compared to BERT and its variants.

This project is based on the official code of ELECTRA: https://github.com/google-research/electra

Chinese LERT | Chinese/English PERT Chinese MacBERT | Chinese ELECTRA | Chinese XLNet | Chinese BERT | TextBrewer | TextPruner

More resources by HFL: https://github.com/ymcui/HFL-Anthology

Mar 28, 2023 We open-sourced Chinese LLaMA&Alpaca LLMs, which can be quickly deployed on PC. Check: https://github.com/ymcui/Chinese-LLaMA-Alpaca

2022/10/29 We release a new pre-trained model called LERT, check https://github.com/ymcui/LERT/

2022/3/30 We release a new pre-trained model called PERT, check https://github.com/ymcui/PERT

2021/12/17 We release a model pruning toolkit - TextPruner, check https://github.com/airaria/TextPruner

Dec 13, 2020 We release Chinese legal ELECTRA series, check Download, Results for Crime Prediction.

Oct 22, 2020 ELECTRA-180g released, which were trained with high-quality CommonCrawl data, check Download.

Sep 15, 2020 Our paper "Revisiting Pre-Trained Models for Chinese Natural Language Processing" is accepted to Findings of EMNLP as a long paper.

ELECTRA provides a new pre-training framework, including two components: Generator and Discriminator.

• Generator: a small MLM that predicts [MASK] to its original token. The generator will replace some of the tokens in the input text.
• Discriminator: detect whether the input token is replaced. ELECTRA uses a pre-training task called Replaced Token Detection (RTD) instead of the Masked Language Model (MLM), which is used by BERT and its variants. Note that there is no Next Sentence Prediction (NSP) applied in ELECTRA.

After the pre-training stage, we only use Discriminator for finetuning the downstream tasks.

For more technical details, please check the paper: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

We provide TensorFlow models at the moment.

• ELECTRA-large, Chinese: 24-layer, 1024-hidden, 16-heads, 324M parameters
• ELECTRA-base, Chinese：12-layer, 768-hidden, 12-heads, 102M parameters
• ELECTRA-small-ex, Chinese: 24-layer, 256-hidden, 4-heads, 25M parameters
• ELECTRA-small, Chinese: 12-layer, 256-hidden, 4-heads, 12M parameters

If you need these models in PyTorch,

1. Convert TensorFlow checkpoint into PyTorch, using 🤗Transformers. Please use the script convert_electra_original_tf_checkpoint_to_pytorch.py provided by 🤗Transformers. For example,

python transformers/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py \
--tf_checkpoint_path ./path-to-large-model/ \
--config_file ./path-to-large-model/discriminator.json \
--pytorch_dump_path ./path-to-output/model.bin \
--discriminator_or_generator discriminator

2. Download from https://huggingface.co/hfl

Steps: select one of the model in the page above → click "list all files in model" at the end of the model page → download bin/json files from the pop-up window

The users from Mainland China are encouraged to use iFLYTEK Cloud download links, and the others may use Google Drive links. The ZIP package includes the following files (For example, ELECTRA-small, Chinese):

chinese_electra_small_L-12_H-256_A-4.zip
    |- checkpoint                           # checkpoint
    |- electra_small.data-00000-of-00001    # Model weights
    |- electra_small.meta                   # Meta info
    |- electra_small.index                  # Index info
    |- vocab.txt                            # Vocabulary

We use the same data for training RoBERTa-wwm-ext model series, which includes 5.4B tokens. We also use the same vocabulary from Chinese BERT, which has 21128 tokens. Other details and hyperparameter settings are listed below (others are remain default):

• ELECTRA-large: 24-layers, 1024-hidden, 16-heads, lr: 2e-4, batch: 96, max_len: 512, 2M steps
• ELECTRA-base: 12-layers, 768-hidden, 12-heads, lr: 2e-4, batch: 256, max_len: 512, 1M steps
• ELECTRA-small-ex: 24-layers, 256-hidden, 4-heads, lr: 5e-4, batch: 384, max_len: 512, 2M steps
• ELECTRA-small: 12-layers, 256-hidden, 4-heads, lr: 5e-4, batch: 1024, max_len: 512, 1M steps

With Huggingface-Transformers 2.8.0, the models in this repository could be easily accessed and loaded through the following codes.

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModel.from_pretrained(MODEL_NAME) 

The actual model and its MODEL_NAME are listed below.

Legal Version：

With PaddleHub, we can download and install the model with one line of code.

import paddlehub as hub
module = hub.Module(name=MODULE_NAME)

The actual model and its MODULE_NAME are listed below.

We compare our Chinese ELECTRA models with BERT-base、BERT-wwm, BERT-wwm-ext, RoBERTa-wwm-ext, RBT3 on six tasks.

• CMRC 2018 (Cui et al., 2019)：Span-Extraction Machine Reading Comprehension (Simplified Chinese)
• DRCD (Shao et al., 2018)：Span-Extraction Machine Reading Comprehension (Traditional Chinese)
• XNLI (Conneau et al., 2018)：Natural Langauge Inference
• ChnSentiCorp：Sentiment Analysis
• LCQMC (Liu et al., 2018)：Sentence Pair Matching
• BQ Corpus (Chen et al., 2018)：Sentence Pair Matching

For ELECTRA-small/base model, we use the learning rate of 3e-4 and 1e-4 according to the original paper. Note that we did NOT tune the hyperparameters w.r.t each task, so it is very likely that you will have better scores than ours. To ensure the stability of the results, we run 10 times for each experiment and report the maximum and average scores (in brackets).

CMRC 2018 dataset is released by the Joint Laboratory of HIT and iFLYTEK Research. The model should answer the questions based on the given passage, which is identical to SQuAD. Evaluation metrics: EM / F1

DRCD is also a span-extraction machine reading comprehension dataset, released by Delta Research Center. The text is written in Traditional Chinese. Evaluation metrics: EM / F1

We use XNLI data for testing the NLI task. Evaluation metric: Accuracy

We use ChnSentiCorp data for testing sentiment analysis. Evaluation metric: Accuracy

LCQMC is a sentence pair matching dataset, which could be seen as a binary classification task. Evaluation metric: Accuracy

BQ Corpus is a sentence pair matching dataset, which could be seen as a binary classification task. Evaluation metric: Accuracy

We adopt CAIL 2018 crime prediction to evaluate the performance of legal ELECTRA. Initial learning rates for small/base/large are 5e-4/3e-4/1e-4. Evaluation metric: Accuracy

Users may utilize the ELECTRA for fine-tuning their own tasks. Here we only illustrate the basic usage, and the users are encouraged to refer to the official guidelines as well.

In this tutorial, we will use ELECTRA-small model for finetuning CMRC 2018 task.

• data-dir：working directory
• model-name：model name, here we set as electra-small
• task-name：task name, here we set as cmrc2018. Our codes are adapted for all six tasks, where the task-names are cmrc2018，drcd，xnli，chnsenticorp，lcqmc，bqcorpus.

Download ELECTRA-small model from Download section, and unzip the files into ${data-dir}/models/${model-name}. The folder should contain five files, including electra_model.*, vocab.txt, checkpoint.

Download CMRC 2018 training and development data, and rename them as train.json, dev.json. Put two files into ${data-dir}/finetuning_data/${task-name} directory.

python run_finetuning.py \
    --data-dir ${data-dir} \
    --model-name ${model-name} \
    --hparams params_cmrc2018.json

The data-dir and model-name are illustrated in previous steps. hparams is a JSON dictionary. In this tutorial, params_cmrc2018.json includes the hyperparameter settings for finetuning.

{
    "task_names": ["cmrc2018"],
    "max_seq_length": 512,
    "vocab_size": 21128,
    "model_size": "small",
    "do_train": true,
    "do_eval": true,
    "write_test_outputs": true,
    "num_train_epochs": 2,
    "learning_rate": 3e-4,
    "train_batch_size": 32,
    "eval_batch_size": 32,
}

In this JSON file, we only listed some of the important hyperparameters. For all hyperparameter entries, please check configure_finetuning.py。

After running the program,

1. For machine reading comprehension tasks, the predicted JSON file cmrc2018_dev_preds.json will be saved in ${data-dir}/results/${task-name}_qa/. You can use evaluation script to get the final scores, such as python cmrc2018_drcd_evaluate.py dev.json cmrc2018_dev_preds.json
2. For text classification tasks, the accuracy will be printed on the screen right away, such as xnli: accuracy: 72.5 - loss: 0.67

Q: How to set learning rate in finetuning stage?
A: We recommend to use the learning rate in the paper as default (3e-4 for small, 1e-4 for base), and adjust according to your own task. Note that the initial learning rate may be higher than that in BERT or RoBERTa.

Q: Do you have PyTorch models?
A: Yes. You can check Download.

Q: Is it possible to share the training data?
A: I am sorry that it is not possible.

Q: Do you have any future plans?
A: Stay tuned!

If you find the technical report or resource is useful, please cite our work in your paper.

• Primary: https://ieeexplore.ieee.org/document/9599397

@journal{cui-etal-2021-pretrain,
  title={Pre-Training with Whole Word Masking for Chinese BERT},
  author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing},
  journal={IEEE Transactions on Audio, Speech and Language Processing},
  year={2021},
  url={https://ieeexplore.ieee.org/document/9599397},
  doi={10.1109/TASLP.2021.3124365},
 }

• Secondary: https://www.aclweb.org/anthology/2020.findings-emnlp.58

@inproceedings{cui-etal-2020-revisiting,
    title = "Revisiting Pre-Trained Models for {C}hinese Natural Language Processing",
    author = "Cui, Yiming  and
      Che, Wanxiang  and
      Liu, Ting  and
      Qin, Bing  and
      Wang, Shijin  and
      Hu, Guoping",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58",
    pages = "657--668",
}

Follow our official WeChat account to keep updated with our latest technologies!

Before you submit an issue:

• You are advised to read FAQ first before you submit an issue.
• Repetitive and irrelevant issues will be ignored and closed by [stable-bot](stale · GitHub Marketplace). Thank you for your understanding and support.
• We cannot acommodate EVERY request, and thus please bare in mind that there is no guarantee that your request will be met.
• Always be polite when you submit an issue.