Recent advances in AI enable smarter applications based on texts. It's good but they are mostly in English due to its abundance of texts available from the Internet.

This repository explores LM in Cantonese (Yue Chinese, 廣東話), a langauge predominantly spoken in Guangzhou, Hong Kong and Macau, and containing very challenging lingual properties for AI to learn.

AI 喺呢幾年發展得好快，好多嘢都話用 AI 處理會醒好多，但其實喺「語言處理」嘅領域入面，好多嘅資源都只係得英文，所以要落手做廣東話嘅 NLP，其實唔容易。

所以諗住喺呢度開個 Repo ，鼓勵更多人開發廣東話 AI。

• Mixed Languages (English, Chinese, Yue)
  夾雜多種語言
• Complex Syntax
  語法複雜
• Scarce Resource
  資源稀少
• Many Homonyms & Homophones in online texts
  網上嘅字通常有好多一語多義／同音異字

We adopt the following preprocessing to the model:
用呢個 model 前我哋會對文字做一啲嘅處理：

• WordPiece Tokenizer from forked 🤗Tokenizers which,

  • strips accents like the original BERT
    除去組合附加符號 (e.g. à → a)

  • uses lower casing
    使用細階英文

  • treats symbols/numers as a separate token
    符號／數字全部當係一個 token

  • Simplified Chinese → Traditional Chinese (Since most of our corpus are in Trad. Chinese)
    簡轉繁（因為文本大部分都係繁體字）

    Using OpenCC v1.1.1 from here

  • normalizes Unicode Characters (Some are hand-crafted) by
    統一中文字符（其中一啲係人手分類）

    • Symbols of the same functionality 相同功能嘅符號 (e.g. 【 → [ )
    • Variant Chinese characters 異體字 (e.g. 俢 → 修 )
    • Deomposing rare characters 將罕見字拆開 (e.g. 偆 → 亻春 )

    (Mapping here)

• Newlines are regarded as a token, i.e. <nl>

• Tensorflow
• Pytorch

• OpenCC (Simpl-to-Trad, 簡轉繁) @ v1.1.1
• 🤗Tokenizers (forked version is used for normalization)

# Installing OpenCC v1.1.1 by
sudo bash ./install_opencc.sh

# Installing by forked 🤗 Tokenizers by 
pip3 install 'git+https://github.com/ecchochan/tokenizers.git@zh-norm-4#egg=version_subpkg&subdirectory=bindings/python'
# This takes some time!

# This is forked from [email protected]
# with python package renamed to tokenizers_zh

Since we have NO datasets in Cantonese, we evaluate the models on both English and Chinese datasets:

• MNLI (Entailment Prediction)
• DRCD (Reading Comprehension)
• SQuAD-v2 (Reading Comprehension)
• CMRC2018 (Reading Comprehension)

1. Sentence Order Prediction (SOP)

   SOP is a pretraining objective that is used in Albert. StructBERT also introduces Sentence Structural Objective, but since the code for electra reads the data sequentially, this repo explores SOP first

2. Cluster Objective

   DocProduct is a cool project training a BERT model to cluster similar Q&A -- if a text A answers the question Q, then Q and A will be close in vector representation.

   This means the model must predict the possible contexts (before and after) in order to embed a vector that can minimize the cost function

   Details refer to the DocProduct repo.

• Normalize Chinese characters
• ELECTRA-small
• ELECTRA-base
• ELECTRA-base-sop
• ELECTRA-albert-base
• ELECTRA-albert-xlarge
• ELECTRA-base-cluster
• ELECTRA-large
• Evaluation in Cantonese dataset
• Upload to 🤗Huggingface

Electra checkpoints are put here in Google Drive.

Electra-albert checkpoints are here in Google Drive

google-research/electra#3

Special thanks to Google's TensorFlow Research Cloud (TFRC) for providing TPU-v3 for all the training in this repo!