As Kaldi recipe develpment is converging, it's time to think about how we organize this text normalization as a post processing before WER calculation.

The processing is pretty simple, containing:

removal of "AH UM UH ER ERR HM ..." etc from both REF and HYP, to avoid error counting like
```
REF:AH THIS ... 
HYP:UH THIS ...
```
these words are just meaningless in WER computation, and it's really HARD to keep them consistent between human transcriber and models(say AH UH ER may sound identical, human-annotators/models may yield these things differently from time to time)
removal of "-" hyphen from both REF and HYP, to avoid error counting like:
```
REF: T-SHIRT
HYP: T SHIRT
```
hyphen is somehow more frequent than I expected, our training text TN kept hyphen because T-SHIRT is indeed a meaningful word other than T SHIRT. But in testing and evaluation, removing them gives more robust and reasonable WER numbers.

Google API has shown these processing can result a WER difference up to 1-2% absolute or even more, so this is necessary for consistent/fair comparison in spontaneous speech.

Now the problem is how we organize the post-processing:

the above normalization is just simple text processing, and it's not subjected to specific formats of asr result from different frameworks(such as kaldi hyp/ref text scp/ark), how do we release these processing to make it "standard" in gigaspeech downstream toolkits?

options that I can think of for now:

we can provide different text processing scripts for each downstream frameworks, e.g. GIGASPEECH_REPO/toolkits/{kaldi,espnet,...}/asr-text-post-processing.py. in this case, we KNOW the detailed format of each toolkits. With agreement, these can also go into downstream recipe code directly.
or we can have a single exampler python function/awk-command in GIGASPEECH_REPO/util/ that deals with pure text instead of dealing with specific formatted files, and let downstream recipe developers decide to import/refer in their code when they feel appropriate.

Which one is better? Any preferences or better suggestion? I personally vote for the first solution. @wangyongqing0731 @chenguoguo @sw005320

Adding scoring scripts. #24

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions