Skip to content

Commit e043114

Browse files
asumagicasumagic
authored
Add results and links for GigaSpeech RNN-T model (speechbrain#2752)
* Provide initial results for GigaSpeech RNN-T model * Update conformer_transducer.yaml with final model hparams * Remove TODO for hparams update * Add notice about WER results
1 parent c4a4243 commit e043114

2 files changed

Lines changed: 22 additions & 19 deletions

File tree

‎recipes/GigaSpeech/ASR/transducer/README.md‎

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -44,24 +44,18 @@ If your GPU effectively supports fp16 (half-precision) computations, it is recom
4444
Enabling half precision can significantly reduce the peak VRAM requirements. For example, in the case of the Conformer Transducer recipe trained with GigaSpeech, the peak VRAM decreases from 39GB to 12GB when using fp16.
4545
According to our tests, the performance is not affected.
4646

47+
## Streaming model
48+
4749
# Results (non-streaming)
4850

4951
Results are obtained with beam search and no LM (no-streaming i.e. full context).
5052

51-
**TBD: The final models are currently in training.** This model has already been successfully trained, though. This will be updated when the checkpoints are ready for download.
52-
53-
<!--
54-
55-
| Language | Release | LM | Val. CER | Val. WER | Test CER | Test WER | Model link | GPUs |
56-
| ------------- |:-------------:| -----:| -----:| -----:| -----:| -----:| :-----------:| :-----------:|
5753

58-
-->
54+
| Release | LM | Val. CER | Val. WER | Test CER | Test WER | Model | GPUs |
55+
|:-------------:| -----:| --------:| --------:| --------:| --------:| :---------:|:-----------:|
56+
| 08-11-2024 | None | 6.09%\* | 11.75%\* | 6.14%\* | 11.97%\* | [Dropbox](https://www.dropbox.com/scl/fo/jg0vzm8l27o9qsixpqzjo/ABpKqmTMg24RVJKLY5Io1eU?rlkey=8z51y0gosme1fh4niahvi6b84&st=6smf7i5z&dl=0), [HuggingFace](https://huggingface.co/speechbrain/asr-streaming-conformer-gigaspeech) | 4x A100 80GB |
5957

60-
<!-- NOT READY YET: also update the following URL when uploaded
61-
62-
The output folders with checkpoints and logs can be found [here](). -->
63-
64-
## Streaming model
58+
\*: These results were obtained with our usual training scripts and are included for completeness, **but note that we have noticed an unexpected significant improvement to the error rate (see #2753) using the inference code path. Please refer to the table below for better and more accurate results.**
6559

6660
### WER vs chunk size & left context
6761

@@ -79,10 +73,16 @@ The left chunk size is not representative of the receptive field of the model.
7973
Because the model caches the streaming context at different layers, the model
8074
may end up forming indirect dependencies to audio many seconds ago.
8175

82-
| | full | cs=32 (1280ms) | 16 (640ms) | 8 (320ms) |
83-
|:-----:|:----:|:-----:|:-----:|:-----:|
76+
| | full | cs=32 (1280ms) | 24 (960ms) | 16 (640ms) | 12 (480ms) | 8 (320ms) |
77+
|:-----:|:------:|:------:|:------:|:------:|:------:|:------:|
78+
| full | 11.00% | - | - | - | - | - |
79+
| 16 | - | - | - | 11.70% | 11.84% | 12.14% |
80+
| 8 | - | - | 11.50% | 11.72% | 11.88% | 12.28% |
81+
| 4 | - | 11.40% | 11.53% | 11.81% | 12.03% | 12.64% |
82+
| 2 | - | 11.46% | 11.67% | 12.03% | 12.43% | 13.25% |
83+
| 1\*\* | - | 11.59% | 11.85% | 12.39% | 12.93% | 14.13% |
8484

85-
**TBD: The final models are currently in training.** This model has already been successfully trained, though. This will be updated when the checkpoints are ready for download.
85+
(\*\*: model was never explicitly trained with this setting)
8686

8787
### Inference
8888

‎recipes/GigaSpeech/ASR/transducer/hparams/conformer_transducer.yaml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,8 @@ json_file: !ref <data_folder>/GigaSpeech.json
4242
# The global batch size is computed as batch_size * n_gpus * grad_accumulation_factor.
4343
# Empirically, we found that this value should be >= 128.
4444
# Please, set your parameters accordingly.
45-
number_of_epochs: 10
46-
optimizer_step_limit: 400000
45+
number_of_epochs: 40 # limited by the step limit in practice
46+
optimizer_step_limit: 500000
4747
warmup_steps: 30000
4848
num_workers: 4
4949
batch_size_valid: 4
@@ -103,11 +103,14 @@ valid_dataloader_opts:
103103
test_dataloader_opts:
104104
batch_size: !ref <batch_size_valid>
105105

106-
# Using dynamic batching by default. This works with 48GB GPUs
106+
# Using dynamic batching by default. This was tuned for A100 80GB.
107107
# Or turn it off (but training speed will decrease)
108108
# Play with grad_accum_factor such that the total batch is around 600 to 1500 s.
109+
# You may have to adjust down the max_batch_length_train for GPUs with smaller
110+
# VRAM. The grad_accumulation_factor is tuned for 4x A100 80GB. You may have to
111+
# increase this factor if you are training on fewer GPUs or smaller batch sizes.
109112
dynamic_batching: True
110-
max_batch_length_train: 250
113+
max_batch_length_train: 500
111114
max_batch_length_val: 50 # we reduce it as the beam is much wider (VRAM)
112115
num_bucket: 200
113116
shuffle: True # if true re-creates batches at each epoch shuffling examples.

0 commit comments

Comments
 (0)