Skip to content

Commit 2553923

Browse files
sangeet2020sangeet2020
committed
re-commiting changes and fixes
1 parent 51f60fc commit 2553923

5 files changed

Lines changed: 201 additions & 3 deletions

File tree

‎recipes/CommonVoice/ASR/CTC/README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,22 @@ It is important to note that CommonVoice initially offers mp3 audio files at 42H
99

1010
# Languages
1111
Here is a list of the different languages that we tested within the CommonVoice dataset and CTC:
12+
- English
13+
- German
1214
- French
15+
- Italian
1316
- Kinyarwanda
14-
- English
1517

1618
# Results
17-
1819
| Language | CommonVoice Release | hyperparams file | LM | Val. CER | Val. WER | Test CER | Test WER | HuggingFace link | Model link | GPUs |
1920
| ------------- |:-------------:|:---------------------------:| -----:| -----:| -----:| -----:| -----:| :-----------:| :-----------:| :-----------:|
2021
| English | 2020-12-11 | train_en_with_wav2vec.yaml | No | 5.01 | 12.57 | 7.32 | 15.58 | Not Avail. | [model](https://drive.google.com/drive/folders/1tYO__An68xrM5pR1UIXzEkwzvKX2Tz2o?usp=sharing) | 2xV100 32GB |
22+
| German | 2022-08-16 | train_de_with_wav2vec.yaml | No | 1.90 | 8.02 | 2.40 | 9.54 | Not Avail. | [model](https://drive.google.com/drive/folders/19G2Zm8896QSVDqVfs7PS_W86-K0-5xeC?usp=sharing) | 1xRTXA6000 48GB |
2123
| French | 2020-12-11 | train_fr_with_wav2vec.yaml | No | 2.60 | 8.59 | 3.19 | 9.96 | [model](https://huggingface.co/speechbrain/asr-wav2vec2-commonvoice-fr) | [model](https://drive.google.com/drive/folders/1T9DfdZwcNI9CURxhLCi8GA5JVz8adiY8?usp=sharing) | 2xV100 32GB |
2224
| Italian | 2020-12-11 | train_it_with_wav2vec.yaml | No | 2.77 | 9.83 | 3.16 | 10.85 | Not Avail. | [model](https://drive.google.com/drive/folders/1JhlxeA04tWg_vKcNChOoXSnjBe4luRby?usp=sharing) | 2xV100 32GB |
2325
| Kinyarwanda | 2020-12-11 | train_rw_with_wav2vec.yaml | No | 6.20 | 20.07 | 8.25 | 23.12 | Not Avail. | [model](https://drive.google.com/drive/folders/12_BDenvOqEERDZLAN-KdiAHklvuo35tx?usp=sharing) | 2xV100 32GB |
2426

27+
*For German, it takes around 5.5 hrs an epoch.* <br>
2528
The output folders with checkpoints and logs can be found [here](https://drive.google.com/drive/folders/11NMzY0zV-NqJmPMyZfC3RtT64bYe-G_O?usp=sharing).
2629

2730
## How to simply use pretrained models to transcribe my audio file?
@@ -48,3 +51,4 @@ Please, cite SpeechBrain if you use it for your research or business.
4851
note={arXiv:2106.04624}
4952
}
5053
```
54+
Footer

‎recipes/CommonVoice/ASR/CTC/hparams/train_de_with_wav2vec.yaml‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# Model: wav2vec2 + DNN + CTC
2+
# Augmentation: SpecAugment
3+
# Authors: Titouan Parcollet 2021
4+
# Mirco Ravanelli 2021
5+
# Sangeet Sagar 2022
6+
# ################################
7+
8+
# Seed needs to be set at top of yaml, before objects with parameters are made
9+
seed: 8200
10+
__set_seed: !apply:torch.manual_seed [!ref <seed>]
11+
output_folder: !ref results/wav2vec2_ctc_de/<seed>
12+
wer_file: !ref <output_folder>/wer.txt
13+
save_folder: !ref <output_folder>/save
14+
train_log: !ref <output_folder>/train_log.txt
15+
16+
# URL for the LARGE Fairseq German wav2vec2 model.
17+
wav2vec2_hub: facebook/wav2vec2-large-xlsr-53-german
18+
19+
# Dataset prep parameters
20+
data_folder: !PLACEHOLDER
21+
train_tsv_file: !ref <data_folder>/train.tsv
22+
dev_tsv_file: !ref <data_folder>/dev.tsv
23+
test_tsv_file: !ref <data_folder>/test.tsv
24+
accented_letters: True
25+
language: de
26+
train_csv: !ref <save_folder>/train.csv
27+
valid_csv: !ref <save_folder>/dev.csv
28+
test_csv: !ref <save_folder>/test.csv
29+
skip_prep: False
30+
31+
# We remove utterance slonger than 10s in the train/dev/test sets as
32+
# longer sentences certainly correspond to "open microphones".
33+
avoid_if_longer_than: 10.0
34+
35+
# Training parameters
36+
number_of_epochs: 45
37+
lr: 1.0
38+
lr_wav2vec: 0.0001
39+
sorting: ascending
40+
auto_mix_prec: False
41+
sample_rate: 16000
42+
ckpt_interval_minutes: 30 # save checkpoint every N min
43+
44+
# With data_parallel batch_size is split into N jobs
45+
# With DDP batch_size is multiplied by N jobs
46+
# Must be 6 per GPU to fit 16GB of VRAM
47+
batch_size: 8
48+
test_batch_size: 8
49+
dataloader_num_workers: 8
50+
test_num_workers: 8
51+
52+
dataloader_options:
53+
batch_size: !ref <batch_size>
54+
num_workers: !ref <dataloader_num_workers>
55+
test_dataloader_options:
56+
batch_size: !ref <test_batch_size>
57+
num_workers: !ref <test_num_workers>
58+
59+
# BPE parameters
60+
token_type: char # ["unigram", "bpe", "char"]
61+
character_coverage: 1.0
62+
63+
# Model parameters
64+
# activation: !name:torch.nn.LeakyReLU
65+
dnn_neurons: 1024
66+
wav2vec_output_dim: !ref <dnn_neurons>
67+
freeze_wav2vec: False
68+
dropout: 0.15
69+
70+
# Outputs
71+
output_neurons: 32 # BPE size, index(blank/eos/bos) = 0
72+
73+
# Be sure that the bos and eos index match with the BPEs ones
74+
blank_index: 0
75+
bos_index: 1
76+
eos_index: 2
77+
78+
# Functions and classes
79+
epoch_counter: !new:speechbrain.utils.epoch_loop.EpochCounter
80+
limit: !ref <number_of_epochs>
81+
82+
augmentation: !new:speechbrain.lobes.augment.TimeDomainSpecAugment
83+
sample_rate: !ref <sample_rate>
84+
speeds: [95, 100, 105]
85+
86+
enc: !new:speechbrain.nnet.containers.Sequential
87+
input_shape: [null, null, !ref <wav2vec_output_dim>]
88+
linear1: !name:speechbrain.nnet.linear.Linear
89+
n_neurons: !ref <dnn_neurons>
90+
bias: True
91+
bn1: !name:speechbrain.nnet.normalization.BatchNorm1d
92+
activation: !new:torch.nn.LeakyReLU
93+
drop: !new:torch.nn.Dropout
94+
p: !ref <dropout>
95+
linear2: !name:speechbrain.nnet.linear.Linear
96+
n_neurons: !ref <dnn_neurons>
97+
bias: True
98+
bn2: !name:speechbrain.nnet.normalization.BatchNorm1d
99+
activation2: !new:torch.nn.LeakyReLU
100+
drop2: !new:torch.nn.Dropout
101+
p: !ref <dropout>
102+
linear3: !name:speechbrain.nnet.linear.Linear
103+
n_neurons: !ref <dnn_neurons>
104+
bias: True
105+
bn3: !name:speechbrain.nnet.normalization.BatchNorm1d
106+
activation3: !new:torch.nn.LeakyReLU
107+
108+
wav2vec2: !new:speechbrain.lobes.models.huggingface_wav2vec.HuggingFaceWav2Vec2
109+
source: !ref <wav2vec2_hub>
110+
output_norm: True
111+
freeze: !ref <freeze_wav2vec>
112+
save_path: !ref <save_folder>/wav2vec2_checkpoint
113+
114+
#####
115+
# Uncomment this block if you prefer to use a Fairseq pretrained model instead
116+
# of a HuggingFace one. Here, we provide an URL that is obtained from the
117+
# Fairseq github for the multilingual XLSR.
118+
#
119+
#wav2vec2_url: https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt
120+
#wav2vec2: !new:speechbrain.lobes.models.fairseq_wav2vec.FairseqWav2Vec2
121+
# pretrained_path: !ref <wav2vec2_url>
122+
# output_norm: True
123+
# freeze: False
124+
# save_path: !ref <save_folder>/wav2vec2_checkpoint/model.pt
125+
126+
ctc_lin: !new:speechbrain.nnet.linear.Linear
127+
input_size: !ref <dnn_neurons>
128+
n_neurons: !ref <output_neurons>
129+
130+
log_softmax: !new:speechbrain.nnet.activations.Softmax
131+
apply_log: True
132+
133+
ctc_cost: !name:speechbrain.nnet.losses.ctc_loss
134+
blank_index: !ref <blank_index>
135+
136+
modules:
137+
wav2vec2: !ref <wav2vec2>
138+
enc: !ref <enc>
139+
ctc_lin: !ref <ctc_lin>
140+
141+
model: !new:torch.nn.ModuleList
142+
- [!ref <enc>, !ref <ctc_lin>]
143+
144+
model_opt_class: !name:torch.optim.Adadelta
145+
lr: !ref <lr>
146+
rho: 0.95
147+
eps: 1.e-8
148+
149+
wav2vec_opt_class: !name:torch.optim.Adam
150+
lr: !ref <lr_wav2vec>
151+
152+
lr_annealing_model: !new:speechbrain.nnet.schedulers.NewBobScheduler
153+
initial_value: !ref <lr>
154+
improvement_threshold: 0.0025
155+
annealing_factor: 0.8
156+
patient: 0
157+
158+
lr_annealing_wav2vec: !new:speechbrain.nnet.schedulers.NewBobScheduler
159+
initial_value: !ref <lr_wav2vec>
160+
improvement_threshold: 0.0025
161+
annealing_factor: 0.9
162+
patient: 0
163+
164+
checkpointer: !new:speechbrain.utils.checkpoints.Checkpointer
165+
checkpoints_dir: !ref <save_folder>
166+
recoverables:
167+
wav2vec2: !ref <wav2vec2>
168+
model: !ref <model>
169+
scheduler_model: !ref <lr_annealing_model>
170+
scheduler_wav2vec: !ref <lr_annealing_wav2vec>
171+
counter: !ref <epoch_counter>
172+
173+
train_logger: !new:speechbrain.utils.train_logger.FileTrainLogger
174+
save_file: !ref <train_log>
175+
176+
error_rate_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
177+
178+
cer_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
179+
split_tokens: True

‎recipes/CommonVoice/common_voice_prepare.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,18 @@ def create_csv(
242242
"[^’'A-Za-z0-9À-ÖØ-öø-ÿЀ-ӿéæœâçèàûî]+", " ", words
243243
).upper()
244244

245+
if language == "de":
246+
# this replacement helps preserve the case of ß
247+
# (and helps retain solitary occurrences of SS)
248+
# since python's upper() converts ß to SS.
249+
words = words.replace("ß", "0000ß0000")
250+
words = re.sub("[^’'A-Za-z0-9öÖäÄüÜß]+", " ", words).upper()
251+
words = words.replace("'", " ")
252+
words = words.replace("’", " ")
253+
words = words.replace(
254+
"0000SS0000", "ß"
255+
) # replace 0000SS0000 back to ß as its initial presence in the corpus
256+
245257
if language == "fr":
246258
# Replace J'y D'hui etc by J_ D_hui
247259
words = words.replace("'", " ")

‎speechbrain/processing/features.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -836,7 +836,9 @@ def forward(self, x):
836836

837837
# Derivative estimation (with a fixed convolutional kernel)
838838
delta_coeff = (
839-
torch.nn.functional.conv1d(x, self.kernel, groups=x.shape[1])
839+
torch.nn.functional.conv1d(
840+
x, self.kernel.to(x.device), groups=x.shape[1]
841+
)
840842
/ self.denom
841843
)
842844

‎tests/recipes.csv‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,3 +146,4 @@ recipe0144,ST,Tamasheq-French,recipes/IWSLT22_lowresource/train.py,recipes/IWSLT
146146
recipe0145,self-supervised-learning,LibriSpeech,recipes/LibriSpeech/self-supervised-learning/wav2vec2/train_sb_wav2vec2.py,recipes/LibriSpeech/self-supervised-learning/wav2vec2/hparams/wav2vec2_base.yaml,recipes/LibriSpeech/self-supervised-learning/wav2vec2/librispeech_prepare.py,recipes/LibriSpeech/self-supervised-learning/wav2vec2/README.md,,,,
147147
recipe0146,ASR,LibriSpeech,recipes/LibriSpeech/ASR/CTC/train_with_wav2vec.py,recipes/LibriSpeech/ASR/CTC/hparams/train_sb_wav2vec.yaml,recipes/LibriSpeech/ASR/CTC/librispeech_prepare.py,recipes/LibriSpeech/ASR/CTC/README.md,,,,
148148
recipe0147,ASR,LibriSpeech,recipes/LibriSpeech/ASR/CTC/train_with_wav2vec.py,recipes/LibriSpeech/ASR/CTC/hparams/train_hf_wav2vec.yaml,recipes/LibriSpeech/ASR/CTC/librispeech_prepare.py,recipes/LibriSpeech/ASR/CTC/README.md,,,,
149+
recipe0148,ASR,CommonVoice,recipes/CommonVoice/ASR/CTC/train_with_wav2vec.py,recipes/CommonVoice/ASR/CTC/hparams/train_de_with_wav2vec.yaml,recipes/CommonVoice/ASR/CTC/common_voice_prepare.py,recipes/CommonVoice/ASR/CTC/README.md,https://drive.google.com/drive/folders/19G2Zm8896QSVDqVfs7PS_W86-K0-5xeC?usp=sharing,,,

0 commit comments

Comments
 (0)