added cached tokenizer #35218
added cached tokenizer #35218
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Yup, large part of the overhead is gone now. I ran a quick check and you save roughly 1 second per batch in the open ASR leaderboard for whisper-turbo. I think there may still be some overhead from the tokenizer, but I'm not sure how much exactly. You should recompute the RTFx on the full test set. |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
Hey! We already have a state for the _added_tokens_encoder which should be filling this!
The main issue is just that when performances are required, _added_tokens_encoder should be "called" instead of added_tokens_encoder!
But very very welcome! Let's reduce the overhead!
| Returns the sorted mapping from string to index. The cache is dynamically invalidated if `_added_tokens_decoder` | ||
| has changed since the last computation. | ||
| """ | ||
| return {k.content: v for v, k in sorted(self._added_tokens_decoder.items(), key=lambda item: item[0])} |
There was a problem hiding this comment.
| return {k.content: v for v, k in sorted(self._added_tokens_decoder.items(), key=lambda item: item[0])} | |
| return self._added_tokens_encoder |
this would not work as you probably need the content to be sorted, which is why we have the non-sorted _added_tokens_encoder. We can actually sort it (define it as OrderedDict) as we deprecated python <= 3.9!
ArthurZucker
added
the
Core: Tokenization
|
Also you could compute with both slow and fast tokenizers, I don't know which you are using! |
5 participants
What does this PR do?
This PR introduces a caching mechanism for the added_tokens_encoder property in tokenizers to improve performance. Previously, the added_tokens_encoder mapping was recomputed every time the property was accessed, leading to redundant computation during tasks that frequently access it, such as tokenization or decoding.
Motivation and Context
The motivation for this change is to optimize tokenizer performance, especially in workflows that repeatedly access the added_tokens_encoder property. By caching the result, this PR reduces overhead and improves runtime efficiency without altering the existing behavior of the library.
Key changes:
The added_tokens_encoder mapping is now cached on the first access and reused for subsequent calls.
The caching mechanism is implemented in a way that is backward-compatible and avoids unnecessary recomputation.
Some benchmarks
Composite Results
Details
AMI Results
Earnings22 Results
Gigaspeech Results
LibriSpeech Clean Results
LibriSpeech Other Results
SPGISpeech Results
TEDLIUM Results
Voxpopuli Results
Benchmark scripts available there: https://github.com/huggingface/open_asr_leaderboard/tree/main/transformers
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@ArthurZucker
@Vaibhavs10
This changes was suggested by @pzelasko