Support disabling loading of quadrigram and fivegram models #136
Conversation
| * language detection. This affects both dynamically loaded models as well as | ||
| * [preloaded models][withPreloadedLanguageModels]. |
There was a problem hiding this comment.
Maybe the wording "dynamically loaded models as well as preloaded models" is a bit misleading. It might not be clear enough what this means by "dynamically loaded", and it might sound as if even a detector with preloaded models could dynamically load models.
Any suggestions for alternative wordings, or is this sentence fine?
|
Marked this as draft again because the usage of ngrams, and specifically the usage of quadrigrams and fivegrams, is probably an implementation detail and I am not sure if it is a good idea to expose this in the public API. Would also be interesting to know how useful such a method would be to users of this library. |
|
Hi @Marcono1234, thank you for this useful idea. I've actually implemented this a bit differently than you. There is now In the accuracy reports and plots, you now find separate statistics for what I call low accuracy mode (trigrams only) and high accuracy mode (all ngrams). |
Marcono1234
deleted the
marcono1234/builder-disable-quadri-fivegram-loading
branch
Relates to #101
Adds the function
LanguageDetectorBuilder.withoutQuadrigramAndFivegramModels()which disables loading of quadrigram and fivegram models. Quadrigram and fivegram models take up the majority of memory during runtime; if my measurements are correct, all language models preloaded require ~1783 MB, whereas only unigram, bigram and trigram models require ~110 MB. However, for larger textsLanguageDetectordoes not actually use them.Therefore, for use cases where it is known beforehand that most or all texts will be longer than ~120 chars, it should be relatively safe to disable of loading quadrigram and fivegram models.
Any feedback, especially regarding the builder function name and documentation, is appreciated.