Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use async coroutines to speed up ngram probability lookup #84

Closed
pemistahl opened this issue Dec 12, 2020 · 1 comment
Closed

Use async coroutines to speed up ngram probability lookup #84

pemistahl opened this issue Dec 12, 2020 · 1 comment
Labels
enhancement New feature or request
Milestone
Lingua 1.1.0

Comments

@pemistahl
Copy link
Owner

pemistahl commented Dec 12, 2020 •
edited
Loading

Currently, loading a LanguageDetector's language models is relatively slow because it is done sequentially. Kotlin's coroutines should be the proper means to speed up this process.
@pemistahl pemistahl added the enhancement New feature or request label Dec 12, 2020
@pemistahl pemistahl added this to the Lingua 1.1.0 milestone Dec 12, 2020
pemistahl added a commit that referenced this issue Dec 12, 2020
@pemistahl
Use parallel streams for ngram probability lookup (#84)
10a4ba7
@pemistahl pemistahl changed the title Use parallel streams to speed up ngram probability lookup Use async coroutines to speed up ngram probability lookup Feb 15, 2021
pemistahl added a commit that referenced this issue Feb 15, 2021
@pemistahl
Replace parallel stream by coroutine (#84)
b7ae49f
pemistahl added a commit that referenced this issue Feb 15, 2021
@pemistahl
Replace parallel stream by coroutine (#84)
c08bfcf
pemistahl added a commit that referenced this issue Feb 15, 2021
@pemistahl
Replace parallel stream by coroutine (#84)
03c9b62
pemistahl added a commit that referenced this issue Feb 16, 2021
@pemistahl
Let multiple instances of LanguageDetector share same language models…
63e2532 
… in memory (#84)
@Marcono1234
Copy link
Contributor

The commit 03c9b62 fixing this might have introduced a thread-safety issue. Reassigning a local variable (here filteredLanguages) concurrently from different threads is not safe, see this StackOverflow answer.

@Marcono1234 Marcono1234 mentioned this issue May 31, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
Lingua 1.1.0
Development

No branches or pull requests
2 participants
@pemistahl @Marcono1234