• this library tries to solve language detection of very short words and phrases, even shorter than tweets
• makes use of both statistical and rule-based approaches
• outperforms Apache Tika, Apache OpenNLP and Optimaize Language Detector for more than 70 languages
• works within every Java 6+ application and on Android
• no additional training of language models necessary
• api for adding your own language models
• offline usage without having to connect to an external service or API
• can be used in a REPL for a quick try-out

1. What does this library do?
2. Why does this library exist?
3. Which languages are supported?
4. How good is it?
5. Why is it better than other libraries?
6. Test report generation
7. How to add it to your project?
   7.1 Using Gradle
   7.2 Using Maven
8. How to build?
9. How to use?
   9.1 Programmatic use
   9.2 Standalone mode
10. How to add new languages?
11. Do you want to contribute?
12. What's next for upcoming versions?

Its task is simple: It tells you which language some provided textual data is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.

Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.

So far, three other comprehensive open source libraries working on the JVM for this task are Apache Tika, Apache OpenNLP and Optimaize Language Detector. Unfortunately, especially the latter has three major drawbacks:

1. Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, it doesn't provide adequate results.
2. The more languages take part in the decision process, the less accurate are the detection results.
3. Configuration of the library is quite cumbersome and requires some knowledge about the statistical methods that are used internally.

Lingua aims at eliminating these problems. It nearly doesn't need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. It draws on both rule-based and statistical methods but does not use any dictionaries of words. It does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, the following 74 languages are supported:

• A
  • Afrikaans
  • Albanian
  • Arabic
  • Armenian
  • Azerbaijani
• B
  • Basque
  • Belarusian
  • Bengali
  • Norwegian Bokmal
  • Bosnian
  • Bulgarian
• C
  • Catalan
  • Chinese
  • Croatian
  • Czech
• D
  • Danish
  • Dutch
• E
  • English
  • Esperanto
  • Estonian
• F
  • Finnish
  • French
• G
  • Ganda
  • Georgian
  • German
  • Greek
  • Gujarati
• H
  • Hebrew
  • Hindi
  • Hungarian
• I
  • Icelandic
  • Indonesian
  • Irish
  • Italian
• J
  • Japanese
• K
  • Kazakh
  • Korean
• L
  • Latin
  • Latvian
  • Lithuanian
• M
  • Macedonian
  • Malay
  • Marathi
  • Mongolian
• N
  • Norwegian Nynorsk
• P
  • Persian
  • Polish
  • Portuguese
  • Punjabi
• R
  • Romanian
  • Russian
• S
  • Serbian
  • Shona
  • Slovak
  • Slovene
  • Somali
  • Sotho
  • Spanish
  • Swahili
  • Swedish
• T
  • Tagalog
  • Tamil
  • Telugu
  • Thai
  • Tsonga
  • Tswana
  • Turkish
• U
  • Ukrainian
  • Urdu
• V
  • Vietnamese
• W
  • Welsh
• X
  • Xhosa
• Y
  • Yoruba
• Z
  • Zulu

Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:

1. a list of single words with a minimum length of 5 characters
2. a list of word pairs with a minimum length of 10 characters
3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the Wortschatz corpora offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua, Apache Tika, Apache OpenNLP and Optimaize Language Detector using parameterized JUnit tests running over the data of Lingua's supported 74 languages. Languages that are not supported by the other libraries are simply ignored for those during the detection process.

The box plot below shows the distribution of the averaged accuracy values for all three performed tasks: Single word detection, word pair detection and sentence detection. Lingua clearly outperforms its contenders. Bar plots for each language and further box plots for the separate detection tasks can be found in the file ACCURACY_PLOTS.md. Detailed statistics including mean, median and standard deviation values for each language and classifier are available in the file ACCURACY_TABLE.md.

Every language detector uses a probabilistic n-gram model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

If you want to reproduce the accuracy results above, you can generate the test reports yourself for all four classifiers and all languages by doing:

./gradlew writeAccuracyReports

You can also restrict the classifiers and languages to generate reports for by passing arguments to the Gradle task. The following task generates reports for Lingua and the languages English and German only:

./gradlew writeAccuracyReports -Pdetectors=Lingua -Planguages=English,German

By default, only a single CPU core is used for report generation. If you have a multi-core CPU in your machine, you can fork as many processes as you have CPU cores. This speeds up report generation significantly. However, be aware that forking more than one process can consume a lot of RAM. You do it like this:

./gradlew writeAccuracyReports -PcpuCores=2

For each detector and language, a test report file is then written into /accuracy-reports, to be found next to the src directory. As an example, here is the current output of the Lingua German report:

com.github.pemistahl.lingua.report.lingua.GermanDetectionAccuracyReport

##### GERMAN #####

>>> Accuracy on average: 89,10%

>> Detection of 1000 single words (average length: 9 chars)
Accuracy: 73,60%
Erroneously classified as DUTCH: 2,30%, ENGLISH: 2,10%, DANISH: 2,10%, LATIN: 2,00%, BOKMAL: 1,60%, ITALIAN: 1,20%, BASQUE: 1,20%, FRENCH: 1,20%, ESPERANTO: 1,10%, SWEDISH: 1,00%, AFRIKAANS: 0,80%, TSONGA: 0,70%, PORTUGUESE: 0,60%, NYNORSK: 0,60%, FINNISH: 0,50%, YORUBA: 0,50%, ESTONIAN: 0,50%, WELSH: 0,50%, SOTHO: 0,50%, SPANISH: 0,40%, SWAHILI: 0,40%, IRISH: 0,40%, ICELANDIC: 0,40%, POLISH: 0,40%, TSWANA: 0,40%, TAGALOG: 0,30%, CATALAN: 0,30%, BOSNIAN: 0,30%, LITHUANIAN: 0,20%, INDONESIAN: 0,20%, ALBANIAN: 0,20%, SLOVAK: 0,20%, ZULU: 0,20%, CROATIAN: 0,20%, ROMANIAN: 0,20%, XHOSA: 0,20%, TURKISH: 0,10%, LATVIAN: 0,10%, MALAY: 0,10%, SLOVENE: 0,10%, SOMALI: 0,10%

>> Detection of 1000 word pairs (average length: 18 chars)
Accuracy: 94,00%
Erroneously classified as DUTCH: 0,90%, LATIN: 0,80%, ENGLISH: 0,70%, SWEDISH: 0,60%, DANISH: 0,50%, FRENCH: 0,40%, BOKMAL: 0,30%, TAGALOG: 0,20%, IRISH: 0,20%, SWAHILI: 0,20%, TURKISH: 0,10%, ZULU: 0,10%, ESPERANTO: 0,10%, ESTONIAN: 0,10%, FINNISH: 0,10%, ITALIAN: 0,10%, NYNORSK: 0,10%, ICELANDIC: 0,10%, AFRIKAANS: 0,10%, SOMALI: 0,10%, TSONGA: 0,10%, WELSH: 0,10%

>> Detection of 1000 sentences (average length: 111 chars)
Accuracy: 99,70%
Erroneously classified as DUTCH: 0,20%, LATIN: 0,10%

>> Exact values: 89.10000000000001 73.6 94.0 99.7

The plots have been created with Python and the libraries Pandas, Matplotlib and Seaborn. If you have a global Python 3 installation and the python3 command available on your command line, you can redraw the plots after modifying the test reports by executing the following Gradle task:

./gradlew drawAccuracyPlots

The detailed table in the file ACCURACY_TABLE.md containing all accuracy values can be written with:

./gradlew writeAccuracyTable

Lingua is hosted on Jcenter and Maven Central.

// Groovy syntax
implementation 'com.github.pemistahl:lingua:1.0.0'

// Kotlin syntax
implementation("com.github.pemistahl:lingua:1.0.0")

<dependency>
    <groupId>com.github.pemistahl</groupId>
    <artifactId>lingua</artifactId>
    <version>1.0.0</version>
</dependency>

Lingua uses Gradle to build.

git clone https://github.com/pemistahl/lingua.git
cd lingua
./gradlew build

Several jar archives can be created from the project.

1. ./gradlew jar assembles lingua-1.0.0.jar containing the compiled sources only.
2. ./gradlew sourcesJar assembles lingua-1.0.0-sources.jar containing the plain source code.
3. ./gradlew jarWithDependencies assembles lingua-1.0.0-with-dependencies.jar containing the compiled sources and all external dependencies needed at runtime. This jar file can be included in projects without dependency management systems. You should be able to use it in your Android project as well by putting it in your project's lib folder. This jar file can also be used to run Lingua in standalone mode (see below).

Lingua can be used programmatically in your own code or in standalone mode.

The API is pretty straightforward and can be used in both Kotlin and Java code.

/* Kotlin */

import com.github.pemistahl.lingua.api.*
import com.github.pemistahl.lingua.api.Language.*

val detector: LanguageDetector = LanguageDetectorBuilder.fromLanguages(ENGLISH, FRENCH, GERMAN, SPANISH).build()
val detectedLanguage: Language = detector.detectLanguageOf(text = "languages are awesome")

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word prologue, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated in the following way:

val detector = LanguageDetectorBuilder
    .fromAllLanguages()
    .withMinimumRelativeDistance(0.25) // minimum: 0.00 maximum: 0.99 default: 0.00
    .build()

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high. Otherwise you will get most results returned as Language.UNKNOWN which is the return value for cases where language detection is not reliably possible.

The public API of Lingua never returns null somewhere, so it is safe to be used from within Java code as well.

/* Java */

import com.github.pemistahl.lingua.api.*;
import static com.github.pemistahl.lingua.api.Language.*;

final LanguageDetector detector = LanguageDetectorBuilder.fromLanguages(ENGLISH, FRENCH, GERMAN, SPANISH).build();
final Language detectedLanguage = detector.detectLanguageOf("languages are awesome");

There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance (what a surprise :-). The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages:

// include all languages available in the library
// WARNING: in the worst case this produces high memory 
//          consumption and slow runtime performance
LanguageDetectorBuilder.fromAllLanguages()

// include only languages that are not yet extinct (= currently excludes Latin)
LanguageDetectorBuilder.fromAllSpokenLanguages()

// include only languages written with Cyrillic script
LanguageDetectorBuilder.fromAllLanguagesWithCyrillicScript()

// exclude only the Spanish language from the decision algorithm
LanguageDetectorBuilder.fromAllLanguagesWithout(Language.SPANISH)

// only decide between English and German
LanguageDetectorBuilder.fromLanguages(Language.ENGLISH, Language.GERMAN)

// select languages by ISO 639-1 code
LanguageDetectorBuilder.fromIsoCodes639_1(IsoCode639_1.EN, IsoCode639_3.DE)

// select languages by ISO 639-3 code
LanguageDetectorBuilder.fromIsoCodes639_3(IsoCode639_3.ENG, IsoCode639_3.DEU)

Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? These questions can be answered as well:

val detector = LanguageDetectorBuilder.fromLanguages(GERMAN, ENGLISH, FRENCH, SPANISH).build()
val confidenceValues = detector.computeLanguageConfidenceValues(text = "Coding is fun.")

// {
//   ENGLISH=1.0, 
//   GERMAN=0.8665738136456169, 
//   FRENCH=0.8249537317466078, 
//   SPANISH=0.7792362923625288
// }

In the example above, a map of all possible languages is returned, sorted by their confidence value in descending order. The values that the detector computes are part of a relative confidence metric, not of an absolute one. Each value is a number between 0.0 and 1.0. The most likely language is always returned with value 1.0. All other languages get values assigned which are lower than 1.0, denoting how less likely those languages are in comparison to the most likely language.

The map returned by this method does not necessarily contain all languages which the calling instance of LanguageDetector was built from. If the rule-based engine decides that a specific language is truly impossible, then it will not be part of the returned map. The confidence value for such a language is assumed to be 0.0.

If you want to try out Lingua before you decide whether to use it or not, you can run it in a REPL and immediately see its detection results.

1. With Gradle: ./gradlew runLinguaOnConsole --console=plain
2. Without Gradle: java -jar lingua-1.0.0-with-dependencies.jar

Then just play around:

This is Lingua.
Select the language models to load.

1: enter language iso codes manually
2: all supported languages

Type a number and press <Enter>.
Type :quit to exit.

> 1
List some language iso 639-1 codes separated by spaces and press <Enter>.
Type :quit to exit.

> en fr de es
Loading language models...
Done. 4 language models loaded lazily.

Type some text and press <Enter> to detect its language.
Type :quit to exit.

> languages
ENGLISH
> Sprachen
GERMAN
> langues
FRENCH
> :quit
Bye! Ciao! Tschüss! Salut!

If you want to add your own languages to Lingua, you can do so. The following steps describe how to proceed.

1. Clone Lingua's repository to your own computer as described in section 8.
2. Open enums IsoCode639_1 and IsoCode639_3 and add the language's iso codes. Among other sites, Wikipedia provides a comprehensive list.
3. Open enum Language and add a new entry for your language. If the language is written with a script that is not yet supported by Lingua's Alphabet enum, then add a new entry for it there as well.
4. If your language's script contains characters that are completely unique to it or that are shared by only a small subset of Lingua's supported languages, then add them either to the respective entry in the Language enum or to the CHARS_TO_LANGUAGES_MAPPING constant in class LanguageDetector.
5. Use LanguageModelFilesWriter to create the language model files. The training data file used for ngram probability estimation is not required to have a specific format other than to be a valid txt file.
6. Create a new subdirectory in /src/main/resources/language-models and put the generated language model files in there. Do not rename the language model files. The name of the subdirectory must be the language's ISO 639-1 code, completely lowercased.
7. Use TestDataFilesWriter to create the test data files used for accuracy report generation. The input file from which to create the test data should have each sentence on a separate line.
8. Put the generated test data files in /src/test/resources/language-testdata. Do not rename the test data files.
9. For accuracy report generation, create an abstract base class for the main logic in /src/test/kotlin/com/github/pemistahl/lingua/report/config. Look at the other languages' files in this directory to see how the class must look like. It should be pretty self-explanatory.
10. Create a concrete test class in /src/test/kotlin/com/github/pemistahl/lingua/report/lingua. Look at the other languages' files in this directory to see how the class must look like. It should be pretty self-explanatory. If one of the other language detector libraries supports your language already, you can add test classes for those as well. Each library has its own directory for this purpose.
11. Fix the existing unit tests by adding your new language.
12. Add your new language to property linguaSupportedLanguages in /gradle.properties.
13. Be happy! :-) Run the accuracy report generation and see how well Lingua (hopefully) performs.

In case you want to contribute something to Lingua, then I encourage you to do so. Do you have ideas for improving the API? Are there some specific languages that you want to have supported early? Or have you found any bugs so far? Feel free to open an issue or send a pull request. It's very much appreciated. :-)

Take a look at the planned issues.