Contents › Section 8: Corpora, tools and language technology › answer 11 of 22
Do standard corpus analysis tools like AntConc and Wordsmith work for foreign languages?
Read this answer on Stack Exchange · See the question
These are two separate questions.
-
Will AntConc and Wordsmith deal with characters not present in English (sometimes referred to as non-ASCII)? This is really a question about encoding of text (not language). AntConc supports unicode UTF-8 which means it should deal with any script. WordSmith only supports a limited subset which means that texts in non-Latin scripts will have to be converted. But none of the examples you give will present any problems. This is a relatively easy issue to resolve depending on the encoding of the texts you provide.
-
The question about dealing with morphology is one of lemmas (which are essentialy word stems without any inflection). Both Wordsmith and AntConc support lemmas but you will have to provide a lemma list for each language. There are guides online on how to do it and a pre-made lemma list for English also exists. A good source for data for generating a lemma list for other languages are the many open source spell checks - but it will require work and coding to create those.
© Dominik Lukeš. Licensed under CC BY-SA 3.0. Originally published on Linguistics Stack Exchange, attributed there to Dominik Lukes (profile).