Key Issues in LanguageContents ↓

Contents › Section 8: Corpora, tools and language technology › answer 11 of 22

Do standard corpus analysis tools like AntConc and Wordsmith work for foreign languages?

Answered 19 July 2015

Read this answer on Stack Exchange · See the question

These are two separate questions.

  1. Will AntConc and Wordsmith deal with characters not present in English (sometimes referred to as non-ASCII)? This is really a question about encoding of text (not language). AntConc supports unicode UTF-8 which means it should deal with any script. WordSmith only supports a limited subset which means that texts in non-Latin scripts will have to be converted. But none of the examples you give will present any problems. This is a relatively easy issue to resolve depending on the encoding of the texts you provide.

  2. The question about dealing with morphology is one of lemmas (which are essentialy word stems without any inflection). Both Wordsmith and AntConc support lemmas but you will have to provide a lemma list for each language. There are guides online on how to do it and a pre-made lemma list for English also exists. A good source for data for generating a lemma list for other languages are the many open source spell checks - but it will require work and coding to create those.

© Dominik Lukeš. Licensed under CC BY-SA 3.0. Originally published on Linguistics Stack Exchange, attributed there to Dominik Lukes (profile).