Should common short words be left out of a concordance?

There is no rule for what should or should not be included in a concordance. Historically, most printed concordances exclude function words because they are very frequent and their meaning is rarely elucidated by reference to any one word in context. They will probably also not include many common content words not deemed significant by the compilers (e.g. go, sit, watch, etc.)

However, a modern computer-generated and computer-navigated concordances do not have any practical restrictions, so function words are more likely to be included. As result, in corpus linguistics, the term concordance has acquired a more generic meaning than it has had historically, i.e. list of all words (or strings of letters between spaces) in a text.

Nevertheless, most concordancing software will include a facility for stop words, i.e. a list of words to exclude from the index. This can be very useful for all sorts of purposes - e.g. calculating semantic density or looking for most frequent content words. Lists of commonly used stop words exist for many languages. Here's one for English: http://www.ranks.nl/stopwords.

In short, it all depends on what the purpose of your concordance is.