Is Language infinite?

The headline question: Is language infinite? should perhaps invite more scrutiny than it's generally given these days. It was posited by Chomsky in the context of a particular view of language: "A small set of rules operating on a large but finite set of words, generates an infinite number of sentences." The problem with this definition is that it assumes a very bounded and discrete view of all elements: rules, lexical items, sentences. Language is not organized in sentences and the difference between words and rules is likely an artifact of dictionary making and grammar writing. But even if we did assume that, there is no guarantee, that the actual number of possible sentences is infinite rather than just unimaginably and practically inexhaustibly large (as was argued by Pullum and Scholtz in 2010). As a matter of practical fact, while the set of possible expressions in a given language may or may not be infinite, the actual set of all expressions ever uttered (even if we ignore language death) is going to be finite (because we cannot ignore the heat death of the universe). Now, the set of expressions actually produced by humans (or even machines) is going to be too large to practically enumerate by current (and possibly any) technology but it's going to be finite.

So it really does not matter whether language is infinite at all (other than to keep certain formal theories internally consistent). What matters is that any language is going to allow a set of expressions that is sufficiently large for any purpose a human language can be put towards.

However, this does not mean that the answers to your other questions are straightforward.

1. Can any language describe anything? Well, this depends on what you mean by describe. There are two ways you can approach it:
2. Exhaustive definition that will communicate between any two people all the relevant features of a phenomenon in such a way that cannot be confused with anything else?
3. Talk about it.

The answer to the first point is no, language cannot describe everything in a perfectly communicative manner. In fact, I'd argue that language cannot describe most things. However, the answer to the second way of putting the question is yes. Language can talk about everything. It uses deixis to point and analogy to compare. These ways are never definitive but they are good enough for language to carry the explicit cultural and interpersonal knowledge it does.

This issue is often tied to infinity of expressions. But that's a non-sequitur. We could have an infinity of expressions in a language that does not match a single item in an infinite set of phenomena.

2. Are our thoughts limited by our language? The problem with this question is that we don't really know what thoughts are (or really what language is). So the answer is yes, our language limits what we tend to think, but no language does not preclude the possibility of any thought (in the way we think of thoughts).

3. Can a finite set of words describe an infinite set of ideas. This makes several unwarranted assumptions. First, words don't describe ideas, they are used to create expressions (by rules if you subscribe to the words/rules theory of language or unification if you don't). Second, expressions do not really directly map onto a discrete set of ideas/notions/concepts. They co-create them. So in some way the question makes no sense. But even if you did subscribe to the simplistic notion of 'String of words'='Idea', you could generate enough strings in a language to match enough ideas to last humans the rest of the existence of Earth regardless of whether a language is finite or not.

4. Is there a core or reference set of vocabulary that can describe any concept? Again, remember all the caveats about the limits of 'description'. However, even with a simplistic view of what 'describe' means, the answer is that in practice there is not. However, different theoreticians have tried to come up with different sets of semantic primitives that can be used to describe anything. Sometimes this only applies to a subset of reality (logic, mathematics) but not necessarily. Anna Weirzbicka constructed a Natural semantic metalanguage that only consists of about 70 words found across all languages. She claims that these can be used redescribe any human concept. However, these descriptions are not very practical for every day use, even if you did subscribe to the very notion of semantic primes.

Note: There has also been some debate as to how large languages actually are, with some linguists claiming that language is actually not just infinite but uncountably infinite (see here for details of the discussion).