How to assess the language register of speech?

Flesch-Kincaid, SMOG etc. are measures of text complexity and not register. They only work on written language of certain length (textbooks, newspaper articles, etc.) Register can be inferred from a single word. However, lower readability will probably correlate quite well with high formality. But not in a way that you can simply translate the scores.

The problem with asking for differentiation of register is that there is no objective definition of the different registers (levels of formality). So you'd have to come up with some exemplars of what the different registers represent. You'll also need some measure of correctness (how do you know your results match the levels). Is the appropriateness to the situation? Socio-economic status of the speaker? All of those in turn are hard to measure.

The efforts of measuring levels of formality or register made by people like Halliday or Biber use proxies (like presence of certain words) and get pretty good results. But their measures may not be your measures. Other proxies for levels of formality is the proportion of deictics like pronouns or determiners which can then be used as input for machine learning.

So to get a reasonable answer, you will need to specify what exactly you're trying to achieve and for what purpose? What are your resources for achieving that? If you're looking for a simple piece of software you can load a sample in and get a result, you'll be disappointed. Have a look at this paper which outlines some of the previous efforts in this area. It shows that if your needs are quite general, you can get decent broad results that can improve subsequent analysis. But if you want to get a single score for an individual text, you will probably not get very reliable results.