How to check for nonsense or gibberish in a text?

Question

I am getting a text string and I want to verify if the text is gibberish or not. By gibberish I mean text like "gfgsgsogjw" i.e. random keyboard presses. Currently I am using NTextCat to simply check if the language of the string matches the language I am expecting (German or English) and this works fine in a majority of the cases. However a small percentage of nonsense strings do manage to slip through the cracks. Is there an alternative way to check for gibberish text that will offer me higher accuracy?

Answer 1

You can use Hidden Markov Models to check the probabilty of sequence of letters in your input in concrete language. And then throw away least probabable ones. You can do the same with collocations of words. In both cases you will have to get a text corpus in every language you use to build a model upon it.