|Pathologically Eclectic Rubbish Lister|
Re: Junk NOT wordsby Molt (Chaplain)
|on Oct 30, 2002 at 17:04 UTC||Need Help??|
Not sure why you need this, but one possibility for an initial feel would be to look into building up a letter-by-letter Markov Chain from a large sample of text, and then use that to see how 'likely' the phrase is to be a word based on letter distribution. This will at least give you an idea of the Englishness of a phrase.
This would need playing with to see if it work reasonably, and to tune the scoring metric. This happily recognise Lewis Carroll-style nonsense as real words ('bewarethejabberwockmyson') though, which is bad if you do literally want to check it's real words, but good when you know your dictionary is potentially more limited than the words you're identifying.
One final warning.. beware of the three/four letter thing. It'll break on word boundaries unless you add all combinations letters possible where one word meets another. Better to treat these as penalties.