|Problems? Is your data what you think it is?|
Junk NOT wordsby Anonymous Monk
|on Oct 30, 2002 at 16:49 UTC||Need Help??|
Anonymous Monk has asked for the
wisdom of the Perl Monks concerning the following question:
I am trying to think my way through the problem of knowing when an arbitrary sting of characters is just junk instead of any arbitrary combination of words with all pounctuation removed.
For instance -- I want a script to recognize:
hudjheuhfnnvgxvbchnfhfujryyfgbch is not a phrase
but: whererangesarealltherage is a nonsensical phrase
My one thought is to try to break it down into bits using verbs to help in the process. One could than state that a string of consenants longer than n characters cannot be a word. Having done that, one could do a dictionary search on the bits containing volwels to see if they appear in words -- or better still, create a dictionary of three or four character bits which could be parts of actual words.
The only three or four character bit I can quickly see in the first arbitary string that is part of a word is "hud"
Perhaps there are better suggestions for solving this problem.