Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Re: Junk NOT words

by kshay (Beadle)
on Oct 31, 2002 at 16:25 UTC ( #209427=note: print w/ replies, xml ) Need Help??


in reply to Re: Junk NOT words
in thread Junk NOT words

I wonder if you' d get better results by building a Markov chain based not on the frequency with which a letter follows another individual letter, but on the frequency with which a letter follows a given pair of letters.

I recently did something like this for a web application that goes the other direction: instead of trying to recognize letter combinations as "words," it generates "words" that "make sense" phonetically based on a given data set—in this case, lists of names. You can play with it here. Try a couple of different categories and you'll see that the resulting made-up names are quite phonetically distinctive; they really do seem French or Shakespearean (and within that, male or female) depending on what category you select.

I got the basic approach from this page by Chris Pound. I can't post the code because I wrote it for my employer, but basically I took each list of names, calculated the frequencies, and generated a Perl library containing a long series of assignments like this:

@{$pairs{'ri'}}{qw(a c e n s u)} = (3, 1, 1, 3, 1, 1);

That means that within this data set, "ri" is followed by "a" three times, "c" one time, etc.

Anyway, I'd be curious how this approach might work for word recognition, and how the results for "pair-letter" frequencies might differ from those obtained with "letter-letter" frequencies.


Comment on Re: Re: Junk NOT words

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://209427]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (7)
As of 2014-07-29 06:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (211 votes), past polls