Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: Junk NOT words

by jotti (Scribe)
on Nov 01, 2002 at 11:55 UTC ( #209684=note: print w/replies, xml ) Need Help??


in reply to Junk NOT words

I once made a program that analysed text and put up a frequency table for letter triplets. Using this table I then created nonsense text. A sample text file of 1 kB was enough to produce a table that could create some nonsense text that unmisstakeabely was English. Or German, Swedish or what ever language I sampled.

If you have sampled some megas of plain English text, you would have a pretty accurate table of triplet frequencies. Then you could sample another mega of text and pick sentences from that text. For each sentence, you pick each triplet and sum its frequency. You have to divide the sum with the number of triplets in the sentence, which gives you an average. Compute the average of all averages, and the standard deviation of the averages. This will give a picture of what is expected in real English. The standard deviation tells how reliable this method is. This method would OTOH recognise my nonsense English as real English.

Johan

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://209684]
help
Chatterbox?
[ambrus]: 1nickt: the tea didn't have sugar or honey or lemon or milk. or even caffeine in a significant quantity. it's really "colored water", warm water with some fruit smell to make it more pleasant than ordinary warm water.
[ambrus]: 1nickt: in theory, it shouldn't hurt the electronics, but also could temporarily cause problems until it dries, and the inside of the keyboard doesn't dry quickly, because it's closed.
[ambrus]: 1nickt: that's the theory. the theory is also that it should have dried by Sunday morning, but it didn't. so I dunno.
[ambrus]: I'll try today evening if it works now.

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (9)
As of 2017-11-21 14:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:













    Results (302 votes). Check out past polls.

    Notices?