Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Using Word Tokens as Features

by igelkott (Priest)
on Apr 11, 2013 at 22:56 UTC ( [id://1028245]=note: print w/replies, xml ) Need Help??


in reply to Using Word Tokens as Features

Rather than building up the "columns" (almost certainly a "hash" in Perl) one new word at a time, I'd suggest that you start out with a reasonably complete corpus for the language/subject you wish to cover. This will make it a bit easier to maintain counts between messages and the number of zero columns for a particular message won't depend on the order it was processed (otherwise, your first message would have no zero columns).

As for the word counting task itself, look into split and hash. The basic procedure is rather simple but you'll need to decide how to handle case, hyphenation and maybe even stemming.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1028245]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others goofing around in the Monastery: (4)
As of 2025-11-13 17:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What's your view on AI coding assistants?





    Results (69 votes). Check out past polls.

    Notices?
    hippoepoptai's answer Re: how do I set a cookie and redirect was blessed by hippo!
    erzuuliAnonymous Monks are no longer allowed to use Super Search, due to an excessive use of this resource by robots.