Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Filtering out stop words

by talexb (Canon)
on Feb 25, 2020 at 13:17 UTC ( #11113398=note: print w/replies, xml ) Need Help??


in reply to Filtering out stop words

You're building a lookup cache .. there are many ways to do it -- I agree that using a hash is the best way. That being said, depending on how often you'll be using this logic, you might want to think about figuring out whether you load the hash every time you do a search, and how often you re-load, when new words are added. You could freeze and thaw your hash to improve start-up performance. You could even see if loading the words into a database (perhaps using sqlite3) would be a better solution. It's all a balancing act between performance and ease of use (when updating).

Actually, that's also a pretty cool interview question. :)

Alex / talexb / Toronto

Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

Replies are listed 'Best First'.
Re^2: Filtering out stop words
by Fletch (Chancellor) on Feb 25, 2020 at 15:27 UTC

    A bloom filter in front of an authoritative sqlite DB might be an interesting / fruitful path of exploration depending on the size of your word sets.

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11113398]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2020-03-29 19:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    To "Disagree to disagree" means to:









    Results (171 votes). Check out past polls.

    Notices?