Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Re^2: large perl module

by techcode (Hermit)
on Mar 05, 2010 at 18:50 UTC ( #827040=note: print w/replies, xml ) Need Help??

in reply to Re: large perl module
in thread large perl module

It depends on the thing you are searching - that is how are you searching it. I've tried DBM::Deep for prefix search exactly because of the speed it's docs mention. However, when tested, even from a RAM disk (aka /dev/shm) it was at least two/tree orders of magnitude (not sure exactly) slower than loading up everything in HoHoH... and searching by going "up the tree".

Of course, I had the luxury or all requests going through one process that's doing prefix search and routing requests depending on it. Which might also be a solution for the original question.

You could implement the mod_perl handler only to pass on the requests, well put in work queue, and wait for results in results queue. And have one process (or several, possibly on several different servers) that do the actual (is it prefix?) searches - reading requests from the in/work queue, and putting them back in the out_queue.

The trie implementation was able to find something like 50K random prefixes per second in a loop - from the pool of 45 to 50K prefixes in the database (loaded on startup into large HoH trie). 50.000 per second should outperform any web server. And that's on x2 AMD with 1 GB or RAM ...

You could use Memcache for both input queue, and output - implementation (in Perl!) can be seen here:

PS - I've been contracting for the past year, for an company that's in SMS gateway business. So I get to tweak code that searches prefixes a lot ;)

Have you tried freelancing/outsourcing? Check out Scriptlance - I work there since 2003. For more info about Scriptlance and freelancing in general check out my home node.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://827040]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (9)
As of 2018-05-28 08:15 GMT
Find Nodes?
    Voting Booth?