Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^2: large perl module

by minek (Novice)
on Mar 06, 2010 at 23:54 UTC ( [id://827182]=note: print w/replies, xml ) Need Help??


in reply to Re: large perl module
in thread large perl module

Hi, this is very valuable post, I appreciate your input.
Yes, I tried Storable first, hash solution is faster.
I think it's worth to explore your solution with arrays, or some kind of trees in more detail.
Few explanations:
The hash structure us never rebuilt. It's static (use constant), the data never changes.
The issue is: to import a list of unknown phone numbers, and based on the area code and 3 or 4 digit prefix, determine the name of carrier for this phone number.
Carrier names (or rather IDs) are kept in a hash of hashes (two level) - that's the structure that is used for lookups, and is big in size. First 'index' level is area code, second one is 3 or 4 digit prefix.
Hash is used for lookups only. The phone data is stored somewhere else (it happens to be object data store based on Storable).
With hashes, the speed of the lookup is pretty good, I have no numbers but:
reading 500k of CSV with 12k phone numbers, parsing it, doing carrier lookup for each single number, removing redundant numbers and storing the numbers with carrier info in the data store takes now 14s, which is sufficient.

Replies are listed 'Best First'.
Re^3: large perl module
by BrowserUk (Patriarch) on Mar 07, 2010 at 01:22 UTC

    How many uniq carrier IDs are there?



      350 or so...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://827182]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others imbibing at the Monastery: (2)
As of 2024-04-20 04:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found