Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

What's wrong with Storable

by PetaMem (Priest)
on Apr 16, 2004 at 18:15 UTC ( #345837=note: print w/ replies, xml ) Need Help??


in reply to Storable

Hi,

I can only wonder about the memory and CPU requirements...
Saving a bigger hash (ca. 515000 entries, where each key is about 7-10 chars long and each value is a Tree::Nary structure), results in a ca. 330MB file. Uncompressed that is.

Restoring it, takes an average of 3,8 times more memory than disk space => ca. 1,2GB!

Saving is weird, after all the 330MB have been saved, store takes another 2200 seconds of 100% CPU time on a P3 1,26GHz even after the complete file is already on disk.

Moreover, store seems to take when storing the same amount of memory as the data structure takes. in the aforementioned case over 2,1GB RAM are allocated! Gosh...

Bye
 PetaMem
    All Perl:   MT, NLP, NLU


Comment on What's wrong with Storable
Re: What's wrong with Storable
by eserte (Deacon) on Apr 16, 2004 at 23:50 UTC
    Restoring it, takes an average of 3,8 times more memory than disk space => ca. 1,2GB!
    This is not surprising --- Storable may store the data more space-efficient than it is in the RAM --- think of malloc overhead, necessary struct alignment, and the size of a SV*.

    Saving is weird, after all the 330MB have been saved, store takes another 2200 seconds of 100% CPU time on a P3 1,26GHz even after the complete file is already on disk.
    Maybe it's the global destruction which is causing the slowness. This may be due to an inefficient malloc. For example, it was reported that on FreeBSD, perl with the system malloc is very slow on deleting a large hash. Similar is true for certain Linux' malloc versions.
      Actually it is what I would call a design flaw of the Storable module, this was discussed (and resolved) in detail at the perl5 porters list. Expect a patch to storable RSN.

      Bye
       PetaMem
          All Perl:   MT, NLP, NLU

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://345837]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (8)
As of 2014-09-23 11:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (219 votes), past polls