Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re: DBD::CSV eats memory

by jZed (Prior)
on Nov 16, 2007 at 02:17 UTC ( #651111=note: print w/ replies, xml ) Need Help??


in reply to DBD::CSV eats memory

I'm not sure what your test is supposed to illustrate or what you think memory has to do with speed in this case, but as the maintainer of DBD::CSV I can assure you that it was not built to handle a million rows quickly. Use SQLite or a full RDBMS for data sets of that size. That said, there are some big speed improvements coming to DBD::CSV soon (a new version of its SQL engine SQL::Statement). I'll announce it on this site when its ready.


Comment on Re: DBD::CSV eats memory
Re^2: DBD::CSV eats memory
by perlmonkdr (Beadle) on Nov 16, 2007 at 14:22 UTC

    Ok, thank, in this example I was tried to show how DBD::CSV eats memory in each loop, I was used a millon of rows becouse this's the way to see the increase of memory.

    Anyway, I'll try then with SQLite or DBM::Deep.

    Thank again for your help

      You wrote "DBD::CSV" eats memory in each loop". How do you know that? Your script doesn't do any memory testting or profiling or printouts. And if it's true, what OS are you on and what versions of the software and its prerequisites are you using? (your version of DBI, for example is more than 5 years old).

        Hello Jeff,

        Yes, i'm following the increase of memory with "Task Monitor" on Windows XP..

        I finally updated DBI to v1.601, yours SQL::Statement to v1.15 and DBD::CSV to v0.22 and Text::CSV_XS to v0.30 and updated this example too,to see a bigger increment, but the problem continue, for 1000 loop eats around of 10MB of memory in my case.

        Well, thk U.

        PD:Let me know if you need some test.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://651111]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (5)
As of 2014-07-10 05:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (199 votes), past polls