Re: DBD::CSV eats memory

[jZed]

I'm not sure what your test is supposed to illustrate or what you think memory has to do with speed in this case, but as the maintainer of DBD::CSV I can assure you that it was not built to handle a million rows quickly. Use SQLite or a full RDBMS for data sets of that size. That said, there are some big speed improvements coming to DBD::CSV soon (a new version of its SQL engine SQL::Statement). I'll announce it on this site when its ready.

[perlmonkdr]

Ok, thank, in this example I was tried to show how DBD::CSV eats memory in each loop, I was used a millon of rows becouse this's the way to see the increase of memory.

Anyway, I'll try then with SQLite or DBM::Deep.

Thank again for your help

[jZed]

You wrote "DBD::CSV" eats memory in each loop". How do you know that? Your script doesn't do any memory testting or profiling or printouts. And if it's true, what OS are you on and what versions of the software and its prerequisites are you using? (your version of DBI, for example is more than 5 years old).

[perlmonkdr]

Hello Jeff,

Yes, i'm following the increase of memory with "Task Monitor" on Windows XP..

I finally updated DBI to v1.601, yours SQL::Statement to v1.15 and DBD::CSV to v0.22 and Text::CSV_XS to v0.30 and updated this example too,to see a bigger increment, but the problem continue, for 1000 loop eats around of 10MB of memory in my case.

Well, thk U.

PD:Let me know if you need some test.