Re^2: How to set variable names using Text::CSV?

[Marshall]

I didn't follow all the details in this thread re Text::CVS, but your comment about DBD::CSV stands out as a good idea!

I have a project with a CSV file that I am starting soon and plan to go with the DBI approach. I had played with this on Unix a while back and it seemed to work great The problem was that the CSV module for the DBI wasn't available on ActiveState and I couldn't use it for Windows based applications - so this idea went on back burner. That problem has evidently been solved with Perl 5.10 and current modules!

If I'm right about this, using the DBI will allow me to write code that could be used with some other DB at some later time if necessary.

Oh well, not directly applicable to the OP's problem at hand, but I thought that this DBI with CSV is now possible on Windows was worth a mention. A thought for other projects.

The code looks like connecting to other underlying Databases:

use DBI; #need to have DBD::CSV installed for this script

my $db = DBI->connect("DBI:CSV:f_dir=./DEMO.db")
         or die "Cannot connect: $DBI::errstr";
....run DBI SQL stuff ...

Update:There is a thing that I've found out with using CSV files...I have some users who haven't written any code in their life, never seen an SQL or a Perl statement, but yet can do magic with modern spreadsheets! A lot of the old limits are gone like number of lines, and there are multiple pages, etc. Sometimes sending CSV spreadsheet data works to everyone's advantage. The users can sometimes do amazing stuff!

[Tux]

my $dbh = DBI->connect ("dbi:CSV:", undef, undef, {
    f_dir      => "DEMO.db:",
    f_ext      => ".csv/r",
    f_schema   => undef,
    PrintError => 1,
    RaiseError => 1,
    });

The disadvantage of this when the data set grows bigger is that all of your CSV data is kept in memory, and probably even twice. That, plus some meta-data will quickly gobble up most of your available valuable memory. If you want an intermediate database to make a move to something real later on, and still want portability, skip DBD::CSV and use DBD::SQLite instead. It is lightweight, fast and portable and supports a lot more than CSV.

[Marshall]

Thanks! for your suggestion of SQLite, I will see if I can get it on Windows! After I wrote my post, I generated the starting database, about 14K lines. This will grow in a very short time to about 50K lines. I know from experience and a lot of testing on other Perl applications that I won't start having trouble until memory footprint reaches > 1/2 GB. But that can happen quickly!

The basic thing is to get started using SQL for this stuff! I want to port the existing process first (which can be done with some "lite" tools), then on the "features to do list" are things that will require MUCH larger DB's. So, yes, portability of code is a factor if it is possible to do.