Re: DBI vs Bulk Loading

[LanceDeeply]

how much of your process is updating records vs inserting new records?

if it's mostly updates:

select out all the data into your perl program

sum logfiles against data structure in memory by timestamp key

write summed values to csv file

delete table

bulk load csv file

if it's mostly inserting new records:

select out all the data into your perl program

sum logfiles against in memory data structure by timestamp key

run updates for timestamps that exist

write summed values to csv file for new timestamps

bulk load csv file

i dont kow what kind of volume you are processing, but if the full table is too much for your process to handle, you can slice it up by day/hour.
-HTH