Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: DBI vs Bulk Loading

by LanceDeeply (Chaplain)
on Sep 14, 2005 at 22:03 UTC ( #492050=note: print w/replies, xml ) Need Help??


in reply to DBI vs Bulk Loading

how much of your process is updating records vs inserting new records?

if it's mostly updates:
  • select out all the data into your perl program
  • sum logfiles against data structure in memory by timestamp key
  • write summed values to csv file
  • delete table
  • bulk load csv file

    if it's mostly inserting new records:
  • select out all the data into your perl program
  • sum logfiles against in memory data structure by timestamp key
  • run updates for timestamps that exist
  • write summed values to csv file for new timestamps
  • bulk load csv file

    i dont kow what kind of volume you are processing, but if the full table is too much for your process to handle, you can slice it up by day/hour.
    -HTH
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Node Status?
    node history
    Node Type: note [id://492050]
    help
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others about the Monastery: (2)
    As of 2021-05-12 05:27 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?
      Perl 7 will be out ...





      Results (124 votes). Check out past polls.

      Notices?