Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re^3: Posgres batch read with DBI? - ( PostgreSQL )

by Marshall (Canon)
on Jan 30, 2021 at 05:37 UTC ( #11127678=note: print w/replies, xml ) Need Help??

in reply to Re^2: Posgres batch read with DBI? - ( PostgreSQL )
in thread Posgres batch read with DBI?

I agree with you.

The OP wrote: "Greetings. I have about 250K records in a postgres db I need to process. I'm wondering if there's a way to read this in batches of 1000 records or something, to cut down on db transaction overhead.
The way to "cut down on transaction overhead" is simply to run fewer write transactions - that doesn't have much to do with how often you read the data.

The DB can perform many, many read operations per second.
It will be able to perform much fewer write transactions per second than read operations.

The DB whichever one you have, will try to maintain a self consistent state on the hard drive. See Wiki - ACID properties.

There is a lot of overhead involved in a "transaction". The DB will have to write to the hard disk multiple times. A commit of a million row update doesn't necessarily take all that much longer than a commit of 100 rows. The commit operation, in and of itself is "very expensive".

The Perl DBI provides:

$rc = $dbh->begin_work or die $dbh->errstr; and $rc = $dbh->commit or die $dbh->errstr;
Wrapping those 2 statements around any DB operations involving many "writes" will have a huge effect upon performance.
There is no problem whatsoever with a 1 million line "commit".
250 commits for 250K records will slow things down considerably - perhaps even an order of magnitude!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11127678]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2021-04-14 22:43 GMT
Find Nodes?
    Voting Booth?

    No recent polls found