Re^4: Ordering Template cards

[bliako]

I always use refs as well. My first question at an API is if it can provide me with refs. Provided that the returned data will be processed locally and immediately

[Marshall]

I could be wrong about this, but in my testing, asking for an array_ref like above, causes the entire result set to be copied into Perl memory. If you undefine the array_ref or assign it another value, Perl can reuse that memory. I haven't benchmarked this, but I'm sure that other Monks have: I suspect that getting an array_ref is faster than going row by row with the exception of an enormous result set.

[bliako]

Marshall, that's a good point regarding db-operations. You are right that there is memory cost for the performance boost of using refs with this specific use-case: db-IO. I was talking more generally so probably I gave the wrong impression for this specific case. Anyway refreshing on the manual is good.

According to fetchall_arrayref of DBI a tradeoff would be to fetchall_arrayref() in batches but still use array-refs, as

That might be the fastest way to fetch and process lots of rows using the DBI, but it depends on the relative cost of method calls vs memory allocation.

my $rows = []; # cache for batches of rows
while( my $row = ( shift(@$rows) || # get row from cache, or reload ca
+che:
                   shift(@{$rows=$sth->fetchall_arrayref(undef,10_000)
+||[]}) )
) {
  ...
}

bw, bliako

[Marshall]

bliako, Most excellent reference!

I am an "old school hardware guy", dating back to the dark ages where 64K bytes of iron core memory was a lot! My current machine is 64 bits with 16 GB of semiconductor memory. Memory usage is something to be aware of, but I don't worry about it often, but "seldom does not mean never".

I suppose that the unwary could get into a "memory thrashing" situation with a huge allocation of virtual memory. With my experience so far, if the result set is so humongous that it won't fit into physical memory, then there may be a problem with the SQL query itself! That is something to be considered also!