Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Large table query code help!

by Anonymous Monk
on Feb 02, 2012 at 18:16 UTC ( #951505=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there Monks!
I am querying a large table 50 records at a time, that works no problem, but at line 34 this code:
$allaccounts = $sql_dbh->exec( "select * from othertable where acc in +($placeholders)", @files );

Is where I am having the issue I can't figure it out.
It seems that the account numbers in this variable "@files" has no comma separated between its values, this might be the cause, I am just confused on how to get it fixed. The reason might be because on line 30, if I use “push” adding those values into “@files, it will find them in the "othertable" table as you can see in this sample code I am posting here. Does the values in @files on line 34 needed to somehow have comma separated in order to the code to work? Any help its appreciated!
Here is the sample code for this:
#!/usr/bin/perl -w use strict; use CGI qw/:standard/; use POSIX qw(strftime); my $db = "test"; my $sql_dbh = lib::Db->connect( 'local' ); # get a list of account numbers my @data = map { $_->{ ACCOUNT } } @{ $sql_dbh->exec( qq|select accoun +t from my_table where type='A'|) }; if (!@data) { die "bye now";exit; } my $numb = @data; print "\n Numbers of Accounts: $numb\n"; # Check 50 records at time my $allaccounts; for( my $i = 0; $i < @data; $i += 50 ) { my @files = @data[$i .. $i+49]; #my $all = join(', ', @files); #print "\n $i - $all\n"; #test data #push(@files, "11223","5776","0001","87100"); my $placeholders = join ',', ('?') x @files; $allaccounts = $sql_dbh->exec( "select * from othertable where acc i +n ($placeholders)", @files ); } my $cc=0; foreach (@$allaccounts) { $cc++; # print report my $found = qq|$_->{ name }$_->{ number }|; print "\n $cc - Reporting: $found\n"; }

Comment on Large table query code help!
Select or Download Code
Re: Large table query code help!
by moritz (Cardinal) on Feb 02, 2012 at 20:03 UTC

    Where does the lib::Db come from? it doesn't seem to be used anywhere.

    Anyway, you speak of "issues", but don't mention what the problem is. Is the database engine throwing an error? If yes, which? Or are you getting a different result than expected? If yes, what do you get, what do you expect?

    If you are unsure what values some variables contain, use Data::Dumper to find out.

    One problem I can see with your code is that if @data doesn't contain a number of items that is divisible by 50, then the last query will have lots of trailing undefs. Not sure if you want that.

      the last query will have lots of trailing undefs. Not sure if you want that.

      If the database library maps those to NULLs, it should not be a problem (they match nothing).

Re: Large table query code help!
by Anonymous Monk on Feb 02, 2012 at 20:05 UTC

    What are you trying to achieve in your code? If you want account numbers that are both in table 1 and table 2, it can be easily done with SQL only. In table 1 but not in table 2? No problem. Database engines have been optimising for these sorts of things for decades, so it will very likely work faster and look cleaner than your current code.

    As I understand it, you query for the second table in batches of fifty, leave the last batch of (up to) 50 accounts in $allaccounts, and then print out that variable.

    Does the values in @files on line 34 needed to somehow have comma separated in order to the code to work?

    I am not familiar with your database library (nor do you anywhere mention which one you are using), but if it supports placeholders such as those you are using, I very much doubt that.

      I see your point and as far as this code sample is, the issue here is that this line:
      for( my $i = 0; $i < @data; $i += 50 ) { ...
      should end after the last foreach loop, no db calls or whatever else should matter since you mentioned that all has been working, or:
      #!/usr/bin/perl -w use strict; use CGI qw/:standard/; use POSIX qw(strftime); my $db = "test"; my $sql_dbh = lib::Db->connect( 'local' ); # get a list of account numbers my @data = map { $_->{ ACCOUNT } } @{ $sql_dbh->exec( qq|select accoun +t from my_table where type='A'|) }; if (!@data) { die "bye now";exit; } my $numb = @data; print "\n Numbers of Accounts: $numb\n"; # Check 50 records at time my $allaccounts; for( my $i = 0; $i < @data; $i += 50 ) { my @files = @data[$i .. $i+49]; #my $all = join(', ', @files); #print "\n $i - $all\n"; #test data #push(@files, "11223","5776","0001","87100"); my $placeholders = join ',', ('?') x @files; $allaccounts = $sql_dbh->exec( "select * from othertable where acc i +n ($placeholders)", @files ); #} # not here my $cc=0; foreach (@$allaccounts) { $cc++; # print report my $found = qq|$_->{ name }$_->{ number }|; print "\n $cc - Reporting: $found\n"; } } # end here

      Enjoy it !!!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://951505]
Approved by Eliya
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (12)
As of 2014-11-25 23:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (160 votes), past polls