Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re^2: How to improve MYSQL search performance of perl?

by nan (Novice)
on Aug 19, 2005 at 15:32 UTC ( #485182=note: print w/ replies, xml ) Need Help??


in reply to Re: How to improve MYSQL search performance of perl?
in thread How to improve MYSQL search performance of perl?

Hi,

As I need to read line by line and search them in the database, I used a subroutine to handle all database work. Below is my code.

sub search{
#take search parameter from html <form/>
my $q = $_[0]; #taking param;
my $found = 0; #initialize category found count;
#open database
my $ref = {RaiseError => 1, AutoCommit => 0};
my $dbh = DBI->connect('DBI:mysql:diet', $ref) || die "Failed to connect: $DBI::errstr";
#prepare SQL search statement
my $sql = qq{select topic FROM table1 WHERE uri LIKE '$q'};
my $sth = $dbh->prepare($sql);
$sth->execute();
while(my $record = $sth->fetchrow_hashref) {
no warnings;
print &topic($record->{topic}), "
\n";
$found++;
}
if ($found == 0){ #if no category was found, output no found
print "

$q is not found in the database!

\n";
}
$sth->finish();
$dbh->disconnect(); #disconnect from database;
}

thanks again,

Nan


Comment on Re^2: How to improve MYSQL search performance of perl?
Re^3: How to improve MYSQL search performance of perl?
by trammell (Priest) on Aug 19, 2005 at 15:55 UTC
    One improvement you can make is to only open your database handle once at the beginning of the script, and reuse that handle instead of recreating it for each query.
Re^3: How to improve MYSQL search performance of perl?
by davidrw (Prior) on Aug 19, 2005 at 17:00 UTC
    I would take trammell's suggestion a step further and also not recreate the statement handle every time and actually take advantage of the statement handle (and placeholders)-- i think this will have a decent improvement in the performance (amount of gain is probably db-dependent):
    my $dbh = DBI->connect('DBI:mysql:diet', {RaiseError => 1, AutoCommit +=> 0} ) || die "Failed to connect: $DBI::errstr"; my $sth = $dbh->prepare( qq{select topic FROM table1 WHERE uri LIKE ?} + ); search($sth, 'foo'); search($sth, 'bar'); $sth->finish(); $dbh->disconnect(); #disconnect from database; sub search{ my $sth = shift; # require statement handle (this could probably be + a global var instead if desired) my $q = shift; # take search parameter from html <form/> my $found = 0; #initialize category found count; $sth->execute($q); my $rows = $sth->fetchall_arrayref( {} ); printf "%d rows found for '%s'.\n", scalar(@$rows), $q; foreach my $row (@$rows){ printf " Topic: %s\n", &topic($row->{topic}); } }

      Hi,

      Many thanks for the code as I learned a lot from you. But even opened database once, the search is still slow (running 200 queries one by one) and I always recieve Internal Server 500 Error after a long waiting and can't see any results back to screen (before modifying, I could see a few results back as the database open and close everytime). I'm not sure it is due to the configuration of apache server or MySQL database so I'm rebuilding the database now to see if it helps.

      thanks again,

      Nan

        Can you post your revised code? Are you re-using the statement handle? What's the error in the error_log (or if you run on command line)?

        Also, try adding $|=1; in your script to make sure it's not buffereing so yuo'll see output as it's happening.

        If the loop to execute the queries is good now (resuing handles, etc), then this probably just goes back to the issue mentioned elsewhere in this thread about SQL performance .. the speed will depend entirely on your SQL, table schema, and indexes (can you provide any of those?).

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://485182]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (5)
As of 2014-09-19 02:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (129 votes), past polls