http://www.perlmonks.org?node_id=984537


in reply to help deriving mysql query from perl script

IANAL, but if you are a non-profit, you might be able to hire cough cough solicit people by giving them tax exempt receipts for their "donations" of services to your org.

Replies are listed 'Best First'.
Re^2: help deriving mysql query from perl script
by alison-g (Novice) on Jul 31, 2012 at 10:14 UTC

    Thanks everyone!

    I figured out that I just needed to get the data that would be entered in the two user entry forms and incorporate them into the sql queries (and I do know how to run them in various remote sql thingies and phpmyadmin) But I couldn't really figure out 1. what exactly the queries would be and 2. the bits that perl was doing - all the whiles and pushes and the output etc. So I hard coded $where_clause and $group_clause which are: event <> "Training" and (year(date)-2011)*12+month(date) > 6 and rh_name#if(month(date)<12,year(date),year(date)+1)#season respectively. I also added $where_clause to $linked_tables rather than the queries "anding" them. Don't ask me why but it fixed syntax error barefoot doodah message number 63 trillion. Ran it from ssh and lo and behold after addressing 742 kershillion more error messages the darling actually ran and produced an output, eventually. Yeehah BUT, and that is a big but, unfortunately it produced 116,500 rows of data rather than the 1,000 to 1,500 expected! (and I suspect it hadn't finished - just filled up temp table) My perl learner badge somewhat tarnished I return to the monks in the hope that someone will prefer just love to solve it.

    Haven't I done all the hard work? ;-)

    PS here's the code I ended up running

    and if it helps anyone I could email them what last years output looked like

    #!/usr/bin/perl $where_clause = "event <> 'Training' and (year(date)-2011)*12+month(da +te) > 6"; $group_clause = "rh_name#if(month(date)<12,year(date),year(date)+1)#se +ason"; print "Content-type: text/html\n\n"; die unless length($where_clause) > 0; $where_clause = "rwsurvey.type = 'LIT' and ".$where_clause; @groups = split('#',$group_clause); $group_clause = join(",",@groups); use DBI; $dbh = DBI->connect("DBI:mysql:conserva_reefwatch:localhost", "conserv +a", "w399eWkWp1"); if (!$dbh) { $dbh = DBI->connect("DBI:mysql:reefwatch:localhost", "root", "mysq +l"); die "Cannot connect: $D: $DBI::errstr\n" unless $dbh; print "<p>Connected locally</p>"; } $linked_tables = "rwsurvey, rwlit, rworganism, rwRHSites, rwSeasons, r +wlocation where rwlit.organism_id = rworganism.org_id and rwsurvey.su +rvey_id = rwlit.survey_id and rwRHSites.site_id = rwsurvey.site_id an +d rwRHSites.exposure = rwsurvey.exposure and rwlocation.site_id = rws +urvey.site_id and season_id = if(month(date)>0,floor(month(date) % 12 +/3),-1) and ".$where_clause; $output_counter= 0; ############################################################ # # Get summary of surveys and metres for each group # ############################################################ $stmt = "select rh_name,if(month(date)<12,year(date),year(date)+1),sea +son, count(distinct rwlit.survey_id), count(distinct CONCAT(rwsurvey. +survey_id,'_',metre)) from $linked_tables GROUP BY rh_name,if(month(d +ate)<12,year(date),year(date)+1),season"; #print "<p>$stmt"; #com $sth = $dbh->prepare($stmt) || print "Can't prepare $stmt: ".$dbh->err +str()."\n"; $sth->execute || print "can't execute the query: ".$sth->errstr()."\n" +; while (@res = $sth->fetchrow) { $line = join(',',@res); push (@group_val_list, $line); } #print @group_val_list; #com ############################################################ # # Extract the data for each group # ############################################################ foreach $line (@group_val_list) { $output_counter++; print "A,$output_counter,$line<br>"; @vals = split (/,/, $line); $last_id = $#vals - 2; @vals = @vals[0..$last_id]; $extra_clause = ""; for ($i=0; $i<=$#vals; $i++) { $extra_clause .= " and $groups[$i] = '$vals[$i]'"; } ############################################################ # # Type B: Raw data # ############################################################ $stmt = "SELECT rwsurvey.survey_id, metre, transition, code from $ +linked_tables $extra_clause order by rwsurvey.survey_id, metre, tran +sition"; # print "<p>$stmt</p>"; $sth = $dbh->prepare($stmt) || print "Can't prepare $stmt: ".$dbh- +>errstr()."\n"; $sth->execute || print "can't execute the query: ".$sth->errstr(). +"\n"; $last_survey_id = -1; $total_metres = 0; $total_surveys = 0; undef %cover; while (($survey_id, $metre, $transition, $code) = $sth->fetchrow) { $output_counter++; print "B,$output_counter,$survey_id,$metre,$transition,$code<br>"; if ($survey_id != $last_survey_id) { $total_surveys++; if ($last_survey_id != -1) { $total_metres += ($last_metre - $min_metre + 1); } $last = ($metre-1)*100; $min_metre = $metre; if (($transition != 0) or ($code != 'START')) { $output_counter++; print "D,$output_counter,'Problem: $survey_id, $last_survey_id +, $transition, $code'<br>"; } } else { $current = ($metre-1) * 100 + $transition; $last_metre = $metre; $delta = $current - $last; #print "delta = $delta<br>"; $cover{$code} += $delta; if ($delta < 0) { $output_counter++; print "D,$output_counter,'Problem: negative $current, $last, $ +metre, $transition, $code'<br>"; } $last = $current; } $last_survey_id = $survey_id; } $total_metres += ($last_metre - $min_metre + 1); #print "<p>$total_surveys surveys (total of $total_metres metres)" +; ############################################################ # # Type C: Percentage cover # ############################################################ while (($lifeform, $total_length) = each(%cover)) { $percent_cover = sprintf("%0.1f",$total_length / $total_metres); $val_list = join(',', @vals); $output_counter++; print "C,$output_counter,$val_list,$lifeform,$total_length,$percen +t_cover<br>"; } }

      Forget this Perl script for now and just run your queries through PhpMyAdmin. If the query produced 100k+ result rows than you either got the query wrong or that's just simply the amount of data in your database now. You need to look at the ouput in PhPMyAdmin to see if that makes sense. It's pretty much impossible to tell just from the script without knowing your database schema.
      There are two queries in that script: are both producing "the wrong amount" of data?
      Once you are satisfied that these queries extract the data you need and you go back to working with this script to process the outputs, you do need to work on those errors of course. There can't be that many in a short script like that. Don't be fooled by a screen-full of error messages. Often, this is just caused by one simple syntax error that could easily be identified if you posted your error messages here.
      But as I said: you need to figure out those queries first. Otherwise, the whole Perl script is pointless.