Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
For starters drop the while( $record = <INFILE>) and try the more idiomatic while( <INFILE> ) then use the implicit $_ instead of $record.

Use references to pass into and recieve back from your subroutines. Don't bother initializing them in the calling block:

@newrecord = (); # Send the record to have any blank lines removed. @newrecord = CleanData( $record ); -vs- $newrecord = CleanData( $record ); sub CleanData { my( $record ) = $_[0]; my ($cleanrecord, $line ); ### Separate the record by newlines. @$record = split /\n/, $record; ### Remove empty lines if they exist. foreach ( @$record ) { chomp; next if /^\s+$/; push @$cleanrecord, $_; } $cleanrecord; }

Also, take a look at your regexes. Try to reduce those to a more reasonable number (and slap the o option on the end to optimize them).

if ($line =~ m/CHROMOSOME:\s*(\d*)\s*$/) { $chrome_num = $1; } # Get TIGR model's reference id if ($line =~ m/MODEL_NAME:\s*(.*)$/) { $tigr_id = $1; } # Get the locus on the bac where the gene was cloned. if ($line =~ m/BAC_LOCUS:\s*(.*)\s*$/) { $bac_locus = $1; } .... might go better as if( $line =~ m/(.*?):\s+(\S+)\s+/o ) { $type = $1; $info = $2; } if( $type eq "CHROMOSONE" ) { # do whatever } elseif( $type eq "NUM_FOO ) { # do num foo }

You get the idea. Running multiple regexes against a single line can take forever. Try to reduce the regexes into simpler one(s) that will run once against the line and also ensure you use the 'o' flag so you're not spinning your wheels always compiling the regexes.

But I think all of those type changes are going to have a minimal impact. You really need to concentrate on your dbs performance and whats going on there. I'm not sure about mysql but some database will commit "transactions" so many inserts (does mysql/innodb table do that?). You can set up your drive to make that number larger and reduce the number of commits.

They're may be other stuff but that's just one person's opinion. It's kinda hard to do without being able to run the stuff. Have you tried profiling the script to see where you're spending all your time? At a minimum, you could liberally sprinke Benchmark timings before and after sections of the code and then use timediff to see how long the snippet took.

Good luck. ++ for an interesting post.


update: ferrency is right about the optimizations. sorry for the mislead.

In reply to Re: Up for Critique by derby
in thread Up for Critique by biograd

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others surveying the Monastery: (3)
    As of 2018-04-21 04:59 GMT
    Find Nodes?
      Voting Booth?