Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
For starters drop the while( $record = <INFILE>) and try the more idiomatic while( <INFILE> ) then use the implicit $_ instead of $record.

Use references to pass into and recieve back from your subroutines. Don't bother initializing them in the calling block:

@newrecord = (); # Send the record to have any blank lines removed. @newrecord = CleanData( $record ); -vs- $newrecord = CleanData( $record ); sub CleanData { my( $record ) = $_[0]; my ($cleanrecord, $line ); ### Separate the record by newlines. @$record = split /\n/, $record; ### Remove empty lines if they exist. foreach ( @$record ) { chomp; next if /^\s+$/; push @$cleanrecord, $_; } $cleanrecord; }

Also, take a look at your regexes. Try to reduce those to a more reasonable number (and slap the o option on the end to optimize them).

if ($line =~ m/CHROMOSOME:\s*(\d*)\s*$/) { $chrome_num = $1; } # Get TIGR model's reference id if ($line =~ m/MODEL_NAME:\s*(.*)$/) { $tigr_id = $1; } # Get the locus on the bac where the gene was cloned. if ($line =~ m/BAC_LOCUS:\s*(.*)\s*$/) { $bac_locus = $1; } .... might go better as if( $line =~ m/(.*?):\s+(\S+)\s+/o ) { $type = $1; $info = $2; } if( $type eq "CHROMOSONE" ) { # do whatever } elseif( $type eq "NUM_FOO ) { # do num foo }

You get the idea. Running multiple regexes against a single line can take forever. Try to reduce the regexes into simpler one(s) that will run once against the line and also ensure you use the 'o' flag so you're not spinning your wheels always compiling the regexes.

But I think all of those type changes are going to have a minimal impact. You really need to concentrate on your dbs performance and whats going on there. I'm not sure about mysql but some database will commit "transactions" so many inserts (does mysql/innodb table do that?). You can set up your drive to make that number larger and reduce the number of commits.

They're may be other stuff but that's just one person's opinion. It's kinda hard to do without being able to run the stuff. Have you tried profiling the script to see where you're spending all your time? At a minimum, you could liberally sprinke Benchmark timings before and after sections of the code and then use timediff to see how long the snippet took.

Good luck. ++ for an interesting post.


update: ferrency is right about the optimizations. sorry for the mislead.

In reply to Re: Up for Critique by derby
in thread Up for Critique by biograd

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    [LanX]: damn my headphones were not plugged and the video blasted thru the office ...
    [LanX]: xD
    [LanX]: .. my boos woke up... ;-)
    [Eily]: nice to see I'm not the only one looking like a fool at work
    Eily answered a message but made the smiley on his face instead of in the message
    [Eily]: when did you become a bishop though? 428 xp that's not too long ago is it?
    [Eily]: and congratulations :P
    [choroba]: Eily++ # made the smiley on his face instead of in the message

    How do I use this? | Other CB clients
    Other Users?
    Others musing on the Monastery: (10)
    As of 2017-09-20 15:47 GMT
    Find Nodes?
      Voting Booth?
      During the recent solar eclipse, I:

      Results (237 votes). Check out past polls.