Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Hi, Need your expert advice on fine tuning the below perl script, which is currently taking more than 30 minutes to extract information from a raw file which has nearly 2million records. below are the requirements met in the script: 1) script should scan through a log file and output the final result into a file called "test.txt" 2)it replaces some strings in the log file into more generic terms 3)it finds the record number of the affected row from the log file and uses this record number to query the main source file for the particular record 4) it generates a string which now contains information from the logfile and patches it along with the affected row from the source 5) the result is written into the output file test .txt

=======================code ============================ #!/usr/bin/perl $read_file = "$ARGV[0]"; $read_source = "$ARGV[1]"; open(LOGFILE,$read_file) or die "An Error Occured : $!"; open(REPORT,">/retsit/systematics/test.txt"); $str1 = 'failed all WHEN clauses'; $str2= 'CUST SEGEMENT IS EMPTY'; $str3= 'unique constraint'; $str4= 'DUPLICATE RECORD'; while(<LOGFILE>) { if ($_ =~ /Record/) { $_ =~ s/$str1/$str2/g; $_ =~ s/$str3/$str4/g; $ind1 = index($_,'Record')+6; $len2 =index($_, ':')-6; $recnum = substr($_,$ind1,$len2); $recnum =~ s/^\s+|\s+$//g ; $strx = "sed -n '".$recnum."p' ".$read_source; $str5 = `$strx`; $_ .= '|'."$recnum"."$str5\n"; print REPORT $_; } } close(LOGFILE); close(REPORT);

In reply to Need help to fine tune perl script to make it faster( currently taking more than 30 minutes) by anujajoseph

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others making s'mores by the fire in the courtyard of the Monastery: (2)
    As of 2018-03-24 08:26 GMT
    Find Nodes?
      Voting Booth?
      When I think of a mole I think of:

      Results (297 votes). Check out past polls.