Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

I have a script that splits a large 15Gb file into several smaller files. It only runs on single cpu core and takes over half an hour. Can I make it run on multiple cores so it runs faster? Here's a abbreviated version. Normally it would create 13 smaller files.

use strict; print "\nSTARTING $0...\n"; if (@ARGV != 3) { print "Usage: <BNH scenario file> <Equity scenario output> <IR + scenario output>\n"; exit -1; } my $infile = $ARGV[0]; my $eqfile = $ARGV[1]; my $irfile = $ARGV[2]; my (@arr,@arr2,$w1,$w2); unless (open(INFILE, "< $infile")) {print "Error: Can not open $infile +\n"; exit -1}; unless (open(EQFILE, "> $eqfile")) {print "Error: Can not open $eqfile +!\n"; exit -1}; unless (open(IRFILE, "> $irfile")) {print "Error: Can not open $irfile +!\n"; exit -1}; while (<INFILE>) { chomp $_; if ($_ =~ /ScenSet/) { print EQFILE "$_\n"; print IRFILE "$_\n";next; +} @arr = split/\,/; if ($arr[1]) { print EQFILE ",equity,Base Scenario,0,\n"; print IRFILE ",irate,Base Scenario,0,\n"; next; } if ($arr[2]) { my @arrn = split(/\_/,$arr[2]); print EQFILE ",,eq_".$arrn[1].",1,\n"; print IRFILE ",,ir_".$arrn[1].",1,\n"; next; } if ($arr[6]) {$w1 = 0;$w2 = 0;} # riskfactor contains the string "Index" will be identified as equity +riskfactor if ($_ =~ /Index/ && $_ !~ /Credit/) { $w1 = 1;} # Assumption for IR - there will be only riskfactor USDSWAP and USDTRE +A if ($_ =~ /USDSWAP/ || $_ =~ /USDTREA/) { $w2 = 1;} if ($w1) { print EQFILE "$_\n"; } if ($w2) { print IRFILE "$_\n"; } } close INFILE; close EQFILE; close IRFILE;

In reply to Multithreading a large file split to multiple files by 10isitch

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others examining the Monastery: (4)
    As of 2019-08-18 10:47 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?
      If you were the first to set foot on the Moon, what would be your epigram?






      Results (134 votes). Check out past polls.

      Notices?