Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Multithreading a large file split to multiple files

by cavac (Curate)
on May 16, 2018 at 12:43 UTC ( #1214651=note: print w/replies, xml ) Need Help??


in reply to Multithreading a large file split to multiple files

While not an answer to multithreading, you might look into some other performance improvements as well. I don't know the format of your source file, but there might be some things to look for:

  • Do you need to split/,/ all the way or do you only need the first few columns? Limiting the splitting might speed things up.
  • Are some of the things you check for by regex actually on a fixed position in the string? substr() might be faster.
  • Are you really limited by the CPU? You might be slowed down by IO. Putting source and destination on different drives might help.
  • The statement print EQFILE ",,eq_".$arrn1.",1,\n" runs a useless string concatenation. Use commas instead of dots between string parts. Also, the first fixed string can use single quotes, this removes the need to run the "check if string contains variables that we need to replace" part of string generation.
"For me, programming in Perl is like my cooking. The result may not always taste nice, but it's quick, painless and it get's food on the table."
  • Comment on Re: Multithreading a large file split to multiple files

Replies are listed 'Best First'.
Re^2: Multithreading a large file split to multiple files
by Anonymous Monk on May 16, 2018 at 18:11 UTC
    One more possible optimization: when you want to check for the presence of constant strings (e.g. Index, USDSWAP) in the input, you can use if (index($_, 'USDSWAP') != -1) { ... } instead of a regex.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1214651]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2018-12-14 22:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How many stories does it take before you've heard them all?







    Results (69 votes). Check out past polls.

    Notices?