Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re^2: Multithreading a large file split to multiple files

by Marshall (Abbot)
on May 15, 2018 at 08:53 UTC ( #1214528=note: print w/replies, xml ) Need Help??


in reply to Re: Multithreading a large file split to multiple files
in thread Multithreading a large file split to multiple files

I agree that multiple cores will not help because there is a blocking point of the sequential read of the input file.

I am not so sure about output buffering. I really don't know in this situation, but depending upon the file system and other factors like the intelligence of the disk controller, increasing the buffer size for write could make a difference?
PerlO::buffersize
Just an idea to try. I would benchmark 64K vs standard size (which I guess is probably 4K) and see if there is any significant difference.

  • Comment on Re^2: Multithreading a large file split to multiple files

Replies are listed 'Best First'.
Re^3: Multithreading a large file split to multiple files
by 10isitch (Initiate) on May 15, 2018 at 14:53 UTC
    Thank you for the suggestion. I'll try that out.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1214528]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (3)
As of 2018-11-14 06:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My code is most likely broken because:
















    Results (163 votes). Check out past polls.

    Notices?