Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Multi threading

by devil_encoder (Initiate)
on May 23, 2013 at 13:03 UTC ( #1034943=perlquestion: print w/replies, xml ) Need Help??
devil_encoder has asked for the wisdom of the Perl Monks concerning the following question:

hello Perler :)

Hi All,I am new to perl and I have the below task, load a file with 10000000 lign with perl, and for some problems about the slownless of the application, i'll introduce the multi threading concept to split the file on 10 part


Replies are listed 'Best First'.
Re: Multi threading
by MidLifeXis (Monsignor) on May 23, 2013 at 13:24 UTC

    From your minimal description, I am assuming this scenario: you have a 100+ MB file that you need to load into some other application. It is slow. You are trying to throw 'threading' at it to speed it up without knowing where the slowness is coming from.

    Where is the bottleneck? You state that there are 10 million lines that you are reading. What is the total size of the file? At 100 chars per line, you would have a 1GB file. This can be slow to read, large to represent fully in memory, etc.

    There are many things that can cause 'slowness'. Until you can characterize where the slowness comes from, hold off on picking a solution. Characterize the method used by your application to accomplish the task. Is it reading all of the data into memory and then doing something with it, or is it interleaving reads with processing? If interleaving, is it discarding information it no longer needs? Is it reading from and writing to the same physical disk, same I/O channel, saturating the network, etc?

    If you are saturating a single resource, threading will probably not solve your problem, and may, in fact, add to it, since you now need to manage the threads as well. Disk, I/O, network, CPU, and Memory are some of the more common resources that can be saturated, but there are others as well. Figure out how to make those resources less saturated (or do other work instead of waiting to use the saturated resource - which may involve threading).

    Update: Added last phrase indicating where threading may be useful


Re: Multi threading
by Laurent_R (Abbot) on May 23, 2013 at 17:47 UTC

    Hi, please explain what you mean when you say you want to load 10 million lines. Is this in a database? If yes, which type of DB. Or something else?

Re: Multi threading
by Anonymous Monk on May 24, 2013 at 04:30 UTC
    Defragment your HD to speed up your application.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1034943]
Approved by davido
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (9)
As of 2017-04-24 17:57 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (443 votes). Check out past polls.