Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re^3: problem of my multithreading perl script

by roboticus (Chancellor)
on Jan 11, 2013 at 13:48 UTC ( #1012892=note: print w/replies, xml ) Need Help??

in reply to Re^2: problem of my multithreading perl script
in thread problem of my multithreading perl script


No, there are multiple ways that a threaded program can be slower than a single threaded program. In this case, I don't know exactly which one(s) you're hitting, but there are two candidates I can think of:

  • Disk thrashing: When you're reading a file sequentially, the disk drive can normally read sector after sector without (many) intermediate seeks. If you're reading multiple files at the same time, your drive needs to seek frequently between file locations. In other words, reading files one at a time generally has fewer disk seeks than reading them in parallel, and seeks are slow operations.
  • Data structure locking: In a single-threaded program, you don't need to worry about locking your data structures. But in a multithreaded program, you need to lock data structures when modifying them if another thread has the potential to also update the data structure. Even when there's no contention when accessing the data structure you're spending time creating and freeing locks.

Having said that, you can probably speed things up without going to a single-threaded program.

If your program is experiencing disk thrashing, you might be able to avoid it by placing your files on different drives. That way you won't have as many disk seeks.

If your program is spending too much time managing data structure locks, you might be able to rearrange your code a bit to reduce the number of locks. For example, you might have each thread read and process a file in a local (nonshared) data structure, and then once it's finished a file, then merge the data into the shared structure.

There are other possible problems and solutions to your problem, these are just the ones that came to mind. (I've run into both of them frequently enough...unfortunately.)


When your only tool is a hammer, all problems look like your thumb.

  • Comment on Re^3: problem of my multithreading perl script

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1012892]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (3)
As of 2018-05-27 18:49 GMT
Find Nodes?
    Voting Booth?