Re: Multi threading

[MidLifeXis]

From your minimal description, I am assuming this scenario: you have a 100+ MB file that you need to load into some other application. It is slow. You are trying to throw 'threading' at it to speed it up without knowing where the slowness is coming from.

Where is the bottleneck? You state that there are 10 million lines that you are reading. What is the total size of the file? At 100 chars per line, you would have a 1GB file. This can be slow to read, large to represent fully in memory, etc.

There are many things that can cause 'slowness'. Until you can characterize where the slowness comes from, hold off on picking a solution. Characterize the method used by your application to accomplish the task. Is it reading all of the data into memory and then doing something with it, or is it interleaving reads with processing? If interleaving, is it discarding information it no longer needs? Is it reading from and writing to the same physical disk, same I/O channel, saturating the network, etc?

If you are saturating a single resource, threading will probably not solve your problem, and may, in fact, add to it, since you now need to manage the threads as well. Disk, I/O, network, CPU, and Memory are some of the more common resources that can be saturated, but there are others as well. Figure out how to make those resources less saturated (or do other work instead of waiting to use the saturated resource - which may involve threading).

Update: Added last phrase indicating where threading may be useful