Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

Re: How to speed up/multi thread extract from txt files?

by KurtSchwind (Chaplain)
on Jan 09, 2008 at 03:05 UTC ( [id://661235]=note: print w/replies, xml ) Need Help??


in reply to How to speed up/multi thread extract from txt files?

You don't mention which OS you are on. If you are on a *nix box you can use split.

Split the file into equal parts and just run multiple instances of your application.

At least that's one way. However, have you considered that the extraction process isn't actually what needs optimizing? If you are running through a list of regex on each line, I'd hazard a guess that that is consuming more of your time than reading each line in. It might help if you'd post some of your code so we can take a gander and see where your bottleneck is.

--
I used to drive a Heisenbergmobile, but every time I looked at the speedometer, I got lost.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://661235]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (4)
As of 2024-04-20 02:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found