Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^4: Looking for ways to speed up the parsing of a file...

by sgifford (Prior)
on May 18, 2008 at 18:59 UTC ( #687239=note: print w/ replies, xml ) Need Help??


in reply to Re^3: Looking for ways to speed up the parsing of a file...
in thread Looking for ways to speed up the parsing of a file...

Instead of actually splitting the file into several additional files, you could just determine the positions as you describe, then work on the different parts by seeking to the right position before starting your processing loop. For example, you could determine the start and end position and then fork() off a new process to work on that chunk. Reading from multiple files (or different places in the same file) in parallel might end up being less efficient from an I/O perspective, though, as it could require the drive to seek a lot more. So you'd need to experiment a bit to find the right way to parallelize this.


Comment on Re^4: Looking for ways to speed up the parsing of a file...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://687239]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2014-07-13 06:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (247 votes), past polls