Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: reading file

by dsheroh (Monsignor)
on Aug 04, 2009 at 09:07 UTC ( [id://785697]=note: print w/replies, xml ) Need Help??


in reply to reading file

If you profile your program you will probably find (after removing the infinite loop that moritz pointed out) that it's spending most of its time waiting for data to be read from or written to the hard drive. File I/O takes a certain amount of time (and is relatively slow) and there's not really anything you can do about it in Perl. In some cases, operating system-level tuning may be able to help with it some, but even that rarely does much, since this is primarily a hardware limitation.

To illustrate this slowness:

$ time cat [a random 700M file] > /dev/null real 0m46.649s user 0m0.020s sys 0m2.278s
For a 1G file, we could reasonably expect the time required to be on the order of 67 seconds. And that's just to read the contents of the file and throw them away without doing any processing.

For comparison, try copying your existing file to a new file using your operating system's normal file-copy methods. Since you're doing a copy, plus also doing some other work in the process, then you can reasonably assume that there's no way you can possibly do it faster than a plain copy.

If you're not just encountering an I/O issue, then anything which can be done in your code to speed things up would be inside of the while loop, which you haven't shown us.

Replies are listed 'Best First'.
A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://785697]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (4)
As of 2024-04-24 12:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found