Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: advice needed for processing largish data in multiple files

by TedPride (Priest)
on Aug 19, 2006 at 07:20 UTC ( #568318=note: print w/ replies, xml ) Need Help??


in reply to advice needed for processing largish data in multiple files

I have no idea what the OP is trying to do here. Perhaps a simplified example of input / output data might be helpful? In general, though, if you don't have enough memory space, the best thing to do is process a section of your data at a time, which in this case means only doing a smaller range of the phone numbers on each pass. Just do x number of passes through all the files, ignoring the data that corresponds to phone numbers outside of the current range, dump the results into a sequence of files, and merge those files into a single file. Perhaps not the most efficient way to solve the problem, but rather simple to implement.


Comment on Re: advice needed for processing largish data in multiple files

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://568318]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (12)
As of 2015-07-06 21:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (83 votes), past polls