Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re: create separate output files based on name

by sundialsvc4 (Abbot)
on Dec 19, 2013 at 00:56 UTC ( #1067755=note: print w/replies, xml ) Need Help??

in reply to create separate output files based on name

As I look through the collected responses to this thread so-far, I would suggest that there are two general approaches that are being (equally seriously ...) offered:

  1. If the total number of output-files is both “truly unpredictable” and “can be counted-on to be small,” then it is possible to have all of the possible output-files open at the same time.   As long as you are sure that the operating system won’t object (fatally... as operating-systems are wot to do when their Godly Prerogatives are crossed by Mere Mortals), then you can simply throw each incoming record into the appropriate (simultaneously...) open bucket.
  2. If this is not the case, then you probably are going to need to sort the incoming records first.   This, by definition, will cause all records having an identical key-value to be physically adjacent ... so that all of the records that are destined for any particular destination are adjacent ... so that you can meaningfully react to a change in the destination, with no need to remember history.   The advantage of this approach is, of course, that there is never more than one destination-bucket (file..) open at any one time.   The disadvantage is “the overhead of sorting.”   (Which may, actually, be quite acceptable.   There is, indeed, a reason why one of Dr. Knuth’s seminal books was titled:   Sorting and Searching ...)

  • Comment on Re: create separate output files based on name

Replies are listed 'Best First'.
Re^2: create separate output files based on name
by Laurent_R (Canon) on Dec 19, 2013 at 07:12 UTC
    Yes, exactly, it is for the purpose of selecting one of these two approaches that i asked the question above.

      Thanks so much for all the input.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1067755]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (5)
As of 2018-06-20 02:42 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (116 votes). Check out past polls.