As I look through the collected responses to this thread so-far, I would suggest that there are two general approaches that are being (equally seriously ...) offered:
-
If the total number of output-files is both “truly unpredictable” and “can be counted-on to be small,” then it is possible to have all of the possible output-files open at the same time. As long as you are sure that the operating system won’t object (fatally... as operating-systems are wot to do when their Godly Prerogatives are crossed by Mere Mortals), then you can simply throw each incoming record into the appropriate (simultaneously...) open bucket.
-
If this is not the case, then you probably are going to need to sort the incoming records first. This, by definition, will cause all records having an identical key-value to be physically adjacent ... so that all of the records that are destined for any particular destination are adjacent ... so that you can meaningfully react to a change in the destination, with no need to remember history. The advantage of this approach is, of course, that there is never more than one destination-bucket (file..) open at any one time. The disadvantage is “the overhead of sorting.” (Which may, actually, be quite acceptable. There is, indeed, a reason why one of Dr. Knuth’s seminal books was titled: Sorting and Searching ...)
