woah! way to get distracted! I just realised I've spent way too much time messin with this.. back to work..

not really a "run once" as it would need tuning to the machine/data etc and the sorts could certainly do with optimising..

max records and max hash keys are configurable dependant on available memory. When there are too many records, the largest file is dumped, when there are too many files (hash keys) we dump the smallest.

#!/usr/bin/perl use strict; use warnings; my $max_records = 10; my $max_files = 4; my %rec; my $record_count = 0; my $total_record_count = 0; my $total_dump_count = 0; while ( <DATA> ) { $record_count++; print $record_count."] $_"; my @field = split /,/; my $file = splice @field, 2, 1; push(@{$rec{$file}},$_); if ($record_count >= $max_records){ print " Too many records..\n"; my ($big_file) = sort { @{$rec{$b}} <=> @{$rec{$a}} } keys %rec; dumping($big_file); } if (scalar keys %rec > $max_files){ print " Too many files..\n"; my ($lil_file) = sort { @{$rec{$a}} <=> @{$rec{$b}} } keys %rec; dumping($lil_file); } $total_record_count++; } print " Data Stopped..\n"; foreach(keys %rec){ dumping($_); } print "Total Records: $total_record_count\n"; print "Total Dumps: $total_dump_count\n"; sub dumping{ my $file = shift; $total_dump_count++; print " Dumping $file (".@{$rec{$file}}." records)\n"; open( FILE, '>>', $file ); print FILE @{$rec{$file}}; close(FILE); $record_count -= @{$rec{$file}}; delete $rec{$file}; } __DATA__ 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 2,3,barbar,4 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 5,6,bazbaz,7 3,4,foo,5 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 ...output... 1] 1,2,foo,3 2] 4,5,bar,6 3] 7,8,foo,9 4] 0,1,baz,2 5] 3,4,foo,5 6] 6,7,bar,8 7] 9,0,baz,1 8] 2,3,barbar,4 9] 1,2,foo,3 10] 4,5,bar,6 Too many records.. Dumping foo (4 records) 7] 7,8,foo,9 8] 0,1,baz,2 9] 5,6,bazbaz,7 Too many files.. Dumping bazbaz (1 records) 9] 3,4,foo,5 10] 1,2,foo,3 Too many records.. Dumping bar (3 records) 8] 4,5,bar,6 9] 7,8,foo,9 10] 0,1,baz,2 Too many records.. Dumping baz (4 records) 7] 3,4,foo,5 8] 6,7,bar,8 9] 9,0,baz,1 Data Stopped.. Dumping bar (2 records) Dumping baz (1 records) Dumping foo (5 records) Dumping barbar (1 records) Total Records: 21 Total Dumps: 8

cheers,

J


In reply to Re: Performance Trap - Opening/Closing Files Inside a Loop by edoc
in thread Performance Trap - Opening/Closing Files Inside a Loop by Limbic~Region

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":