Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Re: Performance Trap - Opening/Closing Files Inside a Loop

by edoc (Chaplain)
on Dec 10, 2004 at 03:21 UTC ( #413743=note: print w/replies, xml ) Need Help??

in reply to Performance Trap - Opening/Closing Files Inside a Loop

woah! way to get distracted! I just realised I've spent way too much time messin with this.. back to work..

not really a "run once" as it would need tuning to the machine/data etc and the sorts could certainly do with optimising..

max records and max hash keys are configurable dependant on available memory. When there are too many records, the largest file is dumped, when there are too many files (hash keys) we dump the smallest.

#!/usr/bin/perl use strict; use warnings; my $max_records = 10; my $max_files = 4; my %rec; my $record_count = 0; my $total_record_count = 0; my $total_dump_count = 0; while ( <DATA> ) { $record_count++; print $record_count."] $_"; my @field = split /,/; my $file = splice @field, 2, 1; push(@{$rec{$file}},$_); if ($record_count >= $max_records){ print " Too many records..\n"; my ($big_file) = sort { @{$rec{$b}} <=> @{$rec{$a}} } keys %rec; dumping($big_file); } if (scalar keys %rec > $max_files){ print " Too many files..\n"; my ($lil_file) = sort { @{$rec{$a}} <=> @{$rec{$b}} } keys %rec; dumping($lil_file); } $total_record_count++; } print " Data Stopped..\n"; foreach(keys %rec){ dumping($_); } print "Total Records: $total_record_count\n"; print "Total Dumps: $total_dump_count\n"; sub dumping{ my $file = shift; $total_dump_count++; print " Dumping $file (".@{$rec{$file}}." records)\n"; open( FILE, '>>', $file ); print FILE @{$rec{$file}}; close(FILE); $record_count -= @{$rec{$file}}; delete $rec{$file}; } __DATA__ 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 2,3,barbar,4 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 5,6,bazbaz,7 3,4,foo,5 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 ...output... 1] 1,2,foo,3 2] 4,5,bar,6 3] 7,8,foo,9 4] 0,1,baz,2 5] 3,4,foo,5 6] 6,7,bar,8 7] 9,0,baz,1 8] 2,3,barbar,4 9] 1,2,foo,3 10] 4,5,bar,6 Too many records.. Dumping foo (4 records) 7] 7,8,foo,9 8] 0,1,baz,2 9] 5,6,bazbaz,7 Too many files.. Dumping bazbaz (1 records) 9] 3,4,foo,5 10] 1,2,foo,3 Too many records.. Dumping bar (3 records) 8] 4,5,bar,6 9] 7,8,foo,9 10] 0,1,baz,2 Too many records.. Dumping baz (4 records) 7] 3,4,foo,5 8] 6,7,bar,8 9] 9,0,baz,1 Data Stopped.. Dumping bar (2 records) Dumping baz (1 records) Dumping foo (5 records) Dumping barbar (1 records) Total Records: 21 Total Dumps: 8



Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://413743]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (4)
As of 2017-06-25 00:38 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (563 votes). Check out past polls.