note
roboticus
<p>[davido], [brad_nov]:</p>
<p>I saw [davido|davido's] solution, and played around with it to add a limit to the number of open files in <c>%handles</c> using a least-recently used (LRU) cache. No real reason, but I thought I'd amuse myself while my son got ready for bed.</p>
<p>You could trim it down a bit, as much of the code just implements traces to show what's happening as it runs.</p>
<c>
$ cat t_file_queue.pl
#!/usr/bin/perl
# Updated PM 1013651 to have a limit on file handles
use strict;
use warnings;
use autodie;
use 5.10.0;
my %handles;
my $MAX_OPEN_FH=3;
while( <DATA> ) {
my( $key ) = m/^[^|]\|([^|]+)/;
if( ! defined $key ) {
warn "Line $. appears malformed. Skipping: $_";
next;
}
print {FH("$key.txt")} $_;
}
close $$_{FH} for values %handles;
sub FH {
# Return file handle for named file
state $cnt=0;
my $key= shift;
# Return current handle if it exists
if (exists $handles{$key}) {
$handles{$key}{cnt}=++$cnt;
print "$key: (cnt=$cnt) found\n";
return $handles{$key}{FH};
}
# Doesn't exist, retire the "oldest" one if we're at the limit
if (keys %handles >= $MAX_OPEN_FH) {
my @tmp = sort { $$a{cnt} <=> $$b{cnt} } values %handles;
say "$key: Too many open files, close one: ",
join(", ",map { "$$_{FName}:$$_{cnt}" } @tmp);
my $hr = $tmp[0];
print " closing $$hr{FName}\n";
close $$hr{FH};
delete $handles{$$hr{FName}};
}
open my $FH, '>>', $key;
$handles{$key} = { cnt=>++$cnt, FName=>$key, FH=>$FH };
print "$key: opened new file ($cnt)\n";
return $FH;
}
__DATA__
a|1|foo
b|1|bar
c|2|baz
d|1|xyzzy
e|2|blarg
f|2|The
g|3|quick
h|2|red
i|2|fox
j|3|jumped
k|4|over
l|1|the
m|1|lazy
n|1|brown
o|1|dog
p|5|gorgonzola
</c>
<p>Running it gives me:</p>
<c>
$ ./t_file_queue.pl
1.txt: opened new file (1)
1.txt: (cnt=2) found
2.txt: opened new file (3)
1.txt: (cnt=4) found
2.txt: (cnt=5) found
2.txt: (cnt=6) found
3.txt: opened new file (7)
2.txt: (cnt=8) found
2.txt: (cnt=9) found
3.txt: (cnt=10) found
4.txt: Too many open files, close one: 1.txt:4, 2.txt:9, 3.txt:10
closing 1.txt
4.txt: opened new file (11)
1.txt: Too many open files, close one: 2.txt:9, 3.txt:10, 4.txt:11
closing 2.txt
1.txt: opened new file (12)
1.txt: (cnt=13) found
1.txt: (cnt=14) found
1.txt: (cnt=15) found
5.txt: Too many open files, close one: 3.txt:10, 4.txt:11, 1.txt:15
closing 3.txt
5.txt: opened new file (16)
</c>
<p>...[roboticus]</p>
<p><i>When your only tool is a hammer, all problems look like your thumb.</i></p>
1013641
1013651