A way to report open file handles a Perl script has open?

nysus has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: A way to report open file handles a Perl script has open? by talexb (Chancellor) on Apr 04, 2017 at 18:15 UTC
Once approach that I can recommend is to open your files inside a scope; once the scope ends, the file handle will automatically be closed. of course, I prefer to explicitly close files handles, but that's just my C background. Something like this: `use autodie; ... my $config_object; { open ( my $config_fh, '<', 'hello.cfg' ); while ( <$config_fh> ) { .. parse config file .. $config_object{ $key } = $something; } } # File handle is now out of scope and therefore closed ..` [download] That's a simple example .. but perhaps you can expand on your question .. why do you think that open file handles are causing you problems? Alex / talexb / Toronto Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.	[reply] [d/l]
Re: A way to report open file handles a Perl script has open? by kennethk (Abbot) on Apr 04, 2017 at 18:11 UTC
Anonymous Monk has answered the immediate question (say my $v=()=`lsof -p $$`), but the following grabbed my attention: leaking file handles like a sieve What is the mechanism here? Lexical filehandles generally address this problem, but will obviously linger if you are stashing them in a persistent data structure. #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.	[reply] [d/l]
Re: A way to report open file handles a Perl script has open? by afoken (Chancellor) on Apr 04, 2017 at 18:18 UTC
A hack highly depending on the OS, in other words: It works on my Linux x86_64 computer. Your mileage may vary. `> perl -E 'open my $f,"<","/etc/hosts"; system "/bin/ls","-lF",qq[/pro +c/$$/fd]' < /etc/passwd total 0 lr-x------ 1 alex users 64 Apr 4 20:17 0 -> /etc/passwd lrwx------ 1 alex users 64 Apr 4 20:17 1 -> /dev/pts/0 lrwx------ 1 alex users 64 Apr 4 20:17 2 -> /dev/pts/0 lr-x------ 1 alex users 64 Apr 4 20:17 3 -> /etc/hosts lr-x------ 1 alex users 64 Apr 4 20:17 4 -> pipe:[2099383] >` [download] See also: https://linux.die.net/man/5/proc https://www.kernel.org/doc/Documentation/filesystems/proc.txt Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply] [d/l]
Re: A way to report open file handles a Perl script has open? by shmem (Chancellor) on Apr 05, 2017 at 01:43 UTC
Is there a way to report how many open file handles are open at once so I can know when I'm being wasteful? Umm... yes, but for now it is - well, a hack. This is a splendid use for inside-out objects, where information about an object is stored in the class which deals out the instances. That way you can ask the class about the state of any known object, if the class provides automatic garbage collection of objects that go out of scope. Perl includes a wonderful inside-out object class which does just this for any class: Hash::Util::FieldHash, based on Alter written by anno. You would need to use modules from the IO family to initialize your file handles, consequently. That way your filehandles become objects and can be registered within the class. The following is a quick hack, monkey-patching IO::File, but if it would be incorporated into IO::File, it would add 2 lines of code at the top level, 1 line of code in each of the methods new and open, and two class methods: open_fds and known_fds to query the class. In fact, known_fds would suffice, if it returns the objects which can then be queried further, but that can lead to more leakage if the variables holding the objects aren't freed. This is for IO::File only; adapting this abomination to other modules of the IO family is left as an excercise for the reader ;-) # file io-open-hack.pl use strict; use warnings; use IO::File; BEGIN{ package IO::File { use Hash::Util::FieldHash qw(id_2obj); Hash::Util::FieldHash::fieldhash my %fh; IO::File::_new = \&IO::File::new; IO::File::_open = \&IO::File::open; { no warnings 'redefine'; IO::File::new = sub { my $io = &_new; $fh{$io} = { new => "@{[caller]}" }; $io; }; IO::File::open = sub { my $io = $_[0]; $fh{$io}->{open} = "$_[1] at @{[caller]}"; &_open; }; } sub open_fds { grep { $_->{fh}->opened } known_fds(); } sub known_fds { map { { new => $fh{$_}->{new}, # next is dubious, it leaks the object... fh => id_2obj($_), # ... so we should return it stringified? #fh => "@{[id_2obj($_)]}", open => $fh{$_}->{open}, } } keys %fh; } } } package main; use IO::File; { my $fh = IO::File->new(); $fh->open( "blorfldyick",'>'); print $fh "foo\n"; print "after open:\n"; report_fds($_) for qw(open known); close $fh; print "after close:\n"; report_fds($_) for qw(open known); } # $fh is out of scope here print "after scope:\n"; report_fds($_) for qw(open known); sub report_fds { my $type = shift; my $meth = "${type}_fds"; my @fds = IO::File->$meth(); print " $type files:"; if (@fds) { print $/; print " fh: $_->{fh}\n" . " new: $_->{new}\n" . " open: $_->{open}\n" for @fds; } else { print " none\n"; } } __END__ after open: open files: fh: IO::File=GLOB(0x137a3b8) new: main io-open-hack.pl 45 open: blorfldyick at main io-open-hack.pl 46 known files: fh: IO::File=GLOB(0x137a3b8) new: main io-open-hack.pl 45 open: blorfldyick at main io-open-hack.pl 46 after close: open files: none known files: fh: IO::File=GLOB(0x137a3b8) new: main io-open-hack.pl 45 open: blorfldyick at main io-open-hack.pl 46 after scope: open files: none known files: none [download] This feature could be off by default and turned on inside IO::File via a debug import tag or some such. perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'	[reply] [d/l]
Re: A way to report open file handles a Perl script has open? by mr_mischief (Monsignor) on Apr 05, 2017 at 00:26 UTC
If I'm opening multiple files in a way that might lead to a leak (that is, a dynamic number of them rather than bespoke input/output file handles for an operation) then I keep track of related ones in a data aggregate. An array of file handles or a hash of descriptive keys with values that are file handles makes keeping information about them in one place more simple. If you're using different file paths or have different IP hosts connecting to your sockets, perhaps a hash like this would be helpful.: `while ( readdir $dirhandle ) { if ( -f $_ ) { if ( open $file{ $_ }, '<', $_ ) { warn sprintf q{I have %d files open from the working direc +tory.}, scalar keys %file; # do stuff } else { delete $files{ $_ }; # report error } } } my @files = keys %file; for ( @files ) { # do something that for some reason requires all the files in the +directory open at once } for ( @files ) { close $file{ $_ }; delete $file{ $_ }; }` [download] The easy solution is to explicitly close file handles as soon as your code is done with them or to open them as lexical handles. If you need to keep multiple file handles open over long periods, close them when you're done with them. Using a hash, you can easily delete the keys for files you no longer need after they're closed, too. Using an array you can shift or pop them if they're closed in array order or use splice. An array or hash and the associated item counts for those structures are easy ways to keep track of things. I have to wonder, how are you storing your file handles in a way that you can't readily know how many you have open? Are you dynamically creating symbolic scalars for file handles? Don't do that. Are you storing them in arrays or hashes and just never closing them?	[reply] [d/l]
Re: A way to report open file handles a Perl script has open? -- using ${^LAST_FH} ? by Discipulus (Canon) on Apr 05, 2017 at 08:39 UTC
Hello nysus, For sure the best thing to do is to open always lexical filehandles to let Perl close them for you. The habit to close them esplicitally is a sane one anyway. Extra scope can help if you have many declared into the top scope. But the strange behaviour you announce (leaking file handles like a sieve) I'd expand the mr_mischief's solution: take count of what you open. You can override `open` builtin function very soon in the program or in a `BEGIN` block, and you can profit of the `${^LAST_FH}` special variable: if I understand the doc correctly it tracks the last open `$fh` (only read-fh ?) taking a reference to it You probably need to override `close` to erase from your tracking variable. An `END` block can dump the tracking datastructure. So given a `%fhs` in the outher scope you can store there your fh `# TOTALLY UNTESTED!! no warning 'redefine'; my %fhs; BEGIN { CORE::GLOBAL::open = sub { my ($fh,$mode,$path) = @_; # open the file $fhs{fileno($fh)} = $path.' '.${^LAST_FH}; }; } END{ print "Files already opened:\n", map {"fileno $_ $fhs{$_}\n"} sor +t keys %fhs; }` [download] PS sorry the node was composed on a spare PC where was impossible to test anything.. L There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l] [select]
Re^2: A way to report open file handles a Perl script has open? -- using ${^LAST_FH} ? by shmem (Chancellor) on Apr 05, 2017 at 12:57 UTC
Nice. A sub wrapping open before overriding is missing... and for close. Combining your and my approach: At toplevel: use Hash::Util::FieldHash qw(id_2obj); my %fd; BEGIN{ Hash::Util::FieldHash::fieldhash %fd; my $open = sub { @_ > 2 ? open $_[0],$_[1],$_[2] : open $_[0], $_[1]; }; my $close = sub { close $_[0] }; CORE::GLOBAL::open = sub { my $result = $open->(@_); if ($result) { $fd{$_[0]} = join " ",@_[1,2],caller; } $result; }; CORE::GLOBAL::close = sub { my $result = $close->(@_); if ($result) { $fd{$_[0]} .= " (closed)"; } else { $fd{$_[0]} .= " (close failed)"; } $result; }; } open my $devnull, '>/dev/null' or die; { open my $fh, '>', "blorfldyick" or die; print $fh "foo!\n" or die; print "(1) ", id_2obj($_), " => $fd{$_}\n" for keys %fd; close $fh; print "after close\n"; print "(2) ", id_2obj($_), " => $fd{$_}\n" for keys %fd; } print "after scope:\n"; print "(3) ", id_2obj($_), " => $fd{$_}\n" for keys %fd; open my $devzero, '<', '/dev/zero' or die; # time passes... print "later:\n"; print "(4) ", id_2obj($_), " => $fd{$_}\n" for keys %fd; __END__ (1) GLOB(0xb22190) => >/dev/null main open.pl 29 (1) GLOB(0xb63780) => > blorfldyick main open.pl 32 after close (2) GLOB(0xb22190) => >/dev/null main open.pl 29 (2) GLOB(0xb63780) => > blorfldyick main open.pl 32 (closed) after scope: (3) GLOB(0xb22190) => >/dev/null main open.pl 29 later: (4) GLOB(0xb363b8) => < /dev/zero main open.pl 42 (4) GLOB(0xb22190) => >/dev/null main open.pl 29 [download] Combine with Time::HiRes and you can sort the history. update: done, made into a package. perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'	[reply] [d/l]
Re^3: A way to report open file handles a Perl script has open? -- using ${^LAST_FH} ? by huck (Prior) on Apr 05, 2017 at 13:14 UTC
That is a neat trick i need to remember	[reply]
Re: A way to report open file handles a Perl script has open? by Anonymous Monk on Apr 04, 2017 at 18:02 UTC
`man lsof`	[reply]
Re: A way to report open file handles a Perl script has open? by Anonymous Monk on Apr 04, 2017 at 23:24 UTC
I doubt leaking filehandles are the problem because most OS have limits, even if configurable, so you'd get a Re: "Too Many Files Open" problem before the actual filehandles are the problem Use one of the leaktrace/nytprof modules to find the real problems	[reply]
Re: A way to report open file handles a Perl script has open? by karlgoethebier (Abbot) on Apr 05, 2017 at 09:00 UTC
Unix::Lsof? �The Crux of the Biscuit is the Apostrophe� Furthermore I consider that Donald Trump must be impeached as soon as possible	[reply]
Re: A way to report open file handles a Perl script has open? by pryrt (Abbot) on Apr 05, 2017 at 14:21 UTC
Windows solution: Download MS TechNet::SysInternals::Handle at https://download.sysinternals.com/files/Handle.zip. Unzip into same directory as the script below, or into PATH if you want it long term use warnings; use strict; sub ph { print `handle -nobanner -u -p $$`; # shows handle id (hex), t +ype (file), mode (rw-), and filename print `handle -nobanner -u blah`; # shows exe, pid, type, ow +ner, id (hex), filename print `handle -nobanner -s -p $$`; # shows summary of all the + different handle types open by that process print 'HIT ENTER'; <STDIN>; } local $\ = $/; print $$; ph(); open my $wr, '>', 'blah'; ph(); open my $rd, '<', 'blah'; ph(); open my $wrnul, '>', 'NUL'; open my $rdnul, '<', 'NUL'; close $wr; close $rd; ph(); [download] It turns out I had recently (last week) had a problem with EU::MM's `dmake distdir && ... && dmake realclean` not being able to delete the distribution directory because something (I still don't know what) was holding the directory open. In fact, whatever it was wouldn't let me even delete it after closing all my cmd.exe and explorer.exe windows -- I had to reboot to get it. Unfortunately, my search-foo wasn't good enough last week to find `handle.exe`, so I didn't figure out what I'd done to hold it open. Thanks to the `lsof` hints in this thread, today I was able to find lsof for windows => `handle.exe` and the similar GUI-based TechNet::SysInternals::Process Explorer. So ++thanks for all the contributors to this thread	[reply] [d/l] [select]


P is for Practical
	PerlMonks