Perl jumps to END logic after fileno (Win32)

dchidelf has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Perl jumps to END logic after fileno (Win32) by talexb (Chancellor) on Aug 02, 2017 at 17:12 UTC
Without a self-contained sample, I can speculate that the filehandle you're calling fileno with is broken, or that something is independently stopping your script from running. Apart from those wild guesses, you're going to have to provide us with a bit more information. Alex / talexb / Toronto Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.	[reply]
Re^2: Perl jumps to END logic after fileno (Win32) by dchidelf (Novice) on Aug 09, 2017 at 20:51 UTC
Alright, so in looking deeper at the warnings I am receiving it seems as though windows is likely reusing the OS level filehandle I am pulling out from under the Perl handle via my hardclose logic. Then when perl implicitly closes the Perl handle it is closing whatever ended up reusing that OS handle causing unspecified behavior. So perhaps to avoid this I should ask for assistance in accomplishing my goal. I want to spawn a thread then close a handle that was open in the main thread prior to the creation of the thread. In experimenting with this I was not able to have the process actually release the OS file handle. I don't remember the details, but I was not able to call close from both the main thread and "child" thread.	[reply]
Re^3: Perl jumps to END logic after fileno (Win32) by Anonymous Monk on Aug 09, 2017 at 23:49 UTC
Hi, What? #!/usr/bin/perl -- use strict; use warnings; use autodie qw/ open close /; use threads stack_size => 4096; open FOO, '<', __FILE__ or die $!; if( @ARGV ){ #~ threads->create(sub { use autodie qw/ close /; close FOO; })->d +etach; threads->create(sub { close FOO; })->join; threads->create(sub { close FOO; ## close FOO; ## this one autodies, "terminated abnormally:" eval { close FOO; 1 } or warn $@; return; })->join; close FOO; close FOO; ## this one autodies } else { close FOO; close FOO; ## this one autodies } __END__ $ perl threads-close-filehandle.pl Can't close filehandle 'FOO': 'Bad file descriptor' at threads-close-f +ilehandle.pl line 22 $ perl threads-close-filehandle.pl 1 Can't close filehandle 'FOO': 'Bad file descriptor' at threads-close-f +ilehandle.pl line 15 Can't close filehandle 'FOO': 'Bad file descriptor' at threads-close-f +ilehandle.pl line 19 [download]	[reply] [d/l]
Re^4: Perl jumps to END logic after fileno (Win32) by dchidelf (Novice) on Aug 10, 2017 at 18:41 UTC
Re^5: Perl jumps to END logic after fileno (Win32) by Anonymous Monk on Aug 10, 2017 at 23:33 UTC
Re^2: Perl jumps to END logic after fileno (Win32) by dchidelf (Novice) on Aug 08, 2017 at 22:06 UTC
I have been able to narrow down the code a little. In doing so I can see that my call to threads->create is attempting to close half of my pipe that I am passing to it. Due to Win32 process peculiarities I need to forcefully close some file handles using a hardclose function. I also needed to close the STDOUT/STDERR handles in my thread, so I wasn't seeing the warnings. Somehow the combination of the warnings and hardcloses causes file descriptors to get corrupted on SOME systems. It is possibly a race condition, as the Perl software is identical between the systems I test on, yet one fails within the first 50 iterations of the loop in this test, whereas the other system has processed over 40000 iterations of the loop with no issues (aside from the warning messages). The behavior also changes when removing the print statements. failure is either the report that either STDOUT or STDERR are gone, or message that new_from_fd failed use threads; use threads::shared; use Time::HiRes; require Win32API::File; Win32API::File->import(":Func",":Misc",":HANDLE_FLAG_"); my $running :shared = 0; my $goreaders :shared = 0; END { print "This is the end\n"; } my $i = 0; while(1) { $running = 0; $goreaders = 0; my $fh = Scalar::Util::openhandle(\STDOUT); if (! defined $fh) { print STDERR "STDOUT is gone\n"; } $fh = Scalar::Util::openhandle(\STDERR); if (! defined $fh) { print STDOUT "STDERR is gone\n"; } pipe my $outread, my $outwrite; pipe my $errread, my $errwrite; my $outthread = threads->create('fhreader', \$goreaders, \$run +ning, $outread); my $errthread = threads->create('fhreader', \$goreaders, \$run +ning, $errread); # wait for threads to initialize lock($running); if ($running < 2) { my $absto = time() + 4; # 4 second timeout do { cond_timedwait($running, $absto) \|\| last; } while ($running < 2); if ($running < 2) { die "reader threads did not start!\n"; } } print "Threads Started\n"; # connect STDOUT/STDERR to the write end of the pipes # save a copy of the original STDOUT/STDERR my $saveIN = IO::Handle->new_from_fd(\STDIN, 'r'); my $saveOUT = IO::Handle->new_from_fd(\STDOUT, 'w'); my $saveERR = IO::Handle->new_from_fd(\STDERR, 'w'); if (! defined $saveOUT) { print STDERR "new_from_fd(STDOUT) failed!!!!!\n"; } if (! defined $saveERR) { print STDERR "new_from_fd(STDERR) failed!!!!!\n"; } pipe STDIN, $inwrite; STDOUT->fdopen($outwrite, 'w'); STDERR->fdopen($errwrite, 'w'); # Need hardclose because our threads hold copies of the handle +s and prevent # the refcnt on the handles from hitting zero and closing the +OS Handle # hardclose will close the OS Handle hardclose($outwrite); hardclose($errwrite); print $saveOUT "Swapped STDOUT/STDERR\n"; # fork subprocess -- unnecessary to demonstrate issue # Win32::Process::Create ... hardclose(\STDIN); hardclose(\STDOUT); hardclose(\STDERR); print $saveOUT "Closed pipes\n"; STDIN->fdopen($saveIN, 'r'); STDOUT->fdopen($saveOUT, 'w'); STDERR->fdopen($saveERR, 'w'); softclose($saveIN); softclose($saveOUT); softclose($saveERR); print STDOUT "Restored STDOUT/STDERR\n"; # signal threads to process { lock($goreaders); $goreaders=1; cond_broadcast($goreaders); } for my $th ($outthread,$errthread) { if (! $th->is_joinable()) { eval { $th->exit(); }; my $quitinTime = Time::HiRes::time() + 2; while($th->is_running() && ($quitinTime > Time::HiRes: +:time())) { Time::HiRes::sleep(0.05); } if ($th->is_running()) { eval { $th->kill('KILL')->detach(); } ; next; } } eval { $th->join(); } ; } printf "Done %d\n", ++$i; } sub fhreader { my ($start, $running, $reader) = @_; { lock($$start); { lock($$running); ++$$running; cond_signal($$running); } cond_wait($$start); # wait to start } while(<$reader>) { } $reader->close(); return(0); } sub hardclose { # not only close the regular Perl IO Handle, but close the und +erlying OS Handle as well # This is primarily useful for closing the handles we had open + when creating a new thread # forcing the handle closed in the entire process, even though + the ithread holds a copy of # the perl handle (can't use :shared on the handle, might yet + be another way though) my (@handles) = @_; for my $h (@handles) { next if(! defined $h); #my $fd = eval { fileno $h }; my $fd = fileno $h; if (defined $fd && $fd >= 0) { my $osfh = FdGetOsFHandle($fd); # get the OS +native file handle if (! CloseHandle($osfh)) { print STDERR "CloseHandle failed\n"; } } else { print STDERR "fileno failed\n"; } $h->close(); } } sub softclose { # normal Perl IO Handle close on each handle passed my (@handles) = @_; for my $h (@handles) { next if (! defined $h); $h->close(); } } [download]	[reply] [d/l]
Re^3: Perl jumps to END logic after fileno (Win32) by Anonymous Monk on Aug 09, 2017 at 01:32 UTC
It is possibly a race condition, as the Perl software is identical between the systems I test on, yet one fails within the first 50 iterations of the loop in this test, whereas the other system has processed over 40000 iterations of the loop with no issues (aside from the warning messages) ... failure is either the report that either STDOUT or STDERR are gone, or message that new_from_fd failed Hi, What is "perl software" and "systems" (versions)? I cannot reproduce on strawberry-perl-5.18.2.2-32bit-portable on old winxp machine	[reply]
Re^4: Perl jumps to END logic after fileno (Win32) by dchidelf (Novice) on Aug 09, 2017 at 13:02 UTC
Re: Perl jumps to END logic after fileno (Win32) by Anonymous Monk on Aug 03, 2017 at 09:50 UTC
Any assistance on what to look for would be very helpful. If you have a problem with your car, do you show a picture of carpart to your mechanic? Cause you're really sure the hubcap ignition is the problem? Yeah, hes like a mechanic, he knows what those things look like, what he needs is the actual malfunctioning machine Translation, show a 5-20 line program that demonstrates the problem you want to solve, not some 8 lines of code you think are the problem	[reply]
Re^2: Perl jumps to END logic after fileno (Win32) by dchidelf (Novice) on Aug 03, 2017 at 15:40 UTC
I understand the difficulty in trying to troubleshoot a script with just the snippet. That is why I am here. The problem is that it is a little of 6000 lines of code with a dependency on an external application and only fails on a subset of the systems that run the tool. I can not actually get the code to fail on my system, but some of the user's systems fail pretty reliably and always on the fileno command. I was mostly looking for any insight anyone might have into what fileno does internally on a Win32 system that might cause the process termination, or even more generally if there is a way to tell what caused perl to jump to it's end blocks. I am just not familiar enough with the perl source code to follow fileno all the way down to it's actual implementation. And I was not aware if there were any status variables that might indicate what caused the jump to the end blocks. Perl is not executing the line of code after the fileno command, and it not throwing an exception. It simply jumps to the cleanup.	[reply]
Re^3: Perl jumps to END logic after fileno (Win32) by Anonymous Monk on Aug 04, 2017 at 07:10 UTC
This is what you need to do, assume the problem is with the code you wrote , because this is the beginning of debugging Copy your 6000 line program and start deleting lines, delete everything that doesn't trigger the problem, stop when you have it down to 20 lines max or you solved it If your 6000 lines of code are in any way organized it shouldn't take very long (15min-1hour) This is the only shortcut "Thinking" the problem lies in fileno and other pieces of code that have worked fine for a decade or two/three is naive -- if there was any merit to the idea you should be able to demonstrate it with ~20 lines of code	[reply]
Re: Perl jumps to END logic after fileno (Win32) by choroba (Cardinal) on Aug 03, 2017 at 17:25 UTC
From the debugging it seems Wx is involved. What other modules are involved? ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l]
Re^2: Perl jumps to END logic after fileno (Win32) by dchidelf (Novice) on Aug 04, 2017 at 13:12 UTC
This should be everything. Note: my version of Thread::Queue isn't the offical 3.09 version. It includes a patch because, at the time, Jerry hadn't included my fix for queue limits into a released version yet. `use Sys::Hostname; use File::Temp; use File::Basename; use File::Path qw/remove_tree/; use Time::Local; use Getopt::Std; use File::Copy qw(copy); use Cwd; use threads; use threads::shared; use Thread::Queue 3.09; # need limit functionality use IO::Select; use Time::HiRes; use Symbol 'gensym'; use POSIX ":sys_wait_h"; # only necessary for Unix, but available in W +in32 use URI::Escape qw(uri_escape uri_unescape); use IO::Socket::INET; require Wx; require Wx::Event;` [download]	[reply] [d/l]
Re: Perl jumps to END logic after fileno (Win32) by dchidelf (Novice) on Aug 02, 2017 at 16:42 UTC
Just to note. I have many calls to fileno in my code that work prior to a failure. The first couple through this loop may be fine, then the next causes the termination.	[reply]


XP is just a number
	PerlMonks