Beefy Boxes and Bandwidth Generously Provided by pair Networks Joe
Perl Monk, Perl Meditation
 
PerlMonks  

Proposal how to make modules using fork more portable

by Anonymous Monk
on Mar 30, 2011 at 09:47 UTC ( #896360=perlmeditation: print w/ replies, xml ) Need Help??

Hej, this is my first post to so have some forgiveness!

BACKGROUND:

I have used Perl for more than 20 years, but I am quit new to installing from CPAN using the cpan-utility.

I am using:
  • Strawberry-perl-5.12.2.0
  • Perl 5, version 12, subversion 2 (v5.12.2) built for MSWin32-x86-multi-thread
  • Windows 7 Home Premium with Service Pack 1

After some restarts and using force to inhibit the tests I succeeded to install Padre and Dancer.

During all my tries the test of Test::TCP seemed to be a problem, so I started trying to find the cause of blocking of the tests. Sometimes I was using Padre, but in many cases it also blocked. Sometimes Padre even was impossible to kill, so I had to restart Windows.

After a lot of tries I wrote: https://rt.cpan.org/Public/Bug/Display.html?id=66437. See also https://rt.cpan.org/Public/Bug/Display.html?id=66016.

PROPOSAL

To add an advice to make forked processes shut down themselves, by the implementation of modules intended to be portable. The purpose with this is to avoid the related problem in the emulation of fork in Windows, using kill.

In the documentation of fork there could be a warning about the problem and an reference to perlfork http://perldoc.perl.org/perlfork.html.

In perlfork there should be an advice to make forked processes shut down themselves. The purpose of this is to avoid using kill(9, $child).

MOTIVATION

Using kill(9, $child) in Windows, where fork is emulated in the perl interpreter, there is a probability to get a blocked process.

Work is going on to improve the perlfork implementation in Windows (see: http://www.gossamer-threads.com/lists/perl/porters/261805), but it will take a long time until the new versions of Perl are used in all places.

Comment on Proposal how to make modules using fork more portable
Re: Proposal how to make modules using fork more portable
by moritz (Cardinal) on Mar 30, 2011 at 11:35 UTC

      I will try to create a patch, but it will take some weeks.

      Before creating the patch, could I get some feedback?

      This is my proposal to changes:

      Added at end of “=item fork” in pod/perlfunc.pod

      On some platforms such as Windows, where the fork() system call is not available, Perl can be built to emulate fork()in the Perl interpreter. The emulation is designed to, at the level of the Perl program, be as compatible as possible with the “Unix” fork(). It has however some limitation that has to be considered, for all in code intended to be portable. See L<perlfork> for more details.

      Added at end of =item kill SIGNAL, LIST in pod/perlfunc.pod

      On some platforms such as Windows where the fork() system call is not available. Perl can be built to emulate fork() at the interpreter level. The emulation has some limitation related to kill that has to be considered, for code running on Windows and in code intended to be portable. See L<perlfork> for more details.

      Added at the end of =item kill() in pod/perlfork.pod

      There is a probability that the process which implements all the pseudo-processes is blocked.

      New section added before =head1 BUGS in pod/perlfork.pod

      =head1 PORTABLE PERL CODE

      In portable Perl code, kill(9, $child) must not be used. The forked processes must terminate themselves.

      The kill(9) protocol is “taken from Unix,” and it does not have a good parallel in the Windows world. In Windows forked processes are implemented as pseudo-processes.

      To use kill(9, $child) for pseudo-processes is unsafe and there is a probability that the process which implements all the pseudo-processes is blocked. The outcome of kill on a pseudo-process, depends on the timing in the operating system, and code that has worked, suddenly can fail, resulting in errors which are difficult to find.

        =head1 PORTABLE PERL CODE In portable Perl code, kill(9, $child) must not be used. The forked pr +ocesses must terminate themselves.

        This is wrong. The problem only exists for pseudo-processes, not for other processes.

        To me, the following sentence is not sufficiently clear:   “There is a probability that the process which implements all the pseudo-processes is blocked.”

        As an English sentence, the beginning clause makes a clumsy use of third-person passive voice.   (Should I be placidly ignoring an 0.01% probability, or running away for dear life from 99.9?)   The remainder of the sentence might be technically correct, but it does not convey to me what the sentence means to me.   What do I do?

        With regard to the final paragraph, well, I can’t close my eyes and readily visualize what a “pseudo-process” might be, nor does the text (as I read it on my computer now) contain a prior use of that term.   (Note that I am not, right now, reading that perlpod on a Win32 computer.)   Whether I am “familiar with Unix but not with Win32,” or vice-versa, or I am equally acquainted with both, I am not prepared by this sentence to know what I am dealing with, nor to understand both its superficial and its deeper implications.

        The same line of reasoning could also be applied to the earlier sentence ... “the emulation has some limitation related to kill.”   (Oh, my!   And what on earth might that “limitation” be, and why do you say, “some?”)   Once again, I need to understand what it means to me.   The documentation should have a clear meaning, and a definite, un-ambiguous answer for:   “so what?”

        (Note:   I am delighted to see you working on this documentation!   I mean my feedback only in the most constructive way possible.)

Re: Proposal how to make modules using fork more portable
by BrowserUk (Pope) on Mar 30, 2011 at 12:14 UTC

    Personally, I would translate any normally fatal signal into a direct call to TerminateThread() if the process ID that is the subject of the signal is a pseudo-process ID.

    Whilst TerminateThread() is usually a bad thing, given the aim here is to allow the main thread to get past a wait or waitpid and exit the process, when the thread will be summarily abandoned anyway, it amounts to the same thing.

    The advantage is that you supply TerminateThread() with an exit code, which will then be reported back (via GetExitCodeThread()) as if it were the return value from the thread proc in the normal way, hence to the the rest of perl, and the program, it will seems as if the thread just ended.

    Of course, clean-up will not have been done, so it will likely leak like a bitch and possibly cause deadlocks through unreleased semaphores, mutexes and critsecs. But if the goal is just to allow the main process to terminate, that doesn't matter.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Proposal how to make modules using fork more portable
by sundialsvc4 (Monsignor) on Mar 30, 2011 at 18:14 UTC

    I am always of the opinion that threads and processes should terminate themselves, and that they should, where possible, “stick around, waiting for more work to do,” instead of coming into existence to service one request and then flaming-out.

    The kill(9) protocol is definitely one that was “taken from Unix,” and it really does not have a good parallel in the Windows (i.e. DEC VAX 11/780...) world-view.   And yet, I have never recommended “management by the use of overwhelming deadly-force”   ;-)   anywhere.

    Building process contexts, and tearing them down, are best thought of as expensive operations.   And, the maximum number of processes that may exist at one time is usually something that you want to govern.

Re: Proposal how to make modules using fork more portable
by sundialsvc4 (Monsignor) on Mar 31, 2011 at 23:56 UTC

    Please... welcome!   Don’t remain “Anonymous,” Monk!   You will very quickly appreciate being able to locate “your” posts and threads, and of course, your considered responses on other matters are looked forward to!

Re: Proposal how to make modules using fork more portable
by Anonymous Monk on Apr 01, 2011 at 08:51 UTC

    Thanks for your feedback, corion and sundialsvc4!

    This is a updated proposal of changes.

    Changes to pod/perlfunc.pod

    Added at end of fork

    On some platforms such as Windows, where the fork() system call is not + available, Perl can be built to emulate fork() in the Perl interpreter. The emula +tion is designed to, at the level of the Perl program, be as compatible as possible with th +e "Unix" fork(). However it has limitation that has to be considered in code intended t +o be portable. See L<perlfork> for more details.

    Added at end of kill

    On some platforms such as Windows where the fork() system call is not +available. Perl can be built to emulate fork() at the interpreter level. This emulation has limitation related to kill that has to be considere +d, for code running on Windows and in code intended to be portable. See L<perlfork> for more details.

    Changes to pod/perlfork.pod

    perlfork

    Added at the end of =item kill()

    The process which implements the pseudo-processes can be blocked and t +he Perl interpreter hangs.

    New section added before =head1 BUGS

    =head1 PORTABLE PERL CODE In portable Perl code, kill(9, $child) must not be used on forked proc +esses. Forked process are in Windows implemented as a pseudo-processes. To use kill(9, $child) on pseudo-processes is unsafe. The process which implements the pseudo-processes can be blocked and t +he Perl interpreter hangs. The outcome of kill on a pseudo-process is unpredictable. It depends on the timing in the Windows operating system. Code that has worked, suddenly can fail, resulting in errors which are + difficult to find.

      The last sentence can be omitted.   Just saying that there is a potential race-condition in the Windows implementation is enough.

      The statement concerning pseudo-processes “can be blocked,” etc., still feels odd to me, but it could “ship” as-is.

        Just saying that there is a potential race-condition in the Windows implementation is enough.

        There is no "race condition". Throwing out random cool phrases at problems of which you have no fundamental understanding helps no one.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

      For the record, this is all garbage.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      For the record, this is all garbage.

      The perceived 'problem' with kill has nothing whatsoever to do with perl's fork emulation.

      It has everything to do with signals not being able to interrupt blocking IO.

      It is the same problem that means that this doesn't work:

      { alarm 10; my $in = <STDIN>; }

      Signals won't interrupt IO. Therefore, if you send a signal to a pseudo-process (thread) that is currently blocked doing IO, it won't interrupt it until the IO completes. The limitation is with the signals, not the pseudo-fork.

      And documenting a limitation of signals, as a bug with fork emulation is just dumb. Presumably, p5p will reject the attempt.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Proposal how to make modules using fork more portable
by Anonymous Monk on Apr 04, 2011 at 05:19 UTC

    Thanks BrowserUk, corion and sundialsvc4 for the interesting discussion!

    (I can not follow all the implementation details.) The discussion has made me aware of that there are many things to consider to write portable Perl code. Portability issues are dealt with in separate documents or sections and this make programmers working on the reference platform (Unix/Linux) less aware of problems in other platforms.

    PROPOSAL

    To add information about portability issues in the "Alphabetical Listing of Perl Functions" in perlfunc.

    This can be done by at the end of function descriptions adding links to the the corresponding entry in "Alphabetical Listing of Perl Functions" in perlport.

    UNPREDICTABLE RESULT OF KILL ON PSEUDO-PROCESS

    Observations

    1) This always blocks in Windows 7, but not on Windows 2000:

    perl -e "if ($pid=fork){kill(9, $pid);} else {sleep 1000 }"

    2) This sometimes blocks and sometimes gives the wrong exit value 9 in Windows 7:

    perl -e "if ($pid=fork){Win32::Sleep(0); kill(9, $pid);} else {sleep 1000}"

    3) Running this batch file a number of times on Windows 7:

    @echo off set count=0 :loop set /a count=%count%+1 echo Count %count% @echo on perl -e "if ($pid=fork){Win32::Sleep(0); kill(9, $pid); Win32::Sleep(0 +)} else {sleep 1000}" if errorlevel 1 goto exit @echo off goto loop :exit ECHO.%ERRORLEVEL%

    gave the results:

    • Perl is blocked after between 30 to several thousand loops.
    • Sometimes perl exits with the wrong exit value 9.
    • One time using the same command window for more than 20 times, perl exited all the times after less the 10 loops with the wrong exit value 9. Is there a missing initialization in the Perl (or Windows) implementation?

    4) The installations of module on Windows 7 using install in cpan frequently hangs in the tests.

    Conclusions and implications

  • There seem to be a Perl portability problem between different versions of Windows
  • If a test uses kill on a pseudo-process, the test must be run many times to detect the infrequent errors. It is not enough to run a test one time.
  • There is problems to install modules using Windows 7 and Strawberry Perl. Cpan install blocks and the installation has to be restarted.
  • A better way to inform about the problem?

    I tried to describe the problems above with:

    The outcome of kill on a pseudo-process is unpredictable. It depends on the timing in the Windows operating system. Code that has worked, suddenly can fail, resulting in errors which are + difficult to find.

    My intention was do describe the consequence of the implementation (as opposed to describe the implementation and forcing the Perl programmer to draw the conclusions).

    How can this be stated in a better way?

    CAN WARNINGS BE USED TO IMPROVE PORTABILITY?

    A way to support creation of more portable code could be to extend the "Perl Lexical Warnings" system, with optional warnings given when non portable Perl functions (and other non portable constructs) are used. Is this a feasible way to go?

    OBSERVATIONS and OTHER THOUGHS

    The discussion indicates that to write good Perl code you need to know Perl, the implementation of Perl and the underlying operating system. To write portable Perl code you need also to know the Perl implementation in all used platforms and in the underlying operating systems.

    This puts very high demands on Perl programmers supposed to create portable modules and programs!

    How is portability solved in Python and other dynamic languages? Can "Perl" learn something?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://896360]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2014-04-18 12:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (466 votes), past polls