Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Threads and signals in Windows

by bojinlund (Curate)
on Mar 08, 2013 at 09:39 UTC ( #1022364=perlmeditation: print w/ replies, xml ) Need Help??

Hej!

The purpose of this post is to make me and hopefully also other to understand more about the handling of signal in Windows. Knowledge about signals in Windows is essential, if you want to implement a portable module which uses signals.

I have several times been “hit” by signals in Windows, mainly as a user of modules. I have also tried to understand how signals can be used in Windows. In the Perl documentation I have only found some sentences about it. If you are a UNIX/Linux user and want to write portable code, you get probably very little help from the documentation. Without clear statements of what is wrong it is also difficult to send bug reports.

Super Search gives a lot of information about the topic, but often in specific situation. The results from Super Search show that this is a difficult area. It also show that there is a lot of knowledge in the Monastery.

Some of the background to my proposals and questions are at the end of the post.

What could be done?

Here follows a list of things that could be done to improve portability of modules, in particular from UNIX to Windows. Some of them are perhaps not realistic but can anyway trigger other ideas.

Discourage use of signals

State that: “If you want to write cross-platform code, you shouldn't use signals, full stop.”

Perlwin32 in the Perl documentation says: “Using signals under this port should currently be considered unsupported.” If this still applies, this information should also be placed so module developer using other operating system is aware of it and do not use signals in modules intended to be portable.

Depreciate signals between threads in Windows

Eventually make it impossible from Perl to use signals between threads in Windows.

If you use signal in a wrong way there is a risk to also disturb thing outside the Perl program. Using signal in an inappropriate way can result in a stochastic behaviour of Perl programs.

Discourage use of fork

Make people aware of the big difference between fork in UNIX and Windows. In many cases will the differences result in portability problems.

To support the moved from fork, there should be examples showing how to, in a portable way, solve function typically solved by using fork.

Depreciate fork in Windows

Phase out fork in Windows.

New pragma use portable

Introduce a pragma which issues a warning when non portable functions are used.

Use Perl::Critic to enforce portability

Use the Perl::Critic extensible framework to enforce coding guidelines for portability. It can also be used to indicate “dangerous” functions and constructs.

Questions

Portable alternatives

Is threads a portable alternative to fork?

Is there any portable and feasible ITC alternative to signals in Windows?

Documentation of signals in Windows

In “win32/win32.c” there is a mapping between signals and Windows events. You can also read things like “the child may be blocked in a system call and never receive the signal”.

Where can I find information about this in the documentation of Perl?

Background

TerminateThread is a dangerous function

In the win32 implementation of Perl, ”kill -9 style un-graceful exit” is using ”TerminateThread”.

The “Remarks” in the documention of TerminateThread contains:

TerminateThread is used to cause a thread to exit. When this occurs, the target thread has no chance to execute any user-mode code. DLLs attached to the thread are not notified that the thread is terminating. The system frees the thread's initial stack.

...

TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination. For example, TerminateThread can result in the following problems:

  • If the target thread owns a critical section, the critical section will not be released.
  • If the target thread is allocating memory from the heap, the heap lock will not be released.
  • If the target thread is executing certain kernel32 calls when it is terminated, the kernel32 state for the thread's process could be inconsistent.
  • If the target thread is manipulating the global state of a shared DLL, the state of the DLL could be destroyed, affecting other users of the DLL.

Threads and processes should terminate themselves

Re: Proposal how to make modules using fork more portable

IPC and ITC are different but mixed up

The common fork model is to create a (child) process and the communication between parent and child is Inter Process Communication (IPC).

However in Windows fork creates a (child) thread within the process. The communication between parent and child in Windows is an Inter Thread Communication (ITC). The separation between processes are different than that between threads. Threads are typically sharing more things than processes.

An ITC signal in Windows do not interrupt a blocked I/O operation.

“Signals and Windows: Very limited emulation by Perl. Not reliable.”

“Signals and threads: They are not a useful mechanism of Inter-Thread Communications.”

Stochastic behaviour of Perl program in Windows

Using signal in an inappropriate way can result in a stochastic behaviour of Perl programs. I have seen cases when this not just influence the Perl program, but also other processes in the Windows system. In some cases it is even not possible to switch off the system in the normal way.

I would hesitate to use a program with a stochastic behaviour in a production system.

Another consequences of the "none-deterministic" behaviour is that the module tests should be run several times.

Best Regards

Bo Johansson

Comment on Threads and signals in Windows
Re: Threads and signals in Windows
by BrowserUk (Pope) on Mar 08, 2013 at 10:37 UTC
    Using signal in an inappropriate way can result in a stochastic behaviour of Perl programs. I have seen cases when this not just influence the Perl program, but also other processes in the Windows system. In some cases it is even not possible to switch off the system in the normal way.

    Sorry n'all, but that is garbage! Whilst you may have seen some symptoms like these and have attributed them to the use of signals; I'll bet that you cannot post code to demonstrate it.

    As for the rest: Unix Signals are (one of several) "Unfixable designs". And most all the caveats and warnings you quoted from the MS docs for TerminateThread() are equally applicable to causing any piece of code to be interrupted without resumption using a longjump. Ie. Signals.

    As an IPC mechanism, that only conveys 1 bit of information and is edge triggered -- ie. missable unless you happen to be ready for them -- signals are a piss poor substitute for proper IPC mechanisms.

    And the need to use them to abandon in-progress processing is a sign of badly designed code. Describe the use case for using them and I'll outline a better way of tackling the problem. (Though it may or may not be currently available from Perl.)


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      BrowserUk, thank you! I appreciate your straightforward answers.

      Sorry n'all, but that is garbage! Whilst you may have seen some symptoms like these and have attributed them to the use of signals; I'll bet that you cannot post code to demonstrate it.
      This is a case of, as I believe, an inappropriate usage of signals, and the related symptoms of system degeneration. It is from https://rt.cpan.org/Public/Bug/Display.html?id=66437
      I am using:
      • Strawberry-perl-5.12.2.0
      • Perl 5, version 12, subversion 2 (v5.12.2) built for MSWin32-x86-multi-thread
      • Windows 7 Home Premium with Service Pack 1

      I have used version 1.11 and 1.12 of Test::TCP.

      Problem: The test in Test::TCP is blocking. Get the system in state, that it must be restarted. It is sometimes not possible to kill the blocked processes. I even get errors like “Can't spawn "cmd.exe"” when using the system call. It indicates a degeneration of the perl interpreter.

      One example of stochastic behavior (from https://rt.cpan.org/Public/Bug/Display.html?id=66016) is running this batch file:
      @echo off set count=0 :loop set /a count=%count%+1 echo Count %count% @echo on perl -e "if ($pid=fork){Win32::Sleep(0); kill(9, $pid); Win32::Sleep(0 +)}else {sleep 1000}” if errorlevel 1 goto exit @echo off goto loop :exit ECHO.%ERRORLEVEL%

      Examples of results I got was:

      • Blocking after 38 to 2806 loops.
      • Running more than 7000 loops.
      • Exiting with the value 9 after 1-7 loops.

      This was done using Strawberry-perl-5.12.2.0. The patch http://perl5.git.perl.org/perl.git/commitdiff/82e24582 removed most of the problem but not all. It was also found that version of the Windows operating system had different behavior.

      As for the rest: Unix Signals are (one of several) "Unfixable designs". And most all the caveats and warnings you quoted from the MS docs for TerminateThread() are equally applicable to causing any piece of code to be interrupted without resumption using a longjump. Ie. Signals.

      As an IPC mechanism, that only conveys 1 bit of information and is edge triggered -- ie. missable unless you happen to be ready for them -- signals are a piss poor substitute for proper IPC mechanisms.

      And the need to use them to abandon in-progress processing is a sign of badly designed code. Describe the use case for using them and I'll outline a better way of tackling the problem. (Though it may or may not be currently available from Perl.)

      I interpret your standpoint as:

      • Many implementations of signals are "unfixable". Not just the one in the implementation of Perl in Windows.
      • Signals are no proper IPC mechanisms.

      Could you accept (or even recommend) the addition of something like this:

      It is strongly advice against using signals for Inter Process Communication. This is particular important in Perl code intended to be portable.

      to the Perl documentation? The addition could be done to the function kill in the Perl Language reference.

        One example of stochastic behavior...

        You reinvented a fork bomb. Except on windows, fork actually spawns a thread; so it might better be termed a thread-bomb in this case.

        Try feeding this:  :(){ :|:& };: to bash on a *nix system and see how deterministic the DoS is.

        You are also asking the child 'process' to kill its parent 'process'; but on windows they are actually just threads within the same process. Ie. You are attempting to kill your own process. In addition, you are also thread-bombing the system from within that same process. Is there any wonder that the process may not respond to the request in a predictable manner?

        And you are using a random delay -- sleep 0 equates to "relinquish the rest of the current timeslice" which is an unknowable quantum of time, that could range from 0 microseconds -- if the timeslice was about to end anyway; or if there are no other processes in the system immediately eligible to run -- to N * q where N is the number of other processes and threads in the system that are eligible to run; and q is the scheduler quantum which can vary from ~5 to ~180 microseconds depending upon: a) the version of Windows; b) how that version is configured (workstation or server; foreground or background priority); c) a whole bunch of other factors.

        In other words; you have programmed a random delay into your program as surely as if you had written Win32::Sleep( rand()*100 ).

        Of course the behaviour is "stochastic"; That's how you programmed it to be!

        Could you accept (or even recommend) the addition of something like this:

        Neither!

        I use kill (on windows) to good effect all the time. The difference is that I have spent enough time to understand the limitations of the emulation on my platform. I understand that they give me a fairly crude interface to the native Console control handler functionality from Perl; and that -- used carefully -- can provide functionality that I might otherwise have to dip into Inline::C or XS to gain access to.

        What I would say; and have said many times; is do not attempt to use fork & the windows signals emulation to try to port *nix idioms to windows; because you will be sadly let down.

        The basic problem here is the entire attempt to make Windows look like or work like some variant of *nix. It is simply far easier to write two scripts -- one for each platform -- than to try and construct and maintain one that will operate correctly on both platforms.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Threads and signals in Windows
by zentara (Archbishop) on Mar 08, 2013 at 12:34 UTC
    It sounds like you are a MSWindows promoter and are trying to alter Perl's development to suit Windows by taking away all the useful Unix tools ... how is that being cross-platform? Why should Unix/Linux developers give a damn about Window's problems? After all, most of our so-called cyber-security dangers come from the widespread use of the inferior Windows OS.

    I say phase out MSWindows, and move everyone to Linux.


    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
      zentara thank you for your comment!
      It sounds like you are a MSWindows promoter and are trying to alter Perl's development to suit Windows by taking away all the useful Unix tools ... how is that being cross-platform? Why should Unix/Linux developers give a damn about Window's problems?

      No, I am NOT a MSWindows promoter! However in many cases I have been forced to use Windows and I am mainly a user of it. I want, as you, Perl to be cross-platform and I do not want to remove any thing!

      Not all modules need to be portable between the different implementations of Perl. However I want it to be easy to develop a portable module. If you want to write portable code you need to consider the limitations of the Perl implementation in all the intended target systems. The current Perl documentation gives very little support to avoid non portable constructs. But even with good documentation many non portable function will been used by mistake. This is one of the reasons for my sometimes non realistic proposals.

      When I started to use Strawberry Perl http://strawberryperl.com/ I got the possibility to install modules from CPAN. I was a big improvement for me. The downside was the difficulties to know which module can be used in Windows.

      The Windows operating system seems to have less good resistance for programming errors. Installing some CPAN modules degenerate the Windows system and it has to be repaired. This is especially a problem running smoke test on Windows.

      The fact that your system can be harmed by installing Perl modules, can and probably will, give Perl bad reputation.

      So I believe Unix/Linux developers should give a damn about Window's problems at least when developing portable modules.

        The current Perl documentation gives very little support to avoid non portable constructs

        perlport

        When I started to use Strawberry Perl http://strawberryperl.com/ I got the possibility to install modules from CPAN.

        That is such a clear indication of someone who doesn't know anything about Windows.

        I've been installing from CPAN for years; but I don't use Strawberry Perl.

        Using SP doesn't make it easier to use CPAN; or Perl. It only makes it easier to pretend you aren't using Windows.

        And that is counter productive because it just means that people blames Windows for the limitations of the Perl port to Windows.

        Which means the Perl Port never improves because the blame for problems is wrongly attributed.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Threads and signals in Windows
by Anonymous Monk on Mar 09, 2013 at 02:01 UTC

    Is threads a portable alternative to fork?

    No, but it is an alternative to forks :p

Re: Threads and signals in Windows
by sundialsvc4 (Monsignor) on Mar 11, 2013 at 16:08 UTC

    Excellent comments, BrowserUK.   Very informative.

    My take on the matter is that, “if you are killing threads or processes, or giving up time slices, you’re going about it the wrong way.”   The timing of threads/processes is and must be completely unpredictable.   Furthermore, the things that they do are things that they need to be able to finish doing.   Instead of being “killed,” they should be “informed that it is now time for them to please roll over and die.”   Therefore, the consistently best and most-stable arrangement, I think, is to construct persistent threads (designed to handle many units of work in their lifetime), and to connect them with flexible hoses ... i.e. IPC queues.   Control the system with messages.   Indicate that it’s time to die with a shared flag, and also send a message to that effect so that any thread that’s listening on a queue will wake up, read it, put its own house in order, and die.   You can reliably implement this protocol on any system, using existing CPAN packages, thereby neatly eliding yourself out of the entire cross-platform question.   Somebody else has already done it for you.   Timing problems are eliminated by adopting a better and more rugged design.

    “When faced with an edge-case, do the smart thing ... stay away from the edge.   Looks like it’s a long way down.”

      Pure, unadulterated garbage.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      Excellent comments, BrowserUK. Very informative.

      Invoking the name failed again, give it up now?

        (Shrug...)   I have long ago resigned myself to the realization that there is nothing I can do or say to change this man’s opinions about me, and so I spend very little time thinking about that.   Furthermore, that’s not why I said what I did.   I meant it, nothing more or less.

        And, “unadulterated garbage” or no, I meant my follow-on comment also.   Multi-threaded solutions have a long reputation for being unreliable, and it always comes down to:   a timing problem.   Something that “appeared to work okay” on the developer’s machine that fell apart when subjected to real-world conditions.   The very same edge-cases that were discussed earlier in this thread.   “Signals,” which are by definition a form of rendezvous that occurs at a particular arbitrary instant in time (and which are handled asynchronously) ... instability which occasionally but not consistently shows up even to the developer ... attempts to “fix the problem” with pauses ... and at the end of the day a design problem that simply will not go-away like that.   Signal-based designs are tightly coupled, while resilient and scalable designs are loosely coupled.   You see evidence of that in all kinds of workload-management software ... JBoss, for instance.   Stuff that has nothing to do with Perl.   Any timing-based assumption is a kiss o’s death.

        If you want to see how a reliable multi-threaded system works, just go order a burger at McDonald’s and watch how their system works.   It is all implemented by human beings and connected by queues.   Machines signal humans, but humans don’t signal each other.   It works just as well, and it works the same way, whether the place is empty or full.   If something breaks down, it automatically recovers.   The computer systems that they use today are simply an implementation of the workflow management system that first attracted the attention of Ray Kroc to what the McDonald brothers were doing.   The thing to observe is ... their process, itself.

        If your design is running anywhere close to being affected by the “edge cases” that differentiate Linux from Windows, then IMHO you need to change that design.   To get back well away from that edge.   To use something other than bare signals ... to build a different design that does not require these things.   A design with no need to care.

        I will respond as best I can, try to be as helpful as I may be to the original poster and to the thread, and yes, I will give credit where credit properly is due.   I said what I meant, and I meant what I said.   And somehow, despite all that, accumulate 12,530 experience points and counting.   (And, yeah, I know that’s small change by comparison, but to me this is not a competition.   It’s just trying to help other Monks.)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://1022364]
Approved by Corion
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (10)
As of 2014-09-03 08:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (35 votes), past polls