Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

threads->create hangs

by duane_ellis2 (Initiate)
on Aug 14, 2015 at 23:40 UTC ( #1138655=perlquestion: print w/replies, xml ) Need Help??

duane_ellis2 has asked for the wisdom of the Perl Monks concerning the following question:

I have a problem on win32 (64bit win 7, perl v5.20.2, active state), where threads->create() hangs.

I am looking for debug pointers.

The code launches two processes: Process (1) a background process, and process(2) is GDB, the gdb target talks to the background process via a socket, I am creating an automated sw test environment, I need to control, interact and capture output (stdout) from the processes.

I'm using IPC::Open3() to create both processes processes, like this:

# i,o,e are std in/out/err $p = open3( $i, $o, $e, @cmdline ); $hr->{'alive'} = 1; $hr->{'pid'} = $p; $hr->{'stdout'}= $o; $hr->{'stdin'} = $i; $hr->{'stderr'} = $e;
To process the stdout, I create a reader thread like this:
# The reader is given the hash reference, the 'o' # indicates this is the 'stdout' reader $hr->{'o_thread'} = threads->create( \&reader_thread, $hr, 'o' );
The above works quite well when I run one sub process. It locks up (the create call does not return) if I have two sub processes.

My only means to Debug is via print() - tells me that the call to threads->create() does not return, and print statement I put before threads->create() occurs, the after does not. And the print statement I put at the entry to the reader_thread never happens,

To be clear, this works a few times then does not work again.

Any suggestions how to dig deeper and figure out what I am doing wrong. Often, I insert print statements into various PM modules, but - threads are not perl, they are native code.

The reader thread is quite simple:

sub reader_thread { my ($hr,$who) = @_; my $c; my $h; my $r; my $q; threads->detach(); # determine handle. if( $who eq 'o' ){ $h = $hr->{'stdout'}; } else { $h = $hr->{'stderr'}; } # Get the output queue $q = $hr->{'Q'}; while( 1 ){ if( $hr->{'alive'} == 0 ){ last; } # Try to read *ONE* byte $r = read( $h, $c, 1 ); if( $r > 0 ){ if( length($c) ){ $q->enqueue($c); } } else { threads->yield(); } } }

Replies are listed 'Best First'.
Re: threads->create hangs
by BrowserUk (Pope) on Aug 15, 2015 at 00:33 UTC
Re: threads->create hangs
by ikegami (Pope) on Aug 17, 2015 at 01:48 UTC

    Did you do anything to prevent read from blocking?

    You should get rid of that yield. If you need it, you're doing something really, really wrong.

Re: threads->create hangs
by (anonymized user) (Curate) on Aug 16, 2015 at 00:12 UTC
    The concept of threads is to control concurrent use of the same code. If the two processes are intended to execute different code concurrently, you probably want to use fork instead, e.g.:
    my $pid = fork; if (!$pid) { # execute background code exit; } else { # execute concurrent foreground code waitpid $pid, 0; } # execute non-concurrent foreground code

    One world, one people

      The concept of threads is to control concurrent use of the same code. If the two processes are intended to execute different code concurrently, you probably want to use fork instead

      No, the difference between threads and forking is that threads share the same memory while forked processes do not. Forked processes often run the same code and threads often run different code (though they can't run different programs, of course).

      The practical differences between sharing memory and not is that threads will usually have to use locks to prevent conflicting accesses to the same bits of memory but can also just store data in memory to later be used directly by another thread. While processes need to use IPC to send data back and forth and very often don't need to use locks. Note also that getting locking right can actually be a significant technical challenge when using threads, especially as the software gets older and more complex (so using separate processes is often considered "safer").

      The original poster noted "I create a reader thread". Using fork to create "a reader" doesn't actually help because the new "reader process" would have to send the read data back to the parent via IPC which means the parent would have to read data from the reader process, which likely poses the same problem that prompted creating a new thread in the first place.

      Though, you could use non-blocking read or, to be even more efficient, 4-argument select to avoid creating a reader thread. Though that certainly adds some complexity as well, especially in Perl. And with a thread you can let Perl mostly deal with the locks for you so that is likely the simplest approach to code, anyway.

      - tye        

      A reply falls below the community's threshold of quality. You may see it by logging in.
Re: threads->create hangs
by (anonymized user) (Curate) on Aug 16, 2015 at 20:44 UTC
    So, the story so far is that some responders avoid the question of the warning against using threads in threads. Conversely, an anonymous poster claims the warning can simply be removed. My position is that the correct use of Perl threads and forks is best derived from the pre-existing theory (described surprisingly well on Wikipedia). I have two critics Tye and browserUK who on the one hand make no comment about this warning but are clearly knowledgable about the Perl implementation.

    My challenge to anyone who wants to participate in the public interest is to engage with me in a competition to propose alternative and accurate documentation to threads that enables the reader to have a clear understanding of when to use threads versus forks. I don't set a time limit and I am starting my own efforts forthwith. Submissions, when ready should be linked in a reply to this topic.

    One world, one people

      So, the story so far is that some responders avoid the question of the warning against using threads in threads.

      I asked p5p about that not long ago, and they very clearly said it's not a warning against using threads. It's a warning that threads aren't lightweight. If you're prepared to accept that, then use of threads is perfectly fine.

        Excellent, but then don't you think the documentation should more clearly present that view? If you had to ask, then it didn't do that for you originally, did it?

        One world, one people

      You can read a lot about the motivations for the creation of that warning (certainly in the p5p archive but also other places). It has more to do with many people having a difficult time using threads than with any specific problems with Perl's implementations of threads. Well, except for the heavy per-thread memory cost that is specific to Perl's implementation (which is the part you misread as saying that Perl threads don't use OS threads).

      It is meant to discourage casual use of threads and has more social motivations than technical. Have fun fighting with p5p on what to replace it with. I really do wish you luck (sincerely), but I am not interested in participating in that. I try to stick to activities where my strengths and weaknesses are more beneficial. The warning should certainly be made less likely to mislead.

      - tye        

        tye, many thanks for your sincere and informative response.

        One world, one people

      So, the story so far is that some responders avoid the question of the warning against using threads in threads. Conversely, an anonymous poster claims the warning can simply be removed. My position is

      Your position is irrelevant as its not based on information/knowledge but propaganda

      I don't read it as a warning to not use threads. I read it as a warning that threads are not light-weight and people writing code using them are not likely to get much help.

      Through my own experience in writing threaded programs, I find the warning to be absolutely valid, although I suspect the number of people becoming proficient at using threads has grown somewhat since the warning was first written.

        Fair enough. Perhaps I am setting too high an expectation on the docs. The systems I worked on most recently have too trivial a usage of threads to be pushing the boundaries. So perhaps I should try creating a test set (destruction-testing ;)) for all the methods in the threads space - that might reveal whether there are specific problems that could be documented away - in my world there is such thing as a documentation problem i.e. what appears to be a bug isn't so long as it's documented properly - but for now I can't get past the fact that although some others clearly see it differently, I stare at those words and it's as if it is saying to me: 'abandon hope all ye who enter here!' - I can't remember anything else in perldoc that reads back at me anything even close to that.

        One world, one people

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1138655]
Front-paged by stevieb
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (4)
As of 2020-10-20 03:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My favourite web site is:












    Results (208 votes). Check out past polls.

    Notices?