Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

How I'm sure syswrite has finished?

by casiano (Pilgrim)
on Mar 14, 2009 at 11:58 UTC ( #750611=perlquestion: print w/ replies, xml ) Need Help??
casiano has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I have a race-condition problem.

The problem arises because a child process makes a call to

syswrite handler, message
and finishes. The message is huge and is received by another process running on a remote machine (via SSH). I thought closing the handler would be enough to sync but it does not seem to work. The fragment of code is as follows (I'm using Parallel::ForkManager but the problem arises also with plain fork)
# Parallelise the writing into the pipes my $pm = Parallel::ForkManager->new($lp); foreach (@str_handles) { my $pid = $pm->start and next; syswrite $_->{handle}, join(" ", $_->{chunksize}, $_->{A_cols}, @A_lines[$_->{start}.. $_->{end}], @B_lines, "\cN" ); $pm->finish; }
The problem disappears if I insert a sleep(1) after the syswrite call because the sleep gives time for the write to complete. Is there a way to know when the call to syswrite has fully finished?

Thanks for your help

Casiano

Comment on How I'm sure syswrite has finished?
Select or Download Code
Re: How I'm sure syswrite has finished?
by Perlbotics (Abbot) on Mar 14, 2009 at 12:13 UTC

    Hi. An explicite close might help...

    syswrite ... ; close $_->{handle} or die "cannot close file - $!"; # added $pm->finish;

      No, I already tried and gives the same error.

        What is the "same" error? It's kind of hard to troubleshoot un(der)specified problems.


        --
        "Language shapes the way we think, and determines what we can think about."
        -- B. L. Whorf
Re: How I'm sure syswrite has finished?
by Joost (Canon) on Mar 14, 2009 at 13:55 UTC
      The problem is that at the other side there is a process
Re: How I'm sure syswrite has finished?
by ChemBoy (Priest) on Mar 14, 2009 at 17:26 UTC

    If it were on a local machine I'd suggest waiting for the other process, but that's not really an option in this case (unless you can wait for the local process? I don't know how P::FM works in any detail, nor how you're set up, so this might be totally useless). Looking at the problem as a whole, rather than the specific solution you've already tried, you might want to change your remote process to send an acknowledgement of some kind back at you when it's done—it requires a little more coding and a little more network traffic, but it seems likely to me that it would work, which appears to be an advantage over the current approach.



    If God had meant us to fly, he would *never* have given us the railroads.
        --Michael Flanders

      Many thanks. Apologies for the late answer but I have been away of work a few hours.

      I tried already your suggestion of trying to produce an acknowledge from the other side (the other side is a process running in a remote machine via SSH. It was started using IPC::Open2) didn't work well. May be I did s.t. wrong. As several people mentioned is difficult to give advice without the whole context, but I can post the two programs (local and remote) if you like.

      Many thanks

      Casiano

Re: How I'm sure syswrite has finished?
by perl5ever (Pilgrim) on Mar 14, 2009 at 18:47 UTC
    I agree with ChemBox that the parent should wait for the children to finish. Have a look at the wait_all_children method.

    Your problem is likely happening after the for loop. Note that the parent process will get done with the for loop much earlier than the child processes. One way the messages sent by the children may get truncated is if the parent is doing something after the for loop to cause the ssh processes to terminate.

      [ This was meant as a reply to Re: How I'm sure syswrite has finished? ]

      if the parent is doing something after the for loop to cause the ssh processes to terminate.

      I think it's simpler than that. When the parent exits, doesn't that kill the children? Calling wait_all_children is still the solution as that would make sure the children are done.

        Many thanks ikegami,

        I tried wait_all_children (see answer below) but without success.

        Somehow the $pm->finish (I also tried a plain exit) with the same result aborts the whole writing?.

        I guess that somehow the call to syswrite produces a decoupled process in the remote machine than is in charge of fulfilling the writing (since it is a huge message). When the exit of the forked local child occurs the remote process is aborted?

      I tried this:
      153 foreach (@str_handles) { 154 my $pid = $pm->start and next; 155 156 $b = syswrite $_->{handle}, 157 join(" ", 158 $_->{chunksize}, 159 $_->{A_cols}, 160 @A_lines[$_->{start}.. $_->{end}], 161 @B_lines, 162 "\cN" 163 ); 164 # Wait until the writing has finished 165 #sleep(1); 166 $pm->finish; 167 } 168 $pm->wait_all_children;
      But the behavior is the same.

      Only the sleep seems to work.

      My guess is that

      $pm->finish;
      produces a "fake termination": the local process ends while the remote is still writing?

        I think you need to show us more of your program - especially the part wher eyou create the ssh connections.

        Also, the output of strace or truss would help.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://750611]
Approved by Perlbotics
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (5)
As of 2014-09-30 22:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (385 votes), past polls