Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re^9: Threads Printing Issue - Output Mangled / Term Crashing

by salva (Abbot)
on Apr 15, 2014 at 06:56 UTC ( #1082314=note: print w/replies, xml ) Need Help??

in reply to Re^8: Threads Printing Issue - Output Mangled / Term Crashing
in thread Threads Printing Issue - Output Mangled / Term Crashing

strace may show something interesting. For instance:
strace -o /tmp/out -f perl -e 'print `ssh localhost -tt -l root tcpdum +p -c10 -nntttt`'

On my computer that works pretty fine, and I don't see any tty access. Maybe just updating the ssh client could solve your problem.

update: Ah, I can see what's happening:

ssh -tt ... reads the terminal flags from STDIN (ioctl(0, TCGETS, ...)), sets new ones (ioctl(0, TCSETSW,, ...)) and runs the remote command. Then, upon exit, it restores the original flags.

The issue is that with multiple ssh processes running in parallel, some of them may read the already modified flags and so later, reset STDIN to an incorrect state.

An easy workaround is to wrap the part of the code where the threads are started and then joined with an extra couple of TCGETS/TCSETSW ioctl calls. Another option is to just redirect STDIN from /dev/null.

update 2: The following program works correctly on my computer:

use strict; use warnings; use threads; use threads::shared; my @threads; my @servers = (('localhost') x 100); my $lock:shared; foreach my $server (@servers) { chomp $server; push (@threads, threads->create (\&dumpServer, $server)); } foreach (@threads) { $_->join(); } sub dumpServer { my $server = shift; my $net = `ssh -l root -tt $server '/usr/sbin/tcpdump -c10 -nntttt 2 +>&1' </dev/null`; lock($lock); print "TEST - $server\n$net\n\n"; }

Replies are listed 'Best First'.
Re^10: Threads Printing Issue - Output Mangled / Term Crashing
by bigbot (Beadle) on Apr 16, 2014 at 08:01 UTC
    Thank you, awesome find! This will save me from doing another 8 hours of troubleshooting. :)
Re^10: Threads Printing Issue - Output Mangled / Term Crashing
by bigbot (Beadle) on Apr 18, 2014 at 01:02 UTC

    Well, your solution worked :) At least as far as fixing the term issue. However, a side effect is that the processes continue running on the servers after the SSH connection is killed (arrrrgh). So sorta back to square one.

    I have had some luck with the following line, but it feels like a cheesy way (and not 100%) to fix the issue. Before the print and in various places:

    system('stty sane');

    So confirmed that it's definitely a term settings issue. Over 95% of the time there are no print issues. However, now trying to pipe the output into any other tool crashes the program (leaving tons of threads running and dormant processes). Other users will certainly pipe the output so that's a problem. Maybe I need to pass Perl code to the servers like you were saying (to kill the processes once SSH is killed). This way I don't need to use that buggy -tt SSH option.

      try using /dev/zero instead of /dev/null.

        I was apparently screwing something up, because this worked perfectly:

        2>/dev/null" </dev/null`

        Everything works awesome now. No term issues whatsoever, and no zombie processes! :) Thanks for the excellent help!!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1082314]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (4)
As of 2018-04-20 22:56 GMT
Find Nodes?
    Voting Booth?