Keeping an INET socket inside a shared object

[Superfox il Volpone]

Hello there,
I am trying to create a shared object as wrapper to NET sockets. The idea would be that whatever thread access it, it would be able to read/write in the socket. The problem is that sockets cannot be saved into shared objects. Searching on internet, I found that several people relied on the solution of passing file descriptors, which I am not even sure to have understood it properly ( are not file descriptors meaningful only locally to a process? ). I`ve come up with this code:

package SharedSocket;

use strict;
use warnings;
use threads;
use threads::shared;

use IO::Socket::INET;
use Storable;

sub new(){
    my $class = shift;
    die("Not an instance method") if ref($class);

    
    my $self = &share(bless({},$class));
    $self->{"closed"} = 0;
    
    my $socket = new IO::Socket::INET(@_) or die("SharedSocket $!");
    $socket->autoflush(1);
    
    $self->{"closed"} = 0;

    $self->{"socket_fd"} = $socket->fileno;
    print("[SharedSocket::ctor] fd: " . $self->{"socket_fd"} . "\n");
    
    return $self;
}


sub send{
    my ($self, $msg) = @_;
    die("Not a static method") if ref(!$self);
    
    print("[SharedSocket::send] fd: " . $self->{"socket_fd"} . "\n");
    open(my $socket, ">&=", $self->{"socket_fd"}) or die("SharedSocket
+ $!");
    Storable::fd_nstore($msg, $socket);
}


sub receive{
    my ($self) = @_;
    die("Not a static method") if ref(!$self);
     

    print("[SharedSocket::receive] fd: " . $self->{"socket_fd"} . "\n"
+);
    open(my $socket, "<&=", $self->{"socket_fd"}) or die("SharedSocket
+ $!");
    my $msg = Storable::fd_retrieve($socket);
    
    return $msg;
}

sub close{
    my ($self) = @_;
    die("Not a static method") if ref(!$self);
    
    lock($self);
    return if($self->{"closed"});
    
    open(my $socket, "+<&=", $self->{"socket_fd"}) or die("SharedSocke
+t $!");
    $socket->close(); 
    $self->{"closed"} = 1;
}

obtaining a bad descriptor error:

[SharedSocket::ctor] fd: 4
[SharedSocket::send] fd: 4
SharedSocket Bad file descriptor at SharedSocket.pm line xx.

Besides passing the internal handle, I guess I need to copy the socket structure somehow?

thanks in advance
kind regards,
s.fox

[BrowserUk]

I haven't found any good mechanism for sharing IO::Socket::INET objects between threads.

(I'm the guy that 'discovered' the fileno trick for sharing file handles; so I know a little about this. :)

And, FTR, I am of the opinion that designing architectures that requires or uses shared objects is fundamentally flawed.

This is doubly true if those shared objects encapsulate process-global entities like sockets, filehandles, directory handles etc.

Think about this:

1. if 2 or 3 threads have access to a shared tcp socket; which one listens?
2. And if one thread wants to send something, it can't do so if another thread is reading -- waiting for input.

Whilst sockets are bi-directional, they are serially bi-directional, which is to say they can only be communicating in one direction at any given time.

With a single thread handling a socket, it is easy to manage that it send something, then reads the reply, sends something, then reads the reply. (Or waits for input then responds...)

But coordinating between multiple threads becomes fraught with opportunities for deadlocks and infinite blocks.

In short, a shared object architecture makes no sense when dealing with sockets.

---

The correct alternatives are:

• have a thread that manages the socket and other threads send requests to/receive responses from that thread. This assumes the application is a multi-threaded client talking to a remote tcp server.
• If the application is a multi-threaded server, then the main thread monitors the listening thread and each new client gets it own dedicated thread.

[McA]

Hi s.fox,

I have to come up with the mostly undesireable meta question: What do you want to achieve sharing a socket between threads?

I'm asking because IO in general and IO over inet sockets is assumed to be slow. The complexity of threads is often introduced to have a solution for the blocking character of IO. So, I don't know what your use case is, but I would propose having one thread performing the whole IO for all other working/computing threads. In this case the one thread would try to read from the socket and put the result into a (dispatching) queue which gets read by the working threads and vice versa.

Even if you share the socket/fd between the threads you have to be very very careful in synchronizing the IO requests.

Have you some more isolated code to show your problem/use case?

Best regards
McA

[zentara]

I had to log in just to upvote this discussion. :-)

Just to throw in my 2 cents, it seems all the rage now is non-blocking I/O event loop systems, such as node.js, or that matter, any eventloop system. You don't wait on socket reads, ever. You check to see if something is there, with a proper sysread, and if not, move on to the next event in the loop.

From my prelimary studies of the new techniques, it seems that upon socket connection, the server issues a promise to deliver the requested data, and the eventloop reads it chunk by chunk as it comes in. Then, there are various ways to detect if a socket has gone stale, or broken.

But I would advise using something like AnyEvent or Glib, which are 2 well documented and used eventloop systems.

You might find this interesting. It shows how to use a couple of GUI based event loop systems. Simple threaded chat server

[Superfox il Volpone]

Hi there,
Thank you for replies.

oh, I did not expect such situation when I started to code my project. So basically I have a bunch of workers that should offload the work on different machines, a master that schedule the work to the remote workers, and a set of users that send work to be done and query the current status. The communication is intrinsically non deterministic, because workers can report their status, or the raise of issues, to the master at different times, while the master can demand different services to workers.

In the master, I thought to set a different handler to each worker or user. As the handlers should cooperate each other and keep a consistent view in every moment, I did immediately model them as shared objects. Eventually, since each handler should "own" its connection socket, it came up pretty natural the willing to associate the socket with its handler. But the object is shared for its own nature, while the socket is not, therefore my struggle.

The second motivation is, I imagine, conceptual. Because if TCP is sequentially bidirectional (that`s new to me), it is conceptually bidirectional, i.e. there are two different untied streams to read/write, which allow two different flows to take place...

I did a step ahead, being able to share a socket among certain threads. I guess there was a problem of scope, the socket was destroyed in the ctor itself. This is my updated code:

our %SharedSockets; # keep the created sockets to survive

sub new(){
    my $class = shift;
    die("Not an instance method") if ref($class);

    
    my $self = &share(bless({},$class));
    $self->{"closed"} = 0;
    
    my $socket = new IO::Socket::INET(@_) or die("SharedSocket $!");
    $socket->autoflush(1);

    
    $self->{"closed"} = 0;
    $self->{"socket_fd"} = $socket->fileno;
#   push(@SharedSockets, $socket);
    $SharedSockets{$socket->fileno()} = $socket;
    
#   print("[SharedSocket::ctor] fd: " . $self->{"socket_fd"} . "\n");
    
    return $self;
}

I understand that Perl wants to emulate the threads as heavy processes. When we create a thread, I guess it inherits all the file handles of the creator. If we would create a socket in a child, then neither the father nor its brothers would be aware, wouldn`t be? I wonder what were the initial motivations to introduce a such exotic thread model at the first place.

Again, I am getting the feeling I am trying to bend the language for something it was not supposed to perform... :oO

Kind regards,
s.fox

[BrowserUk]

Again, I am getting the feeling I am trying to bend the language for something it was not supposed to perform... :oO

No. You are simply trying to achieve your goal using techniques more suited to other languages.

Eg. In C, it is quite normal to process files one character at a time using something like this:

while ((c = getchar()) != EOF){
    ...
}

That is quite efficient in C; but doing the same thing in Perl (using getc()) is horribly inefficient; so you never see it done that way.

You appear to be wanting to use javaesque programming techniques in Perl; and then complain because they don't work the same way.

If you work with Perl and iThreads they can be very simple, efficient and effective. Work against them and you'll go away complaining that the tool is faulty ....

[oiskuu]

Have you tried turning off the CLOEXEC flag?

  use Fcntl;
  ...
  my $socket = new IO::Socket::INET(@_) or die;
  $socket->fcntl(F_SETFD, 0) or die; # clear CLOEXEC
  ...
#  $socket->fcntl(F_SETFD, FD_CLOEXEC); # set CLOEXEC

It's also probably a good idea to ->shutdown the sockets once you're done, especially if they may "leak" to some other thread or process.

ps. I've not the foggiest idea what serially bi-directional could mean. AFAIK streams are full-duplex, with separate receive and send queues, and usually have an out-of-band (OOB) mechanism too (e.g. telnet uses this to signal interrupt).

[BrowserUk]

I've not the foggiest idea what serially bi-directional could mean. AFAIK ...

Try it.

1. Connect a socket to a server that attaches, and then tries to read from the client.
2. Start a thread (in the client) that goes in to a read state on that handle.
3. Then try to write something to the server via that handle from the main thread.

The write will block indefinitely pending the completion of the read in the thread, which will never occur because the server won't send anything.

[oiskuu]

That's the blocking mechanism, and it's not limited to just sockets either. Multiple writes may block each other similarily. Set your fd nonblocking if this is a problem...

[andal]

I suspect, there's some confusion here. Within "threads" connections can be shared without problem. Well, there are issues with synchronization of access (and that can be very complex) but still, the IO::Socket::INET object is directly accessible. The same is true for forked processes if the fork has happened AFTER connection acceptance/creation.

In all other cases, connection information has to be passed from one process to another. I don't know about Windows, but in Unix this is done via IO::Socket::UNIX socket. As already mentioned, this involves passing of file descriptor only (fileno). Basically, when kernel gets request to pass file descriptor to the peer, it creates for the destination process new file descriptor that would point to kernel structures to which old file descriptor was pointing. The peer process just needs to wrap file descriptor into IO::Socket::INET structure using IO::Socket::INET->new_from_fd($fd, 'r').

Of course, simple writing of information into file and then reading it in another process won't do any good. Remember, it is kernel that takes file descriptor number and maps it to real connection data, and such mapping is process specific. So file descriptor 4 in one process has nothing in common with file descriptor 4 in another process. One has to ask kernel to allow for process X to have file descriptor that would point to the same connection data as in the current process.