Socket client thread design problem

[photron]

I'm currently porting some C code to Perl and ran into problems in the socket and thread handling design. I want to ask for some guidance.

First I ran into problems with the differences in the threading model of C and Perl. The existing C code uses a socket and shares it between multiple threads. This doesn't work directly in Perl. Due to the way Perl clones variables on thread creation a threads->create call blocks on Windows if another thread is doing a blocking receive on the socket. So I need to avoid this situation by ensuring that the receive thread is not running while other threads are created. This was tricky but I managed to get it working.

But there are still other problems due to the way the C code works. I spent some days on this problem and now think that it's impossible to implement the behavior of the C code in Perl.

The behavior of the C code is as follows: The user can send request to a server and receive responses. The server can also send spontaneous responses that are call callbacks.

The API has a method to establish the socket connection and create a receive and a callback thread. The receive thread does a blocking recv() call on the socket and handles the incoming data. If the incoming data is a response then it is passed back to user via a queue. If it's a callback then it is handed over to the callback thread that will call a user-provide subroutine to handle it. So the socket is shared between three threads.

Then there is auto-reconnect handling. If the socket gets disconnected from the server then this is detected by the receive thread which will then tell the callback thread to try to reconnect. And this is the problem: Once the connection gets established again, only the callback thread has the correct socket. The receive thread and the main thread of the program that's running the user code have old sockets.

I tried to share the socket's fileno between the threads and fdopen sockets based on the fileno. But this gets hairy quickly and I still ran into threads->create blocking in Windows.

So any advise on realizing this behavior without running into all this trouble created by the differences in the threading model of C and Perl?

[BrowserUk]

So any advise on realizing this behavior without running into all this trouble created by the differences in the threading model of C and Perl?

Advise: stop trying to emulate the C code. There are many programming techniques used in C that are either suboptimal or impossible to realise in Perl. There is always a better Perlish solution.

But you have to step back from the C implementation and look at the required functionality, not how that functionality is achieved in C.

If you would describe the program at a higher level I might be able to suggest something. I'm really intrigued by the idea of a C program running user provided callback code.

(From the description so far, it sounds like your trying to implement a multi-server IRC client with Bot logic; but that's just a guess.)

[photron]

Not IRC related at all. It's hardware control over TCP/IP. Here's a example using the C code. There is an IPConnection object that handles all the TCP/IP stuff. Then there is a Temperature object that represents a temperature sensor. To interact with the temperature sensor the example creates an IPConnection object and a Temperature. The Temperature object will then use the IPConnection for its communication needs.

Getter Functions

Now to read the temperature of the sensor you just call temperature_get_temperature. This internally sends a request to the hardware asking for the current temperature. Then the sensor sends a response back containing the current temperature. During this message exchange the temperature_get_temperature blocks while waiting for a response. There is also a timeout mechanism. If the sensor doesn't respond within 2.5 sec then temperature_get_temperature will report an error.

#include <stdio.h>

#include "ip_connection.h"
#include "bricklet_temperature.h"

#define HOST "localhost"
#define PORT 4223
#define UID "XYZ" // Change to your UID

int main() {
    // Create IP connection
    IPConnection ipcon;
    ipcon_create(&ipcon);

    // Create device object
    Temperature t;
    temperature_create(&t, UID, &ipcon); 

    // Connect to brickd
    if(ipcon_connect(&ipcon, HOST, PORT) < 0) {
        fprintf(stderr, "Could not connect\n");
        exit(1);
    }
    // Don't use device before ipcon is connected

    // Get current temperature (unit is °C/100)
    int16_t temperature;
    if(temperature_get_temperature(&t, &temperature) < 0) {
        fprintf(stderr, "Could not get value, probably timeout\n");
        exit(1);
    }

    printf("Temperature: %f °C\n", temperature/100.0);

    printf("Press key to exit\n");
    getchar();
    ipcon_destroy(&ipcon); // Calls ipcon_disconnect internally
}

Just this getter mechanism alone would work without any threads, you only have to receive data during a getter call. If no getter is called no data will arrive from the hardware.

There seems to be a post size limit, so to be continued ...

[photron]

Continuation ...

Callbacks

But there is more. You can tell the sensor to inform you about temperature changes without you having to call the get function over and over again. For this the callback function cb_temperature is registered using temperature_register_callback. It will automatically be called if a temperature callback from the temperature sensor is received.

#include <stdio.h>

#include "ip_connection.h"
#include "bricklet_temperature.h"

#define HOST "localhost"
#define PORT 4223
#define UID "XYZ" // Change to your UID

// Callback function for temperature callback
// (parameter has unit °C/100)
void cb_temperature(int16_t temperature, void *user_data) {
    (void)user_data; // avoid unused parameter warning

    printf("Temperature: %f °C.\n", temperature/100.0);
}

int main() {
    // Create IP connection
    IPConnection ipcon;
    ipcon_create(&ipcon);

    // Create device object
    Temperature t;
    temperature_create(&t, UID, &ipcon); 

    // Connect to brickd
    if(ipcon_connect(&ipcon, HOST, PORT) < 0) {
        fprintf(stderr, "Could not connect\n");
        exit(1);
    }
    // Don't use device before ipcon is connected

    // Set Period for temperature callback to 1s (1000ms)
    // Note: The callback is only called every second if the 
    //       temperature has changed since the last call!
    temperature_set_temperature_callback_period(&t, 1000);

    // Register temperature callback to function cb_temperature
    temperature_register_callback(&t,
                                  TEMPERATURE_CALLBACK_TEMPERATURE, 
                                  (void *)cb_temperature,
                                  NULL);

    printf("Press key to exit\n");
    getchar();
    ipcon_destroy(&ipcon); // Calls ipcon_disconnect internally
}

Now someone has to receive incoming data from the socket even if no getter call is currently in progress. In the C code there is the receive thread of the IPConnection that continuously handles incoming data. If a response for a getter call arrives then the receive thread check if any device object such as the Temperature object is currently wating for a response. If yes then the response is passed to the waiting getter call via a queue. If a callback is received, then it is passed over to a second thread via another queue, the callback thread of the IPConnection.

The callback thread takes a callback packet from its queue and checks if any of the device objects has a callback function registered for it. If yes then it calls it with the data from the callback packet. The callbacks cannot be called from the receive thread because then the user provided getter function would not be able to call another getter function of the device. As the receive thread is currently blocked by the callback call it cannot receive the incoming response for the getter call.

But there is even more. If a disconnect of the socket is detected then the receive thread is stopped and the callback thread is told to reconnect. Once the connection is established again the receive thread start receiving incoming data again.

There seems to be post size limit, so to be continued ...

[photron]

Continuation ...

Summary

So what the C code does is allow to call getter functions and setter functions (just like getters but without a response) to talk to hardware using a TCP/IP connection. The getters can timeout. You can also register for callbacks to be notified by the hardware without having to poll for it. Finally the IPConnection automatically takes care of reestablishing the TCP/IP connection if it gets lost.

The C code does all this by using two threads and sharing a socket between the receive thread the callback thread and the main thread of the program. Obviously this doesn't work 1:1 in Perl due to the different threading model. And I ran into some trouble there already. But the goal is to get the same features in Perl.

For comparison here is the current temperature getter example ( https://raw.githubusercontent.com/Tinkerforge/temperature-bricklet/master/software/examples/perl/example_simple.pl ) and the temperature callback example ( https://raw.githubusercontent.com/Tinkerforge/temperature-bricklet/master/software/examples/perl/example_callback.pl ) in Perl.

Here's the full C code ( http://download.tinkerforge.com/bindings/c/tinkerforge_c_bindings_2_0_13.zip ) and the current Perl code ( http://download.tinkerforge.com/bindings/perl/tinkerforge_perl_bindings_2_0_1.zip ). Note that I'm currently working one Perl code and that the provided version has a bit of code duplication it it that makes the internal logic a bit hard to follow.

Edit: Okay, I cannot use a tags here, contradictory the documentation. I thought the post would have been to long and splitted it into three posts.

[BrowserUk]

I downloaded the C zip you linked only to find it only contains the same trivial, single-threaded examples you already posted (amongst lots of other unrelated examples). But no C code that uses 3 threads and shared sockets and queues and stuff to implement something you claim to be trying to emulate in Perl.

Nothing for me to use as a basis for producing a threaded Perl code to meet the sketchy and confused specs you outline in words.

So, then I start to think about the examples you have posted, and it is my conclusion that you are trying to use threads to mix together two different APIs either of which can serve the purpose on its own.

• Either: your server queries the current value from the device on demand -- ie. when a client connects -- synchronously reads the response and returns it to the connecting client.

  No need for a callback here. Each client gets the latest value on demand.

• Or: your server registers a callback that, once per second or whatever frequency you choose, gets called back (if the temp changes), and updates a (global) cache variable with the latest temperature value.

  And whenever your clients connect, you simply return the current value of that cache variable.

  No threads -- unless you choose to use threads for your client connects -- no need or use for queues.

  Simplicity personified.

So, unless you can provide a reason for your 3 threads + queues design -- and the C code that implements it -- I can see no reason at all to try and create something that matches your earlier descriptions. Especially from scratch and without reference.

[zentara]

Try this: Simple threaded chat server.

I think your problem is that you don't differentiate a new connection, from an existing connection. The code shows how it's done, and those scripts are pretty close to the way C would do it.

You do the initial accept with the main thread, then let the threads handle the connection.

[photron]

Thanks for the hint. The example shows how to share sockets between threads based on their fileno. I figured this out already and got the Perl code working on Linux with this.

But on Windows I still ran into problems with this socket sharing and threads due to the problem of threads->create() blocking if a socket is on a locked state. I tried to workaround this and reached like the third layer of workaround code and it's getting worse each layer. That's why I've asked for general design advise.

[zentara]

But on Windows I still ran into problems with this socket sharing and threads due to the problem of threads->create() blocking if a socket is on a locked state

Maybe ioctl for non-blocking and blocking in Windows would be useful, or google for perl ioctl windows nonblocking.

[Anonymous Monk]

Also ... and as quickly as possible(!) ... step back and look at what this C program is designed to do, and whether an equivalent off-the-shelf bit of functionality already exists somewhere in the vast CPAN library. At the end of the day, you don't have to reinvent the wheel in Perl: you simply have to wind up with a functional equivalent of what that wheel does. And you'd like to do that with as little redundant effort as possible. C programmers had to do everything all-over-again from scratch. You do not.