Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

A server that has a fool as its client: itself

by Anonymous Monk
on Feb 10, 2011 at 18:06 UTC ( [id://887511]=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Greetings folks,

Over the years I have read on and off about how perl manages servers, non-blocking vs blocking, forking vs select, client-server, etc.

One thing that I'm interested in doing is writing a server that listens to a connection but then also acts as a client itself. It in turn would connect to the exact same server running somewhere else.

The purpose is to take information it learns as a client and in turn repeat it to as a server to its clients. Of course, each of these clients are also copies of the same server running elsewhere.

The data is nothing more than a table whose content would occasionally change - but changes must be transmitted and received and duplicates discarded - almost like a database, but purely in memory since it's never meant to be permanently kept.

As suspected, data would need to be shared between each process. We're not talking hundreds of megabytes of data, actually something on the order of a few hundred kilobytes with a little bit of churn (kilobytes).

I have the abstract algorithm down, but I'm not sure if I should use forking with IPC or threads. Anyone have some advice?

Thanks

Replies are listed 'Best First'.
Re: A server that has a fool as its client: itself
by ikegami (Patriarch) on Feb 10, 2011 at 19:22 UTC

    First, it's necessary to understand the data flow.

    The purpose is to take information it learns as a client and in turn repeat it to as a server to its clients.

    So the flow is unidirectional?

    sub run { for (;;) { my $msg = $connection_to_server->get(); $_->send($msg) for @connections_to_clients; } }

    To get parallelism, send simply needs to hand off the work to a thread of some sort (including a separate process).

    # Client class sub send { my ($self, $msg) = @_; $self->{work_queue}->enqueue($msg); } sub worker { my ($self) = @_; my $q = $self->{work_queue}; for (;;) { my $msg = $q->dequeue(); ... send $msg ... } }

    Whether you use Coro, threads, forks or something else seems very incidental.

    You could also use polling, but select loops tend to be complicated. The first I'd want to do is give the select loop the above enqueue/dequeue interface shown above. Might as well just use Coro to save the work.

      So the flow is unidirectional?

      Ah, it's bidirectional. I should have clarified, my bad

      Think of these daemons acting as beacons relaying information to one another. Eventually, all the participants in the network of daemons in different places will have the same information.

      Wow! thanks for that, it does give me a clue into doing this.

Re: A server that has a fool as its client: itself
by BrowserUk (Patriarch) on Feb 10, 2011 at 21:56 UTC

    To me it sounds like you should be using a broadcast mechanism.

    When any instance makes a change, it sends notification of that change to a broadcast group that all the other instances (and itself) are listening to.

    In that way, each instance only sends or receives to/from the broadcast group address, thereby avoiding the problems of runaway cascading notifications.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: A server that has a fool as its client: itself
by Corion (Patriarch) on Feb 10, 2011 at 18:23 UTC

    There is a third alternative to threads or fork - a callback-oriented approach using AnyEvent (and nonblocking sockets).

    Depending on the actual data to be distributed, I'd still look at the established methods of storing and transferring data, like SMTP ("email"), databases, HTTP or DNS before implementing my own scheme.

Re: A server that has a fool as its client: itself
by sundialsvc4 (Abbot) on Feb 10, 2011 at 18:16 UTC

    You can go a long way with a “simple poll() loop” architecture.   The app listens simultaneously to one-or-more sockets and responds appropriately when activity occurs on each socket ... but the activity is not actually “concurrent.”   Instead, the application is doing exactly one thing at a time, but in an unpredictable sequence, and it all works just fine because I/O, including network I/O, is much slower than the CPU.   (CPUs think in terms of nanoseconds...)   As long as each unit-of-work that is to be done requires only an insignificant amount of wall-time to achieve, this internally very-simple architecture works beautifully.   So, I would start with that design and fully expect to also finish with it.

Re: A server that has a fool as its client: itself
by jethro (Monsignor) on Feb 10, 2011 at 20:42 UTC

    Could it be that you confused 'client' and 'server' a few times. At least if you are talking about them in the network sense. If I get you correctly, the combo-client-server would wait for a change to happen locally, then contact (which only can be done as client) to some sibling of itself somewhere else (who listens as a server) and delivers the update to it

    The most important question then is how to stop every server/client from trying to inform every other server/client and so generate traffic that grows quadratic with the number of servers.

      Right. We wouldn't want the data to be replicated ad-nauseum until all resources are exhausted

      At start-up, there would be no data. When the daemon issues a connect to another daemon, only then would the whole shebang be received once, and there after, the daemons in question would send only changes to the data.

      I have also figured out how I would get each daemon to determine who connected to who first: using a configuration file listing IP addresses, I would have the daemon with the highest IP address be the listener, and treat the lower-addressed ones as the client. So if the running daemon sees it has a higher IP address than the rest, it will listen only. If it's somewhere in between within a range of listed addresses, it will listen for the lower addressed clients, but connect to the others since they are daemons.

      Given that, I should have the daemons initiate the data push to the clients upon they connecting.

      Whew!

      Of course, I still need to look at everyone's suggested options.

        Depending on how much control you have over the configuration, and how much "failover" you are willing to sustain, looking into the "peer-to-peer" mechanisms employed by git or even bittorrent (as far as there actually is documentation) might be educative.

        The alternative is to have either a designated master and a tree hierarchy (and a single point of failure), possibly together with an "election scheme" to elect a new master if the current master goes offline. This is unfortunately also not trivial (see "irc netsplit" and bugs in the NT Domain Controller), but maybe you can provide enough configuration information (like IP addresses) where the master is determined by configuration. For example if you can assume that there is no malicious participant, the current master can yield to any master with a lower IP address (or whatever criteria).

Re: A server that has a fool as its client: itself
by mojotoad (Monsignor) on Feb 11, 2011 at 10:00 UTC
    I think you misspelled 'distributed adaptive routing'.

      Great term. Shows up a lot in theoretical papers. Not so prevalent in the sphere of real world perlish implementations though.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://887511]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2024-03-29 06:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found