Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
Esteemed monks,

I have a relatively large data structure made up from interconnected object "nodes" that represent all the physical devices in a network spanning many sites of various sizes.

These nodes are updated all the time by daemons collecting information such as ping status, response times, SNMP interface data, syslog data as well as operator data that chances the node structure by adding, moving and removing devices and connections.

The data structure allows spanning tree loops (because the physical network does) and allows me to do complex searches like determining the overall status of groups of sites, finding common root sources etc. very fast.

At the end of the pipeline sits an HTML/SVG engine that uses HTTP::Server::Simple::CGI to present everything as a web site with full drag/drop support, AJAX menus and the works. A typical request takes only a few milliseconds to serve.

So, what's the problem? As long as the requests are few and queries simple this is good enough but the current solution does not scale. At all. The problem is that my web server process is single threaded, because I can not for the life of me figure out a GOOD way to make it multithreaded:

Reason 1) DBI/Mysql does not work well with multithreading so live updates would be tricky. The current design is to use triggers on the large tables that get updated a lot, these triggers leave "clues" in a dedicated table, telling the server process which records have changed and need to be refreshed. This means updates get picked up without having to search for them but it's not ideal. Some data, like syslog messages, is simply too large to keep in memory and must be fetched from the database every time it's needed.

Reason 2) Copying the entire data structure very slow and difficult, because it's very deep and very recursive. Certainly too slow to do for each and every HTTP request. Besides, any changes made to the structure of a copy would have to somehow find its way back to the master and get distributed to the other copies.

I imagine there must be a way to have like a dozen copies of the same data structure running in its own process. Then, if an update comes in, somehow replicate that change to all of the others. If the HTTP bit scaled well enough, the ping/snmp/etc daemons could be rewritten to post their updates using HTTP and I could do away with the whole insane trigger/polling system.

Has anyone done something like this before? What tools or techniques did you use? Some kind of n-way realtime data replication, serializing stuff and sending it via sockets or whatever?

This is a pet system I develop in my spare time and rely heavily on at work. The current system gets the job done so it doesn't matter if I have to spend a year rewriting version 2 from scratch to get it perfect.

Update: DBI, not CGI stupid.

-- Time flies when you don't know what you're doing

In reply to Multithreaded (or similar) access to a complex data structure by FloydATC

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others making s'mores by the fire in the courtyard of the Monastery: (5)
    As of 2014-11-28 01:56 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      My preferred Perl binaries come from:














      Results (191 votes), past polls