Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
(And that's just with the current data set, which is expected to expand tenfold over the next few years)

And therein lies the root of your problem. It's not just your implementation that doesn't scale, but your design.

You are running a very complex simulation representing many discrete, real-world entities that communicate asynchronously. And you are running that simulation on single processor that has to iterate its way over a single, huge God-object in order to simulate all those asynchronous communications and concurrent state changes.

If you attempt to then further complicate that by having multiple threads concurrently iterating their way through that God-object deriving their own particular views of it, they will need to apply locks either universally or to discrete chunks of the data structure in order to derive a consistent 'snap-shot in time' view of the entire entity.

Each new viewer will therefore be impinging and impeding every other viewer, and as the number of viewers grows, so the time spent waiting for locks will grow exponentially. And as the size of the network grows, so the time required to perform a single traversal -- whether to update the status of the nodes as the effects of the state changes of individual nodes ripple through to their neighbours, and then to their neighbours, and then to theirs; or in order to gather a particular view of the instantaneous state of the network for a particular viewer -- grows exponentially as well.

Sorry to be the bearer of bad tiding, given that you've almost certainly expended considerable effort on achieving what you have already, but your design is unsustainable. Adding further complexity to it -- in the form of concurrent viewers with all the additional lock contention and the potential for cascading deadlocking they represent -- will only further hamper your ability to run the simulation in anything like real time.

The only realistic way forward is to:

  1. Distribute your network representation across multiple processors.

    Ie. break your God-object into manageable chunks each maintained by a different processor.

  2. Distribute the view building.

    By divorcing the presentation logic from the state maintenance.

    I encourage you to think about how Google et al. manage to respond to millions of viewers to their simulations of the entire web in real time. They distribute the state of their model (and the maintenance of that state) across a huge farm of processors. And then service view requests by having front-end processors request the relevant state from that back-office farm before gathering it all together to build their particular view.

That is a bitter pill to deliver, and far worse to swallow, but I seriously believe it is your only way forward. Sorry. I wish you the very best of luck.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re^3: Multithreaded (or similar) access to a complex data structure by BrowserUk
in thread Multithreaded (or similar) access to a complex data structure by FloydATC

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others musing on the Monastery: (4)
    As of 2018-08-14 19:55 GMT
    Find Nodes?
      Voting Booth?
      Asked to put a square peg in a round hole, I would:

      Results (155 votes). Check out past polls.