Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

I've spent a couple of days thinking about this and playing with alternative solutions, but nothing comes close to being a fast as what you're doing now.

The problem of course, is that what you're doing now trades truly huge amounts of space for its speed. And so we come back to the problem of how to avoid having to either:

  • Save and load the huge lookup table quickly.
  • Or, generate it from the more compact representation more quickly.

And given the huge limits you are working with--30,000 X ranges from 0 .. 15,000,000--I don't see a solution. The presence of "inverted" ranges, in concert with the 15e6 overall range, defy all my attempts at a more compact representation. And I don't believe you'll beat Storable easily either.

In short. I have nothing to offer your original question beyond one possibility.

There is a data structure, QuadTree that might come close to achieving the performance of your current solution, but use far less memory. The problem is that writing such a solution in terms of Perl's existing data structures to achieve a pure perl solution isn't effective.

You would need to obtain a C-library and write a wrapper for it. And there is no guarantee that the solution would perform as well.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re^7: Serializing a large object by BrowserUk
in thread Serializing a large object by daverave

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    [Corion]: ambrus: Yeah, I read that, but it's somewhat vague as in what I really have to implement, and where/how my "other" mainloop should/needs to call AnyEvent
    [Corion]: (or maybe I just work better from existing code that I munge until it works and I understand it rather than a short abstract text like "implement everything that's needed" ;) )
    [ambrus]: Corion: I think in this case you can get away with only a stub for idle, one that always dies when you create it, because AnyEvent::HTTP doesn't use it, not even indirectly through AnyEvent::Handle or AnyEvent::Socket or AnyEvent::DNS.
    [Corion]: The "and I understand it" part is optional.
    [Corion]: ambrus: Yes but I also need to implement the file / IO watcher, because Prima has that (in Prima::File), and I need to supply the appropriate thing to make push_write etc. work with Prima
    [ambrus]: Corion: yes, you need to implement the io watcher, which should be simple because Prima::File is basically that, and the timer watcher form Prima::Timer
    [Corion]: ... or so I think. As I said, I'm somewhat vague on how to make AnyEvent cooperate with a callback-driven IO event loop that gives me callbacks when data is available or can be written

    How do I use this? | Other CB clients
    Other Users?
    Others cooling their heels in the Monastery: (9)
    As of 2016-12-08 12:15 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?
      On a regular basis, I'm most likely to spy upon:













      Results (141 votes). Check out past polls.