Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??


Regarding how to choose the size of a dataset to make it take 15 minutes: If I wanted to do that, I'd start out by using progressively larger datasets to see how the time changes with dataset size. For example, look at these three datasets:

Dataset sizeSubroutine ASubroutine BSubroutine C
1000 6 1 30
2000 11 4 40
3000 17 8 48
4000 22 16 55

Once I get a few samples, I'd try to predict the next dataset size. If you look at the values for subroutine A, it looks like a simple linear progression: it looks like it handles about 160-ish items per second for all four dataset sizes. So if I wanted to make it run for 15 minutes, I'd expect it to take 15*60*160 data items. Subroutine B, however isn't linear. It looks like it gets slower and slower as the dataset increases--in this case it takes roughly T = (X/1000)^2 seconds for a dataset. Solve for X when T=15*60 seconds and that would be a reasonable prediction. The third subroutine starts out pretty slow, but you can see that the time it consumes changes less and less as you add data samples. (I was shooting for a logarithmic progression, but I don't feel like doing the math, so that one's left as an exercise for the reader!)

*HOWEVER*, these predictions assume that everything else will remain the same as the dataset grows. But you may find that at a certain dataset size, an algorithm may take a sudden, drastic increase in the time it takes. (For example you might exhaust your main memory and the OS may start swapping.) So rather than immediately going for 15 minutes, you might try to predict a dataset size that would take less time, like one or two minutes and see how far off you are. I frequently approach a final value by doubling each time (unless I'm using something like subroutine B).

I hope this is somewhat helpful...

Modern computers are so fast, though, that I expect it'll take a pretty large dataset to consume 15 minutes. (That, or a sufficiently horrible sort algorithm.)


When your only tool is a hammer, all problems look like your thumb.

In reply to Re^3: How to generate test data? by roboticus
in thread How to generate test data? by abdullah.yildiz

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others meditating upon the Monastery: (5)
    As of 2017-01-20 02:08 GMT
    Find Nodes?
      Voting Booth?
      Do you watch meteor showers?

      Results (173 votes). Check out past polls.