Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

I had a similar requirement a few years ago and (back then) the fastest mechanism available to me that provided shared access and fast lookup, was to use the file system.

For sake of discussion, assuming that your userids consist of mixed case ANSI alphanemerics -- ie. 62 chars. If you have 10 million users and use the first 3 characters in their names as an index into a first level of subdirectories, you'll have (on average) 42 users in each second level subdirectory -- so lookup is fast.

The directory structure looks like this:

/yourapp/index/ash/ashford/7/ /bre/brent/3/ /cra/crawford/4/

And the process of lookup/increment is:

my $prefix = '/yourapp/index'; my $userid = ...; my $idx = substr $userid, 0, 3; my $limitReach = 1; { opendir DIR, "$prefix/$idx/$userid/" or die $!; my $count = readdir DIR; last if $count >= LIMIT; rename "$prefix/$idx/$userid/$count", "$prefix/$idx/$userid/" . $c +ount + 1 or redo; $limitReached = 0; } ## use $limitReached to decide further action

If your data is to persist, you are going to have to do at least one directory lookup to find the DB file -- and usually more than one -- so the directory look is effectively free. And as rename is atomic, the shared data problems are taken care of without the need for time-costly, locking and polling.

The more characters in the alphabet available for your userids, the more well spread your directory structure and the faster the lookups. The only real restriction is that the alphabet must be compatible with your file systems naming conventions, which isn't usually a problem.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
/div

In reply to Re: Daily Counters by BrowserUk
in thread Daily Counters by docbrown25

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others romping around the Monastery: (13)
    As of 2014-10-22 13:59 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      For retirement, I am banking on:










      Results (118 votes), past polls