Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number

Re^5: Scaling Hash Limits

by BrowserUk (Pope)
on Sep 20, 2013 at 23:13 UTC ( #1055080=note: print w/replies, xml ) Need Help??

in reply to Re^4: Scaling Hash Limits
in thread Scaling Hash Limits

but I can't see how to reduce this memory requirement to only 4 megabytes.

I said ~ 8 megabytes not 4.

In Scaling Hash Limits the OP said: "my simple hash of scalars (for duplicate detection) hits a size of over 55 million"

55 million / 8 / 1024**2 = 6.55651092529296875 MB.

He also mentions 180 million: 180e6 / 8 / 1024**2 = 21.457672119140625 MB. But that's before de-duping, the purpose of the exercise. But its possible his list contains no duplicates.

Of course, looking around I see that twitter uses 64-bit numbers for their user ids. And that it 20 digits not 12. Then again, they are only just now claiming 500 million users which is: 500e6 / 8 / 1024**2 = 59.604644775390625 MB which should be handleable by any modern machine with ease.

Of course, it is also possible that they do not use sequential numbers for their IDs, but rather the 64-bit number is a hash of some aspect of the account -- the name or similar -- in which case the idea won't work because 2**64 / 8 / 1024**3 ~= 2 billion GB.

But if that were the case, the OP probably wouldn't be talking about "12-digit numbers".

Of course, the OP also doesn't explicitly mention 'user' ids, just "ids", and given the number -- 180 million; roughly the number of twits per day currently -- these could be message ids; which probably are allocated sequentially?

Had the OP deigned to answer a couple of questions, much of the uncertainty could have been resolved.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^6: Scaling Hash Limits
by Laurent_R (Canon) on Sep 21, 2013 at 09:11 UTC

    Thank you for your interesting answer, BrowserUk, I can see that we are thinking of more or less the same type of techniques. It is just that the OP did not provide enough detailed information for us to be able to make very accurate predictions on what it would take. My own understanding was that the IDs probably referred to message IDs, thus making sequential allocation quite plausible, but, of course, we don't know for sure.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1055080]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (5)
As of 2018-06-20 04:51 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (116 votes). Check out past polls.