Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Maintaining uniqueness across millions of items

by ChuckularOne (Parson)
on Aug 13, 2004 at 13:03 UTC ( #382649=note: print w/replies, xml ) Need Help??


in reply to Maintaining uniqueness across millions of items

There is always the possibility that you could generate the numbers and create a checkssm that is significantly smaller than 16 digits.

Then you would only have to compare the checksums (and hold them in memory) for uniquness.

The only caveat being that in a list of 600,000,000 keys you, most likely, will have some codes/numbers which are unique but have the same checksum.
  • Comment on Re: Maintaining uniqueness across millions of items

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://382649]
help
Chatterbox?
[Corion]: 1nickt: Not in the general sense... I only have very specific crawlers, but not a simple crawler like that ;) But maybe that would be a good application/( stress) test for Future::HTTP to parallelize
[Corion]: Also, a good application to test my API to rate limit things

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (6)
As of 2017-10-18 11:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My fridge is mostly full of:

















    Results (244 votes). Check out past polls.

    Notices?