Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: how to speed up dupe checking of arrays

by radiantmatrix (Parson)
on Jul 31, 2007 at 14:45 UTC ( #629840=note: print w/replies, xml ) Need Help??


in reply to how to speed up dupe checking of arrays

Just a thought that a DB could be leveraged here? LOAD DATA INFILE is pretty fast, and SELECT DISTINCT is both easy to code and pretty quick...

<radiant.matrix>
Ramblings and references
The Code that can be seen is not the true Code
I haven't found a problem yet that can't be solved by a well-placed trebuchet

Replies are listed 'Best First'.
Re^2: how to speed up dupe checking of arrays
by ultibuzz (Monk) on Jul 31, 2007 at 14:58 UTC

    we have a oracle 10g on a hp superdome loading all data with sqlloader and direct=true as option takes longer then the dupechecking with perl and then load filtered data in.
    loading the data with perl into the db or normal sqlloader without tunig woud take ages.
    with direct=true the sqlloader pumps in 10million rows in less then 20 sec, without direct=true it takes 10 minutes+ because orcale set commit points and check for data correctness ^^.
    other point is we need several indexes and partiotion groups on it wich woud take hours to create if we used the unfiltered data.

    kd ultibuzz

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://629840]
help
Chatterbox?
[1nickt]: But, they do have red keyboard backlighting! And gargoyles on the front, or words like "Maxxx" here and there. They know their demographic!
[1nickt]: True gamers don;t buy Dells, HPs, or Lenovos, I think ;-)
[ambrus]: The keyboard I bought doesn't have backlighting. I specifically had to select for that, because so many keyboards these days have lights. Why would I want lights in the keyboard? I want lights in the room, such as on the ceiling, not in the keyboard.
choroba has a small USB lamp
[ww]: ++ all (phony labels re "gaming"
[1nickt]: ambrus perhaps you know where the keys are?
[ww]: OTOH, -- for ads suggesting that being good at games is somehow eq "greatness"

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (14)
As of 2017-03-27 13:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Should Pluto Get Its Planethood Back?



    Results (320 votes). Check out past polls.