Re: how to speed up dupe checking of arrays

[radiantmatrix]

in reply to how to speed up dupe checking of arrays

Just a thought that a DB could be leveraged here? LOAD DATA INFILE is pretty fast, and SELECT DISTINCT is both easy to code and pretty quick...

[ultibuzz]

we have a oracle 10g on a hp superdome loading all data with sqlloader and direct=true as option takes longer then the dupechecking with perl and then load filtered data in.
loading the data with perl into the db or normal sqlloader without tunig woud take ages.
with direct=true the sqlloader pumps in 10million rows in less then 20 sec, without direct=true it takes 10 minutes+ because orcale set commit points and check for data correctness ^^.
other point is we need several indexes and partiotion groups on it wich woud take hours to create if we used the unfiltered data.

kd ultibuzz