Re: Huge data file and looping best practices

[przemo]

Looks like we have O(N^2*K) complexity (N=number of patents, K=number of characteristics) here... not good...

If you really have to count difference for every pair of patents, try using Bit::Vector for every line (i.e. one characteristic = one bit), calculate the number of different positions they have with XOR operation and count set bits in the resulting vector. This should take less memory and should be also faster. (?)

Although I have some doubts about the time needed to do so (you need to write to the output about (8*10^6)^2)/2 = 32 "teralines").

[carillonator]

thanks @przemo. I'll check out Bit::Vector. We think there are only about 400,000 unique characteristic sets among the 8 million patents, so we'll probably end up dealing with that data set instead, on account of the absurd amount of data this program would return.

[davido]

If you have 400,000 unique characteristic sets among the 8 million patients, now you're getting somewhere. If you could find a way to consistently stringify a given set the same way each time it comes up, you could turn that into a hash key, and as its value create a datastructure of patient names. Now you have a workable structure that could be split into manageable files based on the groupings by unique characteristics.

...just a thought, though I'm not sure how helpful it is.