Re: Counter - number of tags per interval

50 distinct id's and each id in file.2 has approximately 5-7 * 10^9 lines.

File 2 has ~350 billion lines?

What is the maximum coordinate?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: Counter - number of tags per interval

Replies are listed 'Best First'.
Re^2: Counter - number of tags per interval by baxy77bax (Deacon) on May 04, 2013 at 15:23 UTC
yes that is true (why is this so odd ?? in bioinformatics you have 50-500gb read-mapps (files) and nobody finds that odd :) ). Maximum coordinate ?? well that depends on a simulation. but i would say for id = a, max is app. 5*10^9 i mean i need to find this out by going through the file.	[reply]
Re^3: Counter - number of tags per interval by BrowserUk (Patriarch) on May 04, 2013 at 15:35 UTC
why is this so odd ?? i Not odd. Just important to know as it greatly affects the possible solutions. How long does/do you estimate it to take to process those two files with your current code? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^4: Counter - number of tags per interval by baxy77bax (Deacon) on May 04, 2013 at 16:27 UTC
current estimates: One Id app. 8-12 h 50 id's app. 500h = app. 20 days which is 20 times longer then the actual computation since there will be more of such simulations i think a better solution would be preferable. Unfortunately i cannot temper with the simulation part since i have no source , but even if i did i wouldn't do it since there are a lot of heuristics involved which i honestly don't understand.	[reply]
Re^5: Counter - number of tags per interval by BrowserUk (Patriarch) on May 04, 2013 at 17:27 UTC


Pathologically Eclectic Rubbish Lister
	PerlMonks