Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Re^3: Out of Memory when generating large matrix

by BrowserUk (Pope)
on Mar 05, 2018 at 16:39 UTC ( #1210359=note: print w/replies, xml ) Need Help??

in reply to Re^2: Out of Memory when generating large matrix
in thread Out of Memory when generating large matrix

Therefore, in total, it should have 4**21 combination.

Yes. But not all of them -- in fact a very small percentage -- will appear; so rather than reserving space for trillions of counts that will 99% never be used, you can choose to only count those that do appear. That will save huge amounts of space and speed up your processing.

With 197 files * 8000 lines, there are at most 1.5 million 21kmers -- if they are all unique -- so there in no need to reserve space for all 4.3 trillion possibilities. 197*8000 / 4.3e12 * 100 = 0.0000358%

I'm trying to count how many time each 21mers has shown among these 197 files.That's why I need a matrix to do so.

Fair enough, but you don't need to count them all in one huge matrix. You can process the 197 files one by one and only store the nonzero counts across files.

Could you post a small sample (say 20 lines) of what is in your kmer files?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit
  • Comment on Re^3: Out of Memory when generating large matrix

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1210359]
[marto]: a grain of sand or dust in the keyboard, replace the motherboard. No we can't transfer your data for you

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (6)
As of 2018-07-23 10:03 GMT
Find Nodes?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?

    Results (462 votes). Check out past polls.