Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re^2: Deconvolutinng FastQ files

by BrowserUk (Pope)
on Aug 06, 2012 at 20:55 UTC ( #985833=note: print w/replies, xml ) Need Help??

in reply to Re: Deconvolutinng FastQ files
in thread Deconvolutinng FastQ files

Just to make the point about the inappropraitness of Tie::File for this.

  1. Using normal IO on a 400MB fastQ file takes 20 seconds:

    And use 1.3 MB of ram; performs 96e3 reads 97e3 writes; and consumes 42 billion clock cycles.

    Giving a projected runtime for the OPs 34GB file of 30.8 minutes.

  2. Using Tie::File on that same 400MB file takes 1340 seconds:

    And uses 601MB of ram; performs 10000e3 reads; 95e3 writes; and consumes 3,182 billion clock cycles.

    Giving a projected runtime for the OPs 34GB file, of 34 hours!

Tie::File has its uses. This is not one of them.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Replies are listed 'Best First'.
Re^3: Deconvolutinng FastQ files
by Kenosis (Priest) on Aug 06, 2012 at 21:34 UTC

    I'll remember:

    my %mindHash = ( 'Tie:File + Large File' => '"Run Away!" (King Arthur, Monty Py +thon)' );
      my %mindHash = ( 'Tie:File + Large File' => '"Run Away!" (King Arthur, Monty Py +thon)' );

      Oh noes! Kenosis has deleted all of his memory except for the fact that Tie::File and large files are to be avoided.

      $mindHash{'assign to entire hash'}='only if you want to delete all exi +sting data';


      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

        But he'll never feel the loss ... unless he reads your node.

        This is good! (However, I may have just increased my memory, since the word Kenosis means emptiness... :)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://985833]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (2)
As of 2018-05-26 04:27 GMT
Find Nodes?
    Voting Booth?