Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
Just another Perl shrine
 
PerlMonks  

Re: IO::Uncompress::Gunzip to scalar takes hours (on windows)

by BrowserUk (Pope)
on May 22, 2013 at 05:06 UTC ( #1034671=note: print w/ replies, xml ) Need Help??


in reply to IO::Uncompress::Gunzip to scalar takes hours (on windows)

How big is the expanded file on disk?


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re: IO::Uncompress::Gunzip to scalar takes hours (on windows)
Re^2: IO::Uncompress::Gunzip to scalar takes hours (on windows)
by cmv (Chaplain) on May 22, 2013 at 12:02 UTC
    The expanded file comes to about 355M

      So, that shouldn't be taxing your memory too much.

      I couldn't reproduce your problem. I ran your program verbatim on my windows system running Vista64, (AS)5.10.1 and IO::Compress:2.060 and on files 65/210MB and 80/600MB and they both took roughly 5 seconds to disk and 3 seconds in memory.

      Does the same problem exhibit on all files when decompressed to memory or is it confined to one particular file?

      One possibility (mentioned by Corion) is indicate by excessive page faults for the process(*). If the process shows a page fault delta greater than double digits per second, you have probably encountered the malloc problem. But if that were the case, I would have expected to be able to reproduce it here on my standard AS install.

      (*You'll need Process Explorer or work out how to use perfmon.exe to find this information.)


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        The same problem on all files.

        I've updated the original post to have profile data between a good run and bad run - maybe that will help.

        It seems to be dawning on me, that the problem has more to do with the old AS 5.8.9 that I'm using rather than with the IO::Uncompress::Gunzip module. However updating will take lots of testing. Not sure which is the worst of the two evils.

        Thanks for the help! ++BrowserUk

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1034671]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2014-04-20 15:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (485 votes), past polls