Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^4: IO::Uncompress::Gunzip to scalar takes hours (on windows)

by cmv (Chaplain)
on May 22, 2013 at 15:25 UTC ( #1034767=note: print w/ replies, xml ) Need Help??


in reply to Re^3: IO::Uncompress::Gunzip to scalar takes hours (on windows)
in thread IO::Uncompress::Gunzip to scalar takes hours (on windows)

The same problem on all files.

I've updated the original post to have profile data between a good run and bad run - maybe that will help.

It seems to be dawning on me, that the problem has more to do with the old AS 5.8.9 that I'm using rather than with the IO::Uncompress::Gunzip module. However updating will take lots of testing. Not sure which is the worst of the two evils.

Thanks for the help! ++BrowserUk


Comment on Re^4: IO::Uncompress::Gunzip to scalar takes hours (on windows)
Re^5: IO::Uncompress::Gunzip to scalar takes hours (on windows)
by BrowserUk (Pope) on May 22, 2013 at 16:08 UTC
    I've updated the original post to have profile data between a good run and bad run - maybe that will help.

    Hm. That tells us that the vast majority of the time (98%+) in the slow version is spent in Compress::Raw::Zlib::inflateStream::inflate

    Which doesn't make a whole lot of sense given that they are inflating the same data from the same place in both cases.

    A line profile might shed more light; but it seems doubtful given the time is spent doing what is effectively input; but the difference in the code is the output target.

    Again, I urge you to obtain the page fault stats.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      You are correct! I've updated the original post with the pagefault data, and it looks like this is the problem.

      Do you have a suggestion on the easiest/fastest way for me to fix this (I'd like to avoid upgrading the AS perl if possible - lots of retesting needed for this)?

      Can I do something programmatically? I've tried Corion's suggestion, but that didn't seem to work. Maybe I'm not doing it quite correctly...

        Do you have a suggestion on the easiest/fastest way for me to fix this (I'd like to avoid upgrading the AS perl if possible

        The simplest way would be to get hold of the AS sources for that version of perl, make teh one line patch I posted somewhere on this site. I'd look it up, but unless you are prepared to pay for it, I don't think the AS sources for 5.8.9 are available any longer. You might then be able to replace Perl.exe/perl5.8.9.dll into your existing distribution and have the problem go away. I don;t guarentee it, but it ought to work.

        Can I do something programmatically?

        I seem to recall before working out the patch, I came up with a scheme that seemed to mitigate the problem for the most part.

        That involved pre-allocating memory to cover the final size of the growth pattern that is causing the page faults -- in smallish chunks so that it gets retained by the process pool when freed, rather than in one chunk which would be returned to the OS when freed. From memory, this pre-stuffing of the memory pool avoid many of the page faults, at least some times.

        Something like:

        ## prior to the main memory consuming process. my @dummy; $dummy[ $_ ] .= chr(0) x 4096 for 0 .. 256 * 1024; undef @dummy; ## Add 1GB worth of pages to the memory pool.

        Worth a try, but I can't find any examples of from that far back and cannot verify it.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1034767]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (12)
As of 2014-10-23 21:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (129 votes), past polls