Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Perl v5.6.1 looking for .pmc files

by ruscoekm (Monk)
on Oct 22, 2002 at 14:01 UTC ( #207078=perlquestion: print w/ replies, xml ) Need Help??
ruscoekm has asked for the wisdom of the Perl Monks concerning the following question:

Hi All. I have a fairly large Perl program (approx. 150 classes) and have been spending some time trying to speed it up (particularly startup time). I noticed from the output of strace -c that there were 600 stat64 calls, out of which 512 got ENOENT. Just about all of these failed stat64 calls were looking for foo.pmc (where foo corresponds to the name of a Perl module required by my program).

My Perl program is taking around 10secs to start up, of which 2-3 secs are spent in these stat64 calls.

Now, I am aware that PMC stands for Perl Magic Cookie in Perl6, but this is Perl5 :-) My next thought was that perhaps .pmc refers to a compiled version of a Perl module (that is, compiled with B::Bytecode). I could not find any documentation which verified this.

Does anybody know if Perl is indeed looking for compiled versions of required modules? If so, is there a way to disable this? Given that B::Bytecode is still experimental, it seems odd that this is the default behaviour. (Of course, I may have got entirely the wrong end of the stick :-)

Best wishes, Kevin

Comment on Perl v5.6.1 looking for .pmc files
Re: Perl v5.6.1 looking for .pmc files
by bluto (Curate) on Oct 22, 2002 at 16:03 UTC
    Check your filesystem. 2-3 seconds for 600 stats sounds awfully slow to me. On my AIX box I get around 1000 in about a tenth of a second. If you are running over NFS, your general performance could be slow.

    Also, it might be nice to see how quick your script starts up the *second* time (i.e. after filesystem info is cached).

    bluto

      Yes, it is very slow, thanks to a very slow network. That is why I am particularly concerned to reduce the number of such calls :-)

      Re running the program a second time, I did that. Should have mentioned this. No difference to number of stat64 calls, but then I would not expect any.

      My interest is really why Perl is looking for .pmc files in the first place.

      Cheers, Kevin

        I guess my point was that it seemed like you were trying to solve a problem related to a broken hardware environment by changing the way Perl does it's job. I don't envy you since it sounds like you can't install everything on a local disk, or use something like NFS mounts with cachefs.

        FWIW, the number of stat64 calls would not decrease, but even NFS should (I think) cache the results of these calls rather than going over the network, but I digress...

        bluto

Re: Perl v5.6.1 looking for .pmc files
by Fastolfe (Vicar) on Oct 22, 2002 at 16:17 UTC

    10 seconds to start up sounds about right if you have 150 modules you're loading and compiling also on modest hardware. Consider using Autoloader to load functions on demand instead of at startup to avoid some of that penalty. Also consider using Devel::DProf (run Perl with -d:DProf) and dprofpp to profile your code and see where perl is spending most of its time.

    A 'stat' call is common if Perl is using 'stat' to look through @INC trying to search for your modules. A given module will rarely be found in the very first place perl looks, so you're going to get a lot of these failed calls.

    Though I'm not really sure why it'd be looking for a .pmc and not .pm. I haven't heard anything about that. But still, as another poster mentioned, these calls should take almost no time whatsoever. I doubt these calls (failing or otherwise) are what's causing your performance woes.

      Hi

      Re Autoloader, I did consider it, but all of the modules are required, so I have to take the hit at some point. Admittedly, there is something to be said for spreading the pain a bit :-)

      Re DProf, I find it very useful for general tuning, but do not think it will help me diagnose compile time performance issues. I find strace more helpful for that.

      Re stat calls, I already set up @INC so that only the required dirs are on the path and the dirs are in decreasing order of used modules, in order to obtain the lowest number of open failures.

      Perl does not actually issue stat calls when looking for .pm files. It just issues open calls and gets ENOENT if they are not there. stat calls are issued for .pmc, .so and .bs files (there may be others). In my example, the vast majority of stat calls are for .pmc files.

      Re the stat calls not being the problem, I can see from the strace output that they are taking around three seconds, which is a significant part of the startup time.

      Cheers, Kevin

        I do not like part of my previous reply. It is not true to say that DProf is not useful for diagnosing compile time performance issues in general. I just do not think it is helpful to my particular issue.
Re: Perl v5.6.1 looking for .pmc files
by fglock (Vicar) on Oct 22, 2002 at 17:55 UTC

      Ha! Many thanks for digging that one out of the archives. Nick's comment that "The required stat()s on a sick NFS system are significant!" summed up my feelings exactly :-)

      By the sound of the subsequent mail chain, there should be a workaround in v5.8.1. Until then, I guess I will have to look for improvements elsewhere.

      Cheers, Kevin

      Bravo for finding this!

      I too wondered what the flap was with these .pmc stat calls, because the original question was well-asked.

      Sooo... instead you could speed things up by compiling the modules to bytecode using Module::Compile (okay, in theory).

        yes! except that the feature is more used nowadays to preprocess complex code into plain perl5, instead of bytecode. v6, for example, process perl6 source code into plain perl5 "pmcs".

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://207078]
Front-paged by krisahoch
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (7)
As of 2014-08-23 16:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (175 votes), past polls