Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Module::Compile (or: what's this PMC thingy?)

by Juerd (Abbot)
on Mar 12, 2006 at 20:22 UTC ( #536132=perltutorial: print w/ replies, xml ) Need Help??

This document is written in response to many questions triggered by PMC (Perl Module Compiled) survey. It's not useful as a tutorial, because Module::Compile is in its infancy and very likely to change further, but posted here because it provides information only.

History

Once upon a time, people thought that maybe Perl would be faster if compiled code could be re-used in a later session. (Actually, many people still do.) So they created support for compiled Perl Modules, and where Python libraries got their "c" after their "py", compiled Perl modules would get their "c" after their "pm", and the "pmc" extension was born.

It works somewhat like this: you have a module, say, Test.pm, and the following line gives you the compiled version:

perl -MO=Bytecode,-H -MTest -e1 > Test.pmc
Provided that Test.pmc is in the same directory that Test.pm is, perl will automatically load Test.pmc instead of Test.pm the next time someone uses "use Test" or "require Test". This is because in every directory of @INC, perl first sees if a .pmc exists, and if it does, it loads that, and if not, then it tries to load the .pm.

What's going on?

If you strace the process, you can easily see what was going on before:

stat64("/usr/local/lib/perl/5.8.7/Test.pmc", 0x7fe20a50) = -1 ENOENT ( +No such file or directory) open("/usr/local/lib/perl/5.8.7/Test.pm", O_RDONLY|O_LARGEFILE) = -1 E +NOENT (No such file or directory) stat64("/usr/local/share/perl/5.8.7/Test.pmc", 0x7fe20a50) = -1 ENOENT + (No such file or directory) open("/usr/local/share/perl/5.8.7/Test.pm", O_RDONLY|O_LARGEFILE) = -1 + ENOENT (No such file or directory) stat64("/usr/lib/perl5/Test.pmc", 0x7fe20a50) = -1 ENOENT (No such fil +e or directory) open("/usr/lib/perl5/Test.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No s +uch file or directory) stat64("/usr/share/perl5/Test.pmc", 0x7fe20a50) = -1 ENOENT (No such f +ile or directory) open("/usr/share/perl5/Test.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No + such file or directory) stat64("/usr/lib/perl/5.8/Test.pmc", 0x7fe20a50) = -1 ENOENT (No such +file or directory) open("/usr/lib/perl/5.8/Test.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (N +o such file or directory) stat64("/usr/share/perl/5.8/Test.pmc", 0x7fe20a50) = -1 ENOENT (No suc +h file or directory) open("/usr/share/perl/5.8/Test.pm", O_RDONLY|O_LARGEFILE) = 4
Before perl finds Test.pm in /usr/share/perl/5.8, it first tries a lot of other directories, each time stat()ing Test.pmc first, and then trying to open Test.pm. Now, because I have created a Test.pm in /usr/share/perl/5.8, perl's done a little earlier:
stat64("/usr/local/lib/perl/5.8.7/Test.pmc", 0x7f95aa50) = -1 ENOENT ( +No such file or directory) open("/usr/local/lib/perl/5.8.7/Test.pm", O_RDONLY|O_LARGEFILE) = -1 E +NOENT (No such file or directory) stat64("/usr/local/share/perl/5.8.7/Test.pmc", 0x7f95aa50) = -1 ENOENT + (No such file or directory) open("/usr/local/share/perl/5.8.7/Test.pm", O_RDONLY|O_LARGEFILE) = -1 + ENOENT (No such file or directory) stat64("/usr/lib/perl5/Test.pmc", 0x7f95aa50) = -1 ENOENT (No such fil +e or directory) open("/usr/lib/perl5/Test.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No s +uch file or directory) stat64("/usr/share/perl5/Test.pmc", 0x7f95aa50) = -1 ENOENT (No such f +ile or directory) open("/usr/share/perl5/Test.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No + such file or directory) stat64("/usr/lib/perl/5.8/Test.pmc", 0x7f95aa50) = -1 ENOENT (No such +file or directory) open("/usr/lib/perl/5.8/Test.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (N +o such file or directory) stat64("/usr/share/perl/5.8/Test.pmc", {st_mode=S_IFREG|0644, st_size= +720, ...}) = 0 stat64("/usr/share/perl/5.8/Test.pm", {st_mode=S_IFREG|0644, st_size=2 +8863, ...}) = 0 open("/usr/share/perl/5.8/Test.pmc", O_RDONLY|O_LARGEFILE) = 4
Don't be fooled by the stat() on Test.pm; it's ignored. The actual open() is now on Test.pmc instead of Test.pm.

Sad ending

So, why did (and does) nobody really use this? Well, it turns out that bytecode loading is indeed faster in some (but not all) cases, but not very reliable: segfaults are quite common, and nobody so far has been able to fix it. In addition, the bytecode files are not portable between systems, or even Perl versions.

But there is another problem with Byteloader. The PMC file is actually just a normal Perl file: via a source filter, the bytecode is loaded. That's nasty, and makes using precompiled bytecode even less interesting.

Thus a sad story seemed to end, and the PMC feature was ignored, forgotten, and almost about to be deprecated... (Features aren't removed lightly in Perl - even if nobody uses it, a deprecation cycle is needed.)

Joyful rebirth of PMC and source filters

... if it weren't for the incredible geniuses who work on Perl 6 to find a very neat way to use the PMC feature to make Perl 6 acceptance and initial use less painful!

But actually, it's not even really Perl 6 related. It's just a very neat thing that you can also put to good use if you don't want to use Perl 6 yet. You see, we have source filters that can do very nice and spiffy (no pun intended) things for us, but are also known to have certain negative feelings attached to them. Sourcefilters have problems with mod_perl and other embedded perls, and they cannot be combined with other source filters. Besides that, source filters are always runtime overhead, and the result is usually not cached anywhere. You can't even debug the result easily, because the intermediate code is not available on disk.

The new invention exists in the form of Module::Compile, formerly (for less than a day — that's how fast things go in Freenode #perl6) known as PMC::Filter.

Instead of using the normal source filters, Module::Compile writes the "compiled" code to a .pmc file, and that's loaded from that point forward. The nice trick is that the PMC file doesn't have to be bytecode. It's just a normal pure Perl file.

But we call them compilers now

A "compiler" (read: source filter) is built quite easily:

package Foo; use Module::Compile -base; sub pmc_compile { my ($class, $source) = @_; # Convert $source into (most like Perl 5) $compiled_output return $compiled_output; } 1;
and using that is also very straight forward:
# Unfiltered code here quux(bar); use Foo; # This code is filtered! # Ehh... I mean compiled! no Foo; # Unfiltered code here, again quux(bar);

A nice solution for everyone who has found a good use for source filters, but doesn't use them. Module::Compile supports using multiple compilers (source filters, for those who still don't get it) at the same time, the intermediate code is available for simpler debugging, and it works well even with embedded perls.

Perl 6 again

Remember that it was Perl 6 people that invented all this? It must have something to do with Perl 6, then, right? Well, but of course! The idea is to compile Perl 6 code into Perl 5 code, so that you can use it without Perl 6. You can do this to have a module that has parts in Perl 5 and parts in Perl 6, which is nice if you want to upgrade an existing module with some of the nice new features that Perl 6 has to offer, but don't want to rewrite everything just yet.

An even nicer thing, though, is that this allows you to write modules that are fully Perl 6, that will work both in Perl 5 and Perl 6.

Distribution

There are two problems with the PMC technique, though:

  1. You need to have all the stuff installed that is needed for a succesful compilation
  2. Users don't have write access in the system library directories
But there is one solution that solves both these problems: let Module::Compile generate your PMCs as you develop your software, and when you're done, distribute the compiled result along with the rest, so that people can use that. Also, if the PMC is installed with the rest, then someone with superuser privileges installs it.

Perl 6 example

Oh, you want example code, of course! Here it is, again copied straight from Module::Compile's documentation:

# User.pm use v6-pugs; module User; ...some p6 code here... no v6; ...back to p5 land...
Oh well, I'll steal their explanation too:

On the first time this module is loaded, it will compile Perl 6 chunks into Perl 5 (as soon as the no v6 line is seen), and merge it with the Perl 5 chunks, saving the result into a User.pmc file.

The next time around, Perl 5 will automatically load User.pmc when someone says use Foo. On the other hand, Perl 6 can run User.pm as a Perl 6 module just fine, as "use v6-pugs" and "no v6" both works in a perl 6 setting also.

The pmc module will imbue v6.pm with the ability to check for Foo.pmc's updatedness also, and if it's up to date, then it'd touch its timestamp so the .pmc is loaded on the next time.

Final words

And that's why you shouldn't disable the PMC feature in Perl: although unused since its invention in 1999, there's now a very powerful use for it! Fortunately, so far the survey shows that only Mandrake/Mandriva users have it disabled. They can fix it soon, by upgrading.

Do note that Perl 6 is very far from production quality, and that Module::Compile is also still very young. You should probably not use them for anything serious yet.

Thank you, ingy++ and audreyt++, for bringing us this wonderful innovation!

Comment on Module::Compile (or: what's this PMC thingy?)
Select or Download Code
Re: Module::Compile (or: what's this PMC thingy?)
by Aristotle (Chancellor) on Mar 12, 2006 at 21:20 UTC

    Very nice, that looks much like something I was going to write for a completely different purpose. Very nice. *rubs hands, grinning evilly*

    Makeshifts last the longest.

Re: Module::Compile (or: what's this PMC thingy?)
by zentara (Archbishop) on Mar 13, 2006 at 13:12 UTC
    Is this anything like the trick for making Perl bytecode, where you
    perlcc -B -o script.pby srcscript.pl
    Then the script.pby is precompiled Perl bytes?

    I'm not really a human, but I play one on earth. flash japh
Re: Module::Compile (or: what's this PMC thingy?)
by audreyt (Hermit) on Mar 13, 2006 at 23:50 UTC
    I have summarized my initial proposals for the new direction for .pmc to Perl5-Porters here: http://www.nntp.perl.org/group/perl.perl5.porters/110699 -- it might be worth a look as well.

    Also, to cleanly shift control between .pmc and .pm, #p5p has been talking about a "goto *FH" form for bleadperl -- but this may or may not happen. Would be so much fun, though.

Re: Module::Compile (or: what's this PMC thingy?)
by ingy (Initiate) on Mar 15, 2006 at 10:44 UTC
    Great recap Juerd. You forgot the best part though. perl's "compile" switch (-c) now works.
    perl -c lib/Foo.pm
    dwims! ;)

      Did i miss something? Isn't -c a syntax check mode?


      ___________
      Eric Hodges

        Yah sure. Perl does all the normal compilation stuff it always does. It also does an exit() before runtime. That's all. "Getting it compiled" is bending the truth.

        ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

Re: Module::Compile (or: what's this PMC thingy?)
by stefp (Vicar) on Mar 19, 2006 at 14:40 UTC
    Because we are here in the Perl5 implementation world these PMCs (Perl Module Compiled) have no relationship whatsoever with PMCs (Parrot Magic Cookies) of the parrot implementation.

    -- stefp

Re: Module::Compile (or: what's this PMC thingy?)
by Anonymous Monk on Jan 04, 2008 at 14:04 UTC
    Thank you very much. I found all the answers to my questions. Great explanation. You`re cool.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perltutorial [id://536132]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (15)
As of 2014-07-11 17:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (232 votes), past polls