Re^3: perllVm: A Linux test of how Perl and LLVM would work together.

[BrowserUk]

flexvault. Whilst 15% probably equals as much as p5p have succeeded in clawing back since 5.6.1, if that is all that is available, then the additional complexity of adding LLVM toolchain and its dependencies to the mix isn't worth it for your average perl user and certainly not for p5p -- given the possibilities of per platform problems with the switch from gcc to llvm.

For example, I've spent the week getting changes made to the llvm emitter, to fix up incompatibilities between its .o representation and the MS 64-bit ABI. (To their credit, once I'd figured out the right approach and danced their tune by fulfilling their SOP, they had a patch which fixed the problem within a couple of hours. They are a very impressive bunch of developers and nice guys to boot.)

I've also had my work cut out tracking down a stack dump in their front-end. The source turn out to be a (known) bug in the 64-bit version of the MS STL. But the most impressive bit of the experience, is that the stack trace that the debug version of their tool-set produced when the problem (an assertion failure inside the STL) occured, led me directly to the line&offset in the LLVM source where the problem arose. For now, a hand-coded function substituted for the failing STL routine gets me moving along.

But using LLVM as an alternative C compiler was never going to be enough to warrant the effort. For a start, you do not get JIT from llvm without making changes to the project (Perl) code-base.

For JIT to work, you have to:

1. link the LLVM JIT engine into the Perl binary.
2. convert the perl program being run into LLVM IR at runtime and feed it to that engine.

That is not a transparent -- compile with llvm and get JIT -- automated procedure. It requires (possibly substantial) changes to the perl sources. This is why I isolated the sources required to build the perl executable binaries from the rest of the perl infrastructure.

The (my) idea is to have a process that takes a new perl distribution, sucks out the smallest number of files possible from it, applies LLVM to them and builds versions of perl.exe perl516.dll/.so that they inter-operate with the rest of the standard perl distribution; .pms, XS an'all.

Of course, that breaks the next time p5p break binary compatibility -- about once every 3 days lately -- but if enough gains can be demonstrated in this way -- I don't yet know how much that would need to be? 200%? -- then it might then encourage dialogue, even active participation.

But if not, for those people for whom this is important, the extra work of installing LLVM and a process to produce this alternative binary compatible "fast Perl" would be worth it. (And, for windows at least, making that alternative binary available as a 'plug-in' to a standard binary distribution can make it available more widely.)

And if it meant trailing the latest version of perl, and only catching up every 2 or 3 or 5, p5p distributions, that would be okay. There has been very little new that I've missed since the addition of defined-OR in 5.10; and quite a lot -- like smart-matching, use Switch; those funny arrays as hashes that came and went; etc. -- that I've actively avoided from the get-go, that are now deprecated or seem likely to disappear before too soon. (I don't even use 'say' because they screwed up the implementation and -l is just easier to turn on and off.)

As I've been saying from the start of this, I don't think that LLVM as an 'alternative C compiler' alone is ever going to produce viable benefits for the masses. The addition of llvm JIT to perl will require a substantial effort that will include (amongst other things) the splitting of the perl5x.dll into 2 or 3 seperate binaries such that the 'immutable' bits of perl -- those that define Perl -- can remain unchanged, whilst the potentially mutable bits -- the compiled representation and the runloop -- become accessible.

I also think that there is substantial potential in using the LLVM tool-sets profiling and tracing capabilities to 'zoom in' on particular areas of the perl code-base and highlight source-level changes that could be applied (by p5p), to good effect for the entire Perl community.

It is most interesting to go through the exercise of compiling selected pieces of perl source to LLVM IR and then back to C. The results are, as machine generated source code is apt to be, pretty unreadable. but by looking at small pieces in isolation and comparing it back to the input, it is possible to see the potential of small changes in the existing sources that could have substantial affects.

An old adage goes: a job started is a job half done; and another that says: the first step is the hardest. And yet another: there is a time when the talking must stop.

As a first step, the ability to get 15% (and substantially faster build times if my experience is anything to go by), for nothing more than using a different compiler is nothing to be sneezed at.

But it is only the first step. The potential is all in what comes next -- and that is only limited by the imaginations of the doers, not the nay-sayers.

[flexvault]

BrowserUk,

WOW!

The documentation code to take and and apply LLVM to a 'C' source that I referred to:

'llvm-config --cppflags --ldflags --libs core jit native -O3'

This was used against a .c file to add 'jit'. It also is in the 'new' for version 3.1

My interest is if it improves current Perl5 runtimes by xx%, then the additional work to 'improve' the Perl5 sources is even more justified.

Regards...Ed