Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

YAP6: A p5 approach to p6

by ruoso (Curate)
on Dec 24, 2007 at 17:28 UTC ( #658902=perlmeditation: print w/ replies, xml ) Need Help??

DISCLAIMER: This is a controverse opinion on several matters, and must be understood with a IMHO before every sentence. And all the critics made to other people are intended in the most positive way, and not to make them look bad.

The Perl 6 development have basically two different approaches in course, one focused in the Virtual Machine, which is the Parrot project, and other focused in the language itself, which is Pugs.

The Parrot project aims to build a state-of-the-art virtual machine, and is, in several aspects, a trully impressive vm. Parrot already have a lot of features and can already run an impressive set of languages.

The Pugs project (and here all its child-projects are counted), on the other hand, started as an effort to implement the semantics of the Perl 6 language in "whatever" runtime. The first exercise was Haskell, but some other started soon after, most notabily, the Perl 6 on Perl 5 efforts of v6-alpha, MiniPerl6 and at last, KindaPerl6.

Some marginal efforts come and go along the way, like the Lisp backend to KP6, and YAP6, which is what this post is about.

Why does it take so long?

Seriously, everybody wants Perl 6 out. So, what's the problem? Again, let's look at the two different focus in Perl 6 development.

On the Parrot side, the project took a very hard challenge, which was to create from scratch a software that, at its first version, should be better than perl 5.10. This is an important challenge, in a way that if that is not taken, the project itself may seem pointless. A good analogy I like to make on that is comparing Parrot to GNU Hurd. GNU Hurd isn't out yet because they just can't fix things with hacks in it, simply because if they do it, hurd would simply be yet another operating system.

Parrot is today very powerfull and even in the first versions, it can already be used to a lot of stuff. But it still have a long way for hosting a full Perl 6.

On the language side, the specs are anything but stable, and the lack of a runtime that is actually capable of performing all the semantics the language creates, the concepts can't be tested, and therefore, can't be made stable.

The KP6 project have tested several possible backends, being the P5 the most successfull today. The KP6 tests looks very promising, with features like gather/take. Lazy array and hash autovivification.

On the other hand, the performance is absolutely unpractical, this implementation can only be used as a proof-of-concept. And, in terms of concept, it's fair to say that the concept do work.

Maybe for the first time, a real convergence between the Parrot and Pugs (actually KP6) "communities". There's an effort of porting the features of KP6 to Perl-6-on-Parrot and stablish a cooperation in the two projects. I hope this leads us to a viable short term implementation, but unfortunally, I have to say that we still have a long way to go.

Another approach.

When I and fglock were in the plane to vienna for YAPC::EU::2007, we were discussing how could we make KP6 faster, and at some point we considered and started sketching some structures in C. And at that point I considered the option of creating a as-simple-as-possible runtime in C to run KP6, and that was called YAP6.

During YAPC::EU, I had the opportunity for a very intesting debate with the parrot team about one of the most dense concepts of Perl 6, which is the inter-language interoperability. We didn't came to an agreement on this matter, and I started to become more skeptical about this specific issue, because I realised that in one way or another a typemapping would be necessary. And that typemapping would not be that different from what XS is today. This put my eyes back on how p5 works, and that's when I started taking yap6 seriously.

YAP6 today already implements some types, as the tests show, and it's really near having a practical use as a KP6 backend.

The idea is simple: Let's get the simplest C runtime that can run Perl 6 contexts and requirements and build a interpreter just like perl 5, but with the data structures following the ideas that had grown in KP6.

The main difference between perl 5 and yap6 is the concept of "responder interface" (those who saw nothingmuch's talk in YAPC::EU are familiar with that concept). YAP6 uses DISPATCHERs (as yap6 calls responder interfaces) for absolutely every type. And, like perl 5, implements a set of binary-compatible C structures resambling object orientation to interact with the data. Instead of SV* you have YAP6__CORE__Value*, instead of AV* you have YAP6__CORE__List* and so on.

So, what now?

Well, this post is actually a call for help. If you can program C, if you have any knowledge on the perlguts you certainly can help delivering the version 0.0.2 of YAP6 which will implement the basic runtime on top of which the interpreter in itself will be implemented.

The project is hosted in the pugs svn http://svn.pugscode.org/pugs/v6/yap6/, the README and the ROADMAP documents are a good guide to understand the project. The test files (all the t_*.c files in src/) are a very good way to see the runtime library working. #perl6 at freenode is the place to discuss ideas and implementation. Anyone wanting to help just stop by and commit instructions will be handed.

As I said before, and as I say in the README file, this is not completely a competitor to Parrot as it have different goals, and I certainly think that both projects can happily coexist and share much of the experiences. And I do think that we can have a YAP6 1.0 version very soon.

daniel

Comment on YAP6: A p5 approach to p6
Re: YAP6: A p5 approach to p6
by flexo (Sexton) on Dec 24, 2007 at 19:45 UTC

    This may be ancient history, but I'm curious: why was the original decision made to implement Perl 6 with something fancy like Parrot instead of a simpler approach like YAP6 takes?

    In hindsight, would it have been wiser to just take the approach that YAP6 takes?

    I recently read a Steve Yegge blog post where he mentions actually liking Perl 6 (he's not exactly a Perl 5 fan), but lost interest 6 years ago. That's a very long time in internet years. Could Perl 6 have gotten here (or could it get here) faster if a simpler plan for the implementation had been chosen?

    Is it possible that Parrot is a bigger bite than anyone can actually chew, and it could be dropped in favor of a simpler Perl 5 -style approach that would get us a Perl 6 -only Perl 6 implementation much sooner than anyone expects? Or is that a nightmare scenario that would bring the community to rack and ruin?

      ... why was the original decision made to implement Perl 6 with something fancy like Parrot instead of a simpler approach like YAP6 takes?

      The people who started Parrot had a lot of experience working on the internals of Perl 5 and knew what it would take to support the features that Perl 6 needed in Perl 5: several miracles and tens of megabytes of compressed diffs to rewrite nearly all of the guts of Perl 5.

      Take just one feature, for example: B::Bytecode. That's never worked reliably in Perl 5, and Perl 6 strongly suggests that it should work. The corresponding feature has worked in Parrot for years. No one's hacked on the version in Perl 5 for years, and that's not because it's impossible to work on, but no one with the knowledge to make it work has wanted to make it work.

      Could Perl 6 have gotten here (or could it get here) faster if a simpler plan for the implementation had been chosen?

      Who knows?

      ... it could be dropped in favor of a simpler Perl 5 -style approach...

      Did you intend to call Perl 5's internals simple? It's not clear to me how to implement several features of Perl 6 in an efficient fashion on top of Perl 5, including: reflection, method dispatch and redispatch, parallelism, junctions, concurrency, asynchronous IO, continuations, pervasive closures, and modifiable grammars.

      That doesn't mean it's not possible at all, nor that it's not possible to do such things efficiently, but it's my opinion targeting Perl 5 as a VM would, if it had even worked by now, have resulted in a very stripped down Perl 6 with a feature set roughly on par with Perl 5 right now running much more slowly than Perl 5.

      If instead you meant to ask "Why not target an existing virtual machine?" then please see the Parrot FAQ.

      Update: Upon re-reading, I realize you may have asked "Why build a generic virtual machine optimized for handling dynamic languages (Parrot) instead of a specific virtual machine optimized for handling Perl 6?" That's a slightly different question.

      I suspect but can't prove right now that if YAP6 evolves to be able to handle all of Perl 6, it will resemble Parrot in a lot of ways. It may be simpler in some ways because it avoids some of the customization points necessary to run other languages, but there really aren't that many; Perl 6 is a superset of a lot of language features, and it offers a lot of customization internally to mutate the language into plenty of pidgins.

        Chromatic, the question you rephrased in the update to your post is indeed exactly what I meant to ask. Sorry if I was unclear, and thanks for your answer.

Re: YAP6: A p5 approach to p6
by dragonchild (Archbishop) on Dec 25, 2007 at 03:54 UTC
    chromatic has made a couple of good points. But, I think he has missed the most important point of all and the very reason why anything other than Parrot will, more than likely, be unable to run Perl6 in all it's glory.

    Perl6 isn't a language. It's a runtime lexically-scoped grammar engine. There is a default language being defined and that is being used to drive the vast majority of features that the VM will need to be able to provide. But, and TimToady has said this very quietly on a number of occasions, CP6AN is expected to contain a number modules that will extend (and reduce) the grammar in a lexical fashion. He has even gone so far as to say that Perl6 is really a family of languages with a primary language that everyone generally talks.

    Let me drive it home - the Inline:: modules will, in most cases, be implemented as grammars on top of Parrot. They will not, as is the case right now, be implemented as call-outs to other compilers. Will YAP6 be able to handle that?


    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
      Will YAP6 be able to handle that?

      Nothing precludes it, technically speaking. However, it is the source of a fair bit of complexity within Parrot and PGE, and I think a lot of that complexity is necessary to the problem.

        As I said in the reply to dragonchild, YAP6 should inherit the grammar engine from KP6, and this way, support all this features.

        daniel

      In fact, that is one of the requirements for Perl 6 that I started neglecting some time ago. As I explain in the README file, I still can't see how the interoperability between different languages will work without a typemapping layer. This typemapping layer will be something very similar to what XS is today, except that in a higher-level language.

      This means that the usual way for language interoperability in YAP6 will be the same as p5. But it doesn't mean, however, that yap6 won't be able to handle grammars. YAP6 is a port to C of the concepts created and tested by KP6. And it's in the plans the use of YAP6 as a backend to KP6, which means that it should support anything KP6 does, which includes the KP6 grammar engine.

      In summary, I don't expect high-level languages like Perl 6 and Perl 5 interoperating in the high level directly, I do expect each code to be running in each interpreter interfaced by a typemapping layer. But this doesn't means that Perl 6 based sub-languages and DSLs (domain specific languages) won't be able to run inside yap6 (as long as no typemapping is needed).

      daniel
        You're still missing the point. Perl6 isn't a language - it's a grammar engine. Which means that it will be parsing N languages to the same parse trees to be interpreted on the same VM. Without a typemapping layer. There are no Perl6-based sublanguages. Everything you can express in the grammar engine that runs Perl6 will be a Perl6 language. Even the default language.

        My criteria for good software:
        1. Does it work?
        2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

      It's interesting to me that there seems to be this "holy trinity" of major Perl 6 features (that may even interest different parties):

      • The new language syntax (I'm looking forward to this)
      • The buttons and knobs that will allow you to remake the language into whatever you want it to be (language connoiseurs, or others looking to guide the evolution of Perl 6)
      • having a generic vm that can run multiple languages (I think this may include being able to access libraries written in language A from your program written in language B) (would be nice for many, if it were possible, but Perl 5 seems to have done fine without this ability).

      Is Perl 6 only "Perl 6" if you get all three of those?

Re: YAP6: A p5 approach to p6
by Erez (Curate) on Dec 25, 2007 at 08:17 UTC

    Parrot is today very powerfull <sic> and even in the first versions, it can already be used to a lot of stuff. But it still have a long way for hosting a full Perl 6.

    I may be wrong here, but isn't the issue with hosting a "full Perl 6" the fact that Perl 6 isn't finished? From my understanding, both Parrot and Pugs are implementation of what currently exists of the language, and are used for testing, not as what "perl6" will be. So the Perl 6 development isn't centred around those two projects, and a third one won't help Perl 6 to be released any sooner.

    Software speaks in tongues of man; I debug, therefore I code.

      There are two different things, Perl 6 the language and Perl 6 the runtime (will we call it Perl 6 and perl 6? maybe not)... The Language is already a comprehensive set of features and grammar specs (even if not stable), and no runtime (Pugs, Parrot or KP6) can run the full of the specs.

      daniel
Re: YAP6: A p5 approach to p6
by BrowserUk (Pope) on Dec 25, 2007 at 10:18 UTC

    Aren't you in danger of just further dilluting the gene pool?

    The thing that has always amazed me each time I've downloaded Parrot is that it appears to me that about 50% of the contents, and perhaps a far greater percentage of the actual effort expended, seems to be on what you might loosely term "ancilliary" developments rather than the core.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Aren't you in danger of just further dilluting the gene pool?

      Certainly... but I really don't see it as a bad thing. Each one of the efforts that tries to bring Perl 6 to life gets it from a different perspective, and the experiences from each of this projects are creating a more powerfull knowledge of the Language and the runtime requirements for it.

      Eventually all this projects will implement the full Perl 6 at the same time, even because most of the people working in this projects knows each other and do share experiences.

      And finally, the idea of having alternative runtimes is interesting. Let me bring one considerable difference from Parrot to YAP6. Parrot implements trace gc and YAP6 implements refcount gc. Trace gc may be more efficient and more safe, but is not deterministic. Some applications may need a more deterministic approach to memory use versus processing time.

      daniel
        Parrot implements trace gc and YAP6 implements refcount gc. Trace gc may be more efficient and more safe, but is not deterministic.

        There are ways to add determinism to tracing GCs. Escape analysis is useful for this.

      BrowserUk,
      This has concerned me in the past. I have followed parrot from afar for a number of years. The thing in my lay, and completely unqualified, perspective that parrot should have done different is target a HLL from day 1. This may not seem pertinent to your point right away, but hear me out.

      A great deal of would-be contributors to the project aren't interested in C, PASM, or PIR. A high level language would have opened up the doors to many more people - even if all they did was write code that exposed bugs that a much smaller population is capable of fixing. I think it would have done much more than that though. Back in the days of Dan and Leo squabbling over optimizations and features, one argument that came up was the applicability in a real language. If there had been one with a largeish code base, it would have been much easier to take things out of the realm of theory. I could go on why I think this would have been a good thing, but what do I know (really).

      Ok, so far this still doesn't seem like it is on point. One of the things I have seen over the years of non-core contribution is numerous languages targeting parrot. By and large, each language had one or two individuals and the project would lose steam

      • Questions to the list would go warnocked
      • They would be blocking on a bug
      • They would run out of time (summer break)
      • They would lose interest for whatever reason
      • The internals made continuing undesireable
      • etc, etc, etc

      What I have been noticing lately though is that the folks who have stuck around are now being very collaborative. The pugs folks are working with the parrot folks. The tools available now make contributing much easier. Also, the folks working on language implementation are also core hackers. When something is blocking for them - they can make sure it stays visible. Perhaps I am being too optimistic or naive, but I really think that what was once a problem (tangential projects) is actually becoming a great asset.

      Update: I failed to mention that I doubt YAP6 is necessary. If it can produce a mostly working perl 6 implementation in a short time frame that is at least as fast as perl 5 - great. I suspect that this is not the case. I guess what I am saying is not all non-core projects are bad - but, for me, the jury is out on YAP6.

      Cheers - L~R

      Minor grammar corrections
        The thing in my lay, and completely unqualified, perspective that parrot should have done different is target a HLL from day 1.

        I agree, but remember that most of the reason that we're making such progress on tangential projects right now is that we have the Parrot compiler tools featureful and stable enough to allow us to build or port languages quickly.

        I failed to mention that I doubt YAP6 is necessary.

        It may or may not succeed, but the people who are working on it might not work on Parrot or vice versa. Concurrent evolution and exploration might not be the most efficient use of global resources, but there's rarely a central planning agency that can convince everyone to go in the same direction.

        failed to mention that I doubt YAP6 is necessary. If it can produce a mostly working perl 6 implementation in a short time frame that is at least as fast as perl 5 - great. I suspect that this is not the case. I guess what I am saying is not all non-core projects are bad - but, for me, the jury is out on YAP6.

        You know, that is exactly the point. Before I started thinking about YAP6, I was working to bootstrap KP6, as to release it from MP6 and make it self-hosting. But when I finally got a bootstrapped compiler, it was taking like 20 seconds to compile a simple "say 'Hello World'" line. That was when I realised that the problem was on the fact that KP6 was implementing a runtime enviroment on top of perl 5, which is by itself already a very heavy runtime, and when you stack them, the performance is absolutely unpractical.

        At that point I started to take a deeper look at Parrot, that was also during YAPC::EU::2007, while I had a very good chatting with the parrot folks, and that was when I realised that Parrot has taken a very hard challenge, and that IMVHO, if I wanted something to happen faster, I would have to take a simpler challenge. And that's why YAP6 started in the first place. YAP6 is nothing but a port of the KP6 "object system" to C, as so instead of generating weird perl 5 code, KP6 could generate weird C code that would run faster. Of course generating PIR was a possibility, but (and this is completely my fault), I just couldn't get out with PIR and parrot internals requires skills that I don't have.

        As the ROADMAP and the README says, YAP6 intends simply to be a runtime library for KP6 to be built upon. It is simply the KP6 object system translated to C.

        daniel

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://658902]
Approved by Corion
Front-paged by TStanley
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (12)
As of 2014-10-22 07:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (114 votes), past polls