Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

Re: Slow evolution of Perl = Perl is a closed Word (NQP, parrot concurrency == Oh dear.)

by erroneousBollock (Curate)
on Sep 10, 2007 at 10:21 UTC ( [id://638023]=note: print w/replies, xml ) Need Help??


in reply to Slow evolution of Perl = Perl is a closed Word

Please excuse me, but I'm going to quote from two different sub-threads here, just to get my response in once place... sorry if that's confusing. Re-depth was getting rediculous so....

RE: NQP.

BrowserUK wrote:

I take that to mean, that despite the context and the second half of the sentence, the "prototype" mentioned is some kind of Perl 6 language prototype, and not any kind of Parrot threading prototype.
NQP is "just another language on top of parrot".... like TCL or basic. It's an extremely small subset of perl6 with much a stricter grammar. If I'm reading the meeting minutes correctly it serves a few purposes:
  • something a little more high-level for HLL implementors to write compilers with
  • a testing ground to ensure perl6 features map nicely onto parrot's APIs
  • a bootstrapping stage of the "perl6 compiler written in perl6".

RE: parrot concurrency

That leaves just the existing PPD ... is based upon the Perl 5 iThreads model;
Are we reading the same document?

The PDD describes internal and external (HLL) constructs required for concurrency. They mention two distinct HLL APIs: "shared everything", and "shared nothing". (They're also supporting "completely independent" threads).

The PDD says that the schedulers will manipulate Task PMCs (which describe priorities, constraints, and keep stats?) in order to co-ordinate interpreters and interpreter pools.

User-level PMCs will be the standard PMCs in "shared nothing"; PMCs will have their vtable replaced in "shared everything" such that (internal, ordered) locking of parameters is enforced for PMC method calls.

It looks like they're only sharing PMCs (not small enough for hardware-level atomicity but small enough to be done efficiently)... not externs or strings of bytes. That would imply that HLL compilers organise marshalling of user-level data-structures.

If I read it correctly, a pool of "shared everything" threads will be implemented as a pool of "shared interpreters" implemented in a lightweight manner over plain POSIX threads. Different vtable methods on the (shared) PMCs will support access to either pool-wide or thread-local PMCs.

In the "shared nothing" model they seem to be saying that they're going to implement it on top of the "events" APIs they've marked out in an earlier PDD.

It says that there will be thread-specific and (interpreter) pool-specific event queues and that events will be guaranteed to post in the target thread (and in the "correct" order). At this level what they describe sounds a lot like the way Erlang implements its message-passing for lightweight processes.

[the PDD?] was written by people who had never used that model; makes no concessions to the obvious failures of the only existing implementation of that model; seeks to perpectuate all the mal-features of that model
Well I can't speak as to the "who", but I just don't see how you get to that conclusion. The features described in the PDD are very low-level and could be used in a great many ways by HLLs (such as perl6, tcl and Python). I can't find one place where they (at parrotcode... or the perl6 design team for that matter) say that they're going (overall) for anything like iThreads; rather I see plenty of examples to the contrary.

In the parrot concurrency PDD it says they must "support" something like iThreads for HLLs, but in reading about the APIs they're providing it seems that an iThreads-like model is only one of many possible HLL concurrency models you can build on parrot's APIs.

As an example iThreads could be implemented by an HLL compiler writer (say for per5-on-perl6) with the "shared nothing" parrot APIs. It (like perl5's threads.pm) would need to copy non-shared data to remain API-compatible.

That doesn't stop any HLL designer from building a myriad of other concurrency models over either (or both) of the parrot -supported concurrency models. An HLL compiler could provide POSIX-like, Erlang-like, Java-like or invent-your-own models of concurrency. IIUC it looks like a very broad-minded design.

[concurrency?] will have to be tack-on to the Parrot single tasking architecture after the fact just as the existing implementation was.
I also don't get that impression at all. It seems to me like they're creating a very broad set of possibilities for a variety of HLL compilers (perl6 not being special). I just can't understand how you can say that given the discussion in the PDD.

I'm going to have a closer look at the group archives and meeting minutes to gauge the progress for myself, and to see if anyone's talking about straying from what they've talk about in the PDDs... but on the face of it, I don't see any reason to be pessimistic about it.

I don't see why the parrot team shouldn't continue to work on parrot in an order that seems appropriate (to the parrot developers) while keeping in mind the constraints and decisions implied in the concurrency PDD. It's something that I do all the time in projects, and something which (as you mentioned) can be greatly aided by regression testing.

perl6 concurrency:

Now here's where I start to share your concern (but probably coming from a different tack). The perl6 concurrency DRAFT spec is at an extremely early stage of progress. One could argue (as you have) that it's extremely important to get it correct from the beginning. I don't know what steps have been taken so far by the perl6 team (as distinct from the parrot team).

I for one am concerned about the focus on STM (but that may turn out to be a non-issue... I'll wait to see what Elizabeth et al come up with). I'd wager that there'll be many, many Perlmonks meditations and questions on perl6 concurrency as the implementation starts to fill out... I believe we'll have ample opportunity to discuss that progress at a later date.

-David

Replies are listed 'Best First'.
Re^2: Slow evolution of Perl = Perl is a closed Word (NQP, parrot concurrency == Oh dear.)
by TimToady (Parson) on Sep 11, 2007 at 00:50 UTC
    From the standpoint of Perl 6 (the language), we've thought about a lot of these things from the standpoint of fundamentals. What you find in the draft concurrency spec is more like a laundry list of things people would like. However, part of the reason we've not bothered much with that document is precisely because concurrency must be built into the design from the ground up, not because we're ignoring the topic. Hence, just off the top of my head, in the basic design of Perl 6 you will find various reassuring tendencies:
    • All blocks are (potentially) closures
    • Data tends to be well-typed even if the user doesn't realize it
    • Emphasis is on reentrancy via use of lexicals rather than globals
    • Most Perl 5 globals are gone or are no longer global
    • The global footprint of a Perl 6 thread can be a lot lighter than a Perl 5 interpreter
    • Globals and other strangely scoped variables must be syntactically distinguished with twigils
    • Strictness is now default, so no accidental globals
    • Mutable vs immutable data is semantically distinguished
    • Default of formal parameters is readonly
    • State variables are explicitly declared and clone as lexical variables do
    • Tied variables must be explicitly declared
    • Context lexicals encouraged over temporized globals
    • Most objects stored in opaque storage
    • Mutating methods are syntactically distinguished
    • Symbolic vs hard reference syntactically distinguished
    • Constants are known to be constant at compile time
    • Hyperops, junctions, and feeds all make specific promises about lack of interactions
    • Exceptions may easily be treated as undefined data when doing parallel operations
    • Lazy lists encourage reentrant programming
    • Patterns are reentrant so they can be used in multiple threads on different strings simultaneously
    • Grammars and other logic programs are designed to be allow multiple "in flight" hypotheses in parallel
    These bits of design all contribute to easier concurrency, but are spread out over all the existing synopses, and most of them are not explicitly labeled as "concurrency support". That is not to say that there are not still many open issues, and as you say, we'll have ample opportunity to discuss them.

    Specifically with regard to STM, while STM doesn't itself address the issue of non-reversible external operations, I think it does potentially address the issue of acquiring the locks in preparation for doing such an operation, since the locks themselves are presumably in memory. And I suspect that STM can be extended to manage anything that can be construed as "memory" as long as it has reversible characteristics.

    Anyway, in many cases I think the compiler will have enough hints to determine whether the code is reentrant and whether the data has to be shared without forcing ordinary users into a monadic lifestyle. And where the compiler is not so sure, it can likely ask the programmer politely for a few more clues. In any case, the Perl 6 language is mutable and strongly versioned, so we can hopefully fix it where it goes wrong.

      For my part, I have had no concerns about the plans for concurrency in Perl 6, at the language level, going back a very long time.

      I don't follow how STM can be used to address locking of non-reversible external operations. There seems to be a total disconnect between the requirements and operational modes of the two:

      STM is optimistic: every thread completes its modifications to shared memory without regard for what other threads might be doing, recording every read and write that it makes in a log. Instead of placing the onus on the writer to make sure it does not adversely affect other operations in progress, it is placed on the reader, who after completing an entire transaction verifies that other threads have not concurrently made changes to memory that it accessed in the past. This final operation, in which the changes of a transaction are validated and, if validation is successful, made permanent, is called a commit.

      But even if it cannot, some other mechanism can be used. Provided that other mechanism is availabe to the language from the underlying runtime.

      My concerns have been, and remain, whether Parrot will have the wherewithal to underwrite the demands and operations that Perl 6 will call upon it to make.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
Re^2: Slow evolution of Perl = Perl is a closed Word (NQP, parrot concurrency == Oh dear.)
by BrowserUk (Patriarch) on Sep 10, 2007 at 13:35 UTC

    Thank you for taking my concerns seriously. As you can see, many do not.

    When I first read the PPD (18 months or so ago?), and when I was taking part in the background discussion from which it is in large part derived, 3 or 4 years ago, I too allowed myself unbridled optimism.

    Here, today, it is still not possible to start a second interpreter in parrot. And none of the code that has so far been written has been ever been tested for reentrancy. There is no visible segregation between memory that represents code and data at the VM level. There is no segrgation between local constructs and global constructs.

    Say, as the PPD does, that you are going to 'do everything', but make no provision to ensure the constraints that everything imposes upon the rest of the design, means that all of those constraints will be violated all over the code base and will only be discovered once you come to try and implement the vision.

    At that point, you are basically looking at re-writing everything; or the most likely scenario, deferring 'everything' "for now", with the promise of retrofitting it over future releases. Retro fitting threads into a mature architecture is exactly the problem, with exactly the effects that perl 5 iThreads has suffered.

    But, as the latest anonymonk denial shows, there continue to be a large percentage of persons that believe that my concerns for threading are simply because I'm 'stuck' on a broken platform; that thread is spelt F - O - R - K; that event driven architectures are a substitute for threading; and whilst that remains the case, there is little hope for effective threading on Parrot, and by extensions, those languages that run on top of it.

    But, now I am just hand-wringing, so I'll crawl back in my shell and shut up. Again.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re^2: Slow evolution of Perl = Perl is a closed Word (NQP, parrot concurrency == Oh dear.)
by BrowserUk (Patriarch) on Sep 10, 2007 at 15:08 UTC
    Are we reading the same document?

    There is what is contained in the PPD; and there is what is contained in the source files.

    For example: Here are a couple of comment blocks drawn from the latest Parrot sources:

    Create a local copy of the PMC if necessary. (No copy is made if it is marked shared.) This includes workarounds for Parrot_clone() not doing the Right Thing with subroutines (specifically, code segments aren't preserved and it is difficult to do so as long as Parrot_clone() depends on freezing).
    Fixup a PMC to be sharable. Right now, reassigns the vtable to one owned by some master interpreter, so the PMC can be safely reused after thread death. In the future the PMC returned might be different than the one passed, e.g., if we need to reallocate the PMC in a different interpreter.

    How do you interpret those?

    My interpretation is that

    1. Parrot generated code is not reentrant. It cannot be called from a multiple threads concurrently.
    2. Parrot data structures (PMCs) are allocated on a per thread (interpreter) basis, and have to be cloned (copied by freezing and thawing) in order to be shared.

    That is, as best as I can divine, repeating all the mistakes of the Perl 5 implementation.

    To understand the fundamental (and I do not use that word lightly here) importance of reentrancy to threading, read some wisdom from people who have done it. Pay particular attention to section 12.3.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      If you're looking for the problems of tacking on threads afterwards, you can also take a look at the Python Global Interpreter Lock (and the discussion around it). While I'm not a fan of Guido van Rossum's striving for speed for speeds sake, the discussion highlights the problems that you encounter when you don't plan for share-everything threading/lightweight threading/fibers/coroutines from the start.

      Of course, there remain the semantical problems of the global name space of Perl, especially what happens in a share-everything model with the following code:

      sub quote_string { qq{"\Q$_[0]\E"} }; sub send_to_client { my $client_socket = shift; async { print {$client_socket} quote_string($_) for @_; }; }; function send_strings_to_client { local *quote_string = sub { qq{'\Q$_[0]\E'} }; send_to_client(@_); };

      With a share-everything, it's not really clear how overwriting the globs should be handled. I would be for "overwriting (code) glob entries is discouraged" and there should/could be a strict 'thread-globs' pragma that prevents you from assigning to glob entries. This should prevent 95% of all those problems.

        Corion. Sorry for taking so long to get back to you, but I was hoping for some ... um ... something. Anything. Some reaction from those "in the know", to my earlier posts. Something to indicate that I was wrong about Parrot. But nothing.

        A monk, so confident in his appraisal that I "had done no research", that he posted his "just cos your on a broken platform" condemnation, anonymously.

        Back in the other part of this thread, before erroneousBollocks decided to lift his misunderstanding to a new level, chromatic made a suggestion that he would listen if I had something useful to say. When I asked him off-line if he was seriously inviting me to make a proposal, he opened his reply with..."I'm not a fan of threading but...".

        And in that single phrase he summed up the entire problem. Nobody developing Parrot believes that threading is necessary. And while that situation continues, and whilst those with the insight to see beyond the 22 year old straight jacket that is POSIX, to a future (actually now. I just saw an ad for a 4-core machine for less than $1000), in which multiple processors are ubiquitous, and threading is the only way to usefully exploit them, keep quiet...perl is doomed.

        In just the same way as the Perl experience for Win32 users will never improve, while the key players, (according to one recent post here, from TimToady on down), continue to use a lame unix emulation (Strawberry Perl) as their only experience of Perl on Win32.

        Read again, and weep.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
      I really sincerely hope this doesn't come off as sarcastic, but...

      How do you interpret those?
      Well... depends on surrounding context. :-) I've not read the source code, but I'll play anyway.

      Create a local copy of the PMC if necessary. (No copy is made if it is marked shared.) This includes workarounds for Parrot_clone() not doing the Right Thing with subroutines (specifically, code segments aren't preserved and it is difficult to do so as long as Parrot_clone() depends on freezing).

      Well on first glance that comment seems to hark back to iThreads' "copy unshared data by default" nastiness... but on second reading (without better context) they're talking about PMCs that represent user code.

      Fixup a PMC to be sharable. Right now, reassigns the vtable to one owned by some master interpreter, so the PMC can be safely reused after thread death.
      I think it's clear that this comment pertains to the "shared everything" model rather than the "shared nothing model". It also sounds like exactly what I'd do for a "shared interpreter pool" scenario in a "shared everything" model.

      In the future the PMC returned might be different than the one passed, e.g., if we need to reallocate the PMC in a different interpreter.
      I definitely cannot devine any meaning from that one... what specifically about those words gives you pause?

      In short, I definitely wouldn't interpret those comments (together or separately) to mean as you've concluded. I'd need much more context (and naturally to read the damn code) before I made any such conclusion.

      Actually, if I must interpret those comments as a whole (a silly exercise at best), I get the following:

      • Shared PMCs are never copied.
      • In the "shared everything" model, code that touches shared PMCs needs to be cloned into the offending thread.
      • They're thinking already about what's safe (or not) when considering caching of PMCs in interpreter pools.

      You see how I did that? ;) I got a completely different meaning from the same comments.

      Seriously though, it's not like there's a CMM traceability document tying specific parts of the parrot implementation to words in the TDD... the parrot development has been highly incremental, striving for specific goals at each stage. I can only go by the discussion I read in the group's threads.

      Larry (below) has put my mind at ease about perl6 wrt concurrency, so now I guess I'll just pay a bit more attention to the relevant groups to see what's really going on wrt concurrency in parrot.

      -David

        In the "shared everything" model, code that touches shared PMCs needs to be cloned into the offending thread.

        Ask yourself why? Why does anything need to be cloned? There is only one reason. Fork emulation.

        But if you have a real fork, you don't need to do that.

        And if you don't have a real fork, the fork emulation benefits nobody, because you do not have, and cannot provide, all the other bits that go with it.

        Signals for instance. Sure, Parrot could attempt to emulate signals as perl 5 has done, but it will only ever work for other parrot processes. You won't get a SIGCHLD when a non-perl child terminates because the OS doesn't do that. And that's just the tip of the iceberg.

        Fork emulation just doesn't work, and perpectuating it serves no good purpose.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://638023]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2024-03-19 09:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found