Re^3: what the history behind perl not having "real" threads

[dave_the_m]

Is this required by the language

There's nothing in the language that precludes a 5.0005-style threading implementation. The difficulty was in retrospectively trying to make the existing implementation thread-safe, where it had never been designed for that possibility. This is one of the (many) reasons why it was concluded that a complete from-the-ground-up rewrite of the perl interpreter was required, i.e. perl6.

The main drawbacks of the ithreads model are: that cloning the existing interpreter when creating a new thread is slow; that it uses lots of memory, since the new interpreter doesn't make any use of the OS facilities that a fork() would, of sharing memory by default with copy-on-write pages; and that having shared variables is slow, clunky and is memory-heavy.

Dave.

[perl-diddler]

Dave m wrote:

There's nothing in the language that precludes a 5.0005-style threading implementation.

So that print "$x", currently does modify $x to be a string, that might safely be called an implementation detail that wouldn't need to be kept for compatibility reasons.

I think it's a shame that they gave up on the 5.0005 style threading. The linux kernel didn't become SMP safe or capable overnight. It started out with no SMP, and for a long time, lived with the 'big lock model', where user-land code could be multi-threaded, but by-and-large, the kernel was not. Going from 2.0->2.2-2.4 were long steps...and it took alot of developer education to go from 1 big lock to many smaller locks, and in many cases, non-locking models to reduce bus contention and going higher ordered algorithms to ones that approach O(1).

As near as I can tell, it's an ongoing process. Certainly a redesign of the language could make the process easier, but I have no idea if there was a brick wall, or if some people were too risk aversive to live with something that was in constant evolution.

From an end user-perspective, I never heard about user-level programs needing complete rewrites due to language changes, but at the driver level things were less stable -- not exactly chaotic, but certainly requiring ongoing work.

[dave_the_m]

I think it's a shame that they gave up on the 5.0005 style threading

Well, lots of very smart people decided it was impossible to make work safely.

Note that the linux kernel is a different beast. It has a few well-defined entry points, and it is (relatively) easy to make concurrency initially fairly restrictive, and gradually increase it as code is audited and reworked. With the perl internals, you have XS code that can more or less do anything to anything in memory. Every single thing in the perl internals (and XS code) must be completely thread-safe before you can allow threads to take charge. (Ok I'm simplifying a bit, but you get the general idea.)

Anyway, I'm bored of discussing this now,

Dave.

[perl-diddler]

Dave M yawned, and wrote:

Anyway, I'm bored of discussing this now,

Don't feel a need to respond but something to think about -- device drivers in the linux kernel as well as modules (many of which can be built in statically or loaded dynamically), all had to be thread safe to be usable in the new kernel.

You made an interesting point that all of the 'XS' code (i.e. the binary parts of the modules) had to be thread safe before the perl could allow threads to run. Perhaps it isn't as important, but in the kernel, it was the responsibility of the 'modules' to be thread safe in an SMP kernel, the kernel going 'SMP', wasn't held up going forward unless those modules were an essential part of the core.

It seemed, essentially that it was the responsibility of the modules to become thread safe and follow the lead of the kernel than for the kernel to wait for all the modules to become 'thread' safe. If the latter had been the case, SMP, likely, would never have happened. Dunno if that's what happened w/perl or not, but there does seem to be an inordinate amount of pressure to maintain legacy compatibility at the expense of forward progress of the language.

This has so much been the case that 1) it became required for "forward progress" to become a separate, new project, and 2) has split development resources sufficiently to create questions of sustainability of the language, besides slow the development of the language.

The latter has, IMO, at least in part, been responsible for some not-insignificant amount of defections to other languages even though some of those other languages have gone through significant growing pains (such as incompatible releases (python)).

However, it's unlikely that any other course would have been possible in the perl community. Can you really see people with strong opinions, like Linus, surviving long in the p5p community?