Re: [JOB] The Perl Foundation seeks Windows Developer

[BrowserUk]

I keep reading this, and I'm still lost for an explanation of what the Vanilla and Strawberry projects are aiming to achieve?

As a Win32 Perl user,

1. I can already build my own perl

   I actively sic choose to use Active State Perl for various reasons.

   • They've already solved most of the problems.
   • They've added value in the form of all the must have modules they bundle with their distributions.
   • They've added value in the form of their html-ised documentation sets which are 1000% more usable that perldoc.
   • They've added value in the form of the PPM manager.

     Whilst I can build most modules myself, having a quick, simple and convenient 'just do it' install for the more complex or tricky modules is a huge time saver.

     Why expend time and energy re-discovering all the problems and solutions to installing packages on win32 for every installation, when the problems have already been solved and are available for one-step installation?

2. I can already build my own modules, with or without the use of CPAN.pm

   I can use the CPAN shell. I choose not to, because it carries with it so much baggage that simply does nothing for me.

   For the vast majority of modules, I can manually download, build, test and install them--including 1 or 2 small dependencies before the CPAN shell has finished configuring itself, which it seems to need to do every time?

On those occasions I fail to be able to build a module, (most recently Devel::FastProf), the root cause seems to be much more to do with the way the authors write their modules to make use of inherently *nix-only features of the Perl source code, or by directly accessing parts of the Perl runtime bypassing the good work done the developers to isolate OS dependencies by virtualising them.

I guess what I am saying is, I certainly have the time, and (some of), the skills that would allow me to contribute to this project, but I am at a loss to understand what it is that you envisage that I, and other win32 users, would gain from this effort?

Also, what does the overall Perl community gain from the TPF funding this project?

The bottom line is, you are asking for someone to apply for a TPF grant, and therefore raise a grant application, but it's not at all clear to me what the objectives of the project are? Who is it intended to help and how?

I also think that a much clearer brief is required. In this short thread we've already seen that there is more than one opinion as to the direction that should be pursued. Your brief justification of the motivation for Vanilla & Strawberry

The aim is to provide CPAN authors a Win32 Perl platform that is similar in use and style to the Unix Perls, to make it easier for them to port to Win32, and to provide a suitable basic Perl platform for experienced Perl developers that prefer to install from CPAN, instead of using binary packages

Leaves me uneasy that it wouldn't benefit Win32 Perl users much. Despite the almost obligatory "I don't/only/have to use Win32" disclaimer, the biggest problem facing Win32 Perl users is that the vast majority of Perl developers don't want to develop on Win32, and this project sounds like yet another attempt to make win32 look as much like *nix as possible to allow reluctant but willing CPAN authors to more easily test and bug fix their modules on the win32 platform. Maybe that is the right way to go, but based on my own limited experience of trying to build the 'awkward squad' of modules on Win32, the problems go deeper than the compiler, CRT or build utilities.

As an example of what I mean, the problem I encountered with Devel::FastProf was that although the MSVC CRT has perfectly normal C getc() routine, once you've bypassed various errors to do with simple unixisms, the base Perl headers files are converting the getc() used in the source, to a call to

FastProf.xs(97) : warning C4013: 'PerlSIO_getc' undefined; assuming ex
+tern returning int

And the PerlIO/SIO subsystem doesn't appear to provide this routine. I've tried to unwind the macros to work out why getc() is redefined as PerlSIO_getc(), when that routine does not appear to exist anywhere in the entire Perl source tree, not just the Win32 part, without success.

I imagine that any *nix programmer that encountered this problem when trying to port their module to Win32 would probably take one look, blame it on a broken MS C compiler/CRT and understandably give up, but the problem lies within the the perl sources. Does anyone anywhere use the whole PerlSIO subsystem? What was the purpose of that layer of headers files? And was it ever achieved/completed?

And this is not an isolated incident. Many of the other problems of compiling code written to run on GCC with the MS compiler come down to compiler settings. Things like declaring for loop variables inline, which gcc allows even though it isn't part of the C standard, have to be explicitly enabled using the MS compilers.

My point is, I am not at all sure that simply moving to a perl distribution that builds with gcc is going to fix very much, or help Win32-reluctant module developers to more easily port their modules. And unless Strawberry also included all, (or at least most), of the value added features that come with the AS distributions, then Win32 Perl users, (as opposed to developers), are going to be very reluctant to adopt it. It would be of little benefit to anyone if the only people that made use of Vanilla/Strawberry Perl were only developers looking to simplify the porting of their modules to Win32, because it is quite likely that the results of their efforts would not work for the majority of Win32 users.

Please don't misunderstand me, I would hate to be seen as dismissive of any attempt to make Perl more Win32 friendly. Indeed, I would love to be a part of such an effort and dog knows if there was a little income to be earnt from such efforts it certainly would not go amiss. My fear is that, as I understand the direction outlined here, (which of course my be a completely different thing from the intent), it may not, even probably would not, do much to help in this regard, and that any monies that the TPF feels are justified in making Perl more Win32 friendly/complete might be better directed.

Maybe Strawberry is only the starting point and would enable/encourage more people to become actively involved in making Perl work better on Win32? I think I am the wrong person to make that judgement and so I would encourage an expanded discussion of the overall goals and the best way of achieving them, to help convince me, and others, that this is the right way to go.

[xdg]

Also, what does the overall Perl community gain from the TPF funding this project?

In short: diversity and choice.

I think the Perl community gains by opening up Perl usage on Windows to a wider audience. Until now, Perl on Win32 has really meant two choices: Cygwin or ActiveState, both of which bring a lot of baggage along with their benefits.

One of the downsides of relying on, for example, ActiveState, is that while they solve some problems for you, you wind up held hostage to choices that they make. One of the goals of CamelPack and VanillaPerl, etc. is to make having your own Perl distribution on Win32 as easy as it is on unix-based platforms -- with direct reliance on CPAN rather than intermediate repositories to allow for access to the most up to date modules available.

Here's a real-life example: Scalar::Util. ActiveState's PPM repository targets the first ActiveState release for a particular major version of Perl to maximize backwards compatibility and (presumably) minimize support costs. Because the version of Scalar::Util frozen into the first ActiveState Perl 8xx release didn't include the critical refaddr function, every CPAN module that requires a Scalar::Util with refaddr fails the automated build process for 8xx PPMS -- even though later releases of ActiveState Perl 8xx include more modern versions of Scalar::Util. That's a lot of modules -- including my own Class::InsideOut.

With VanillaPerl, I and other users don't have to rely upon PPM repositories -- they just use CPAN directly and can rely on the dependency management code already built into the CPAN toolchain. And, because a compiler is bundled (unlike with ActiveState), that includes XS modules.

I certainly don't think that's for everyone. Many end users may be perfectly happy with a PPM-dependent approach. But I think TPF and the community will benefit by creating more choices.

[BrowserUk]

I just went to CPAN and pulled Scalar::Util. It took less than 10 seconds to download via my dial-up connection. And doing the prayer-in-four-parts took almost exactly the same amount of time. And that included me mis-typing the name once. There is no reason why AS' autobuild process couldn't download the latest/required version of module dependancies. That's a procedural problem for AS, not a problem for users.

It sucks that perfectly workable modules are getting flagged as unbuildable by the AS process, especially for want of the latest version of a module, (in this case developed by an AS developer?), that they already know builds correctly as they ship it with later versions of their own distributions.

However, the version I just downloaded and built was Scalar::Util v1.17. The latest version available on cpan is v1.18. This fails to build under AS 5.8.6 because of an unresolved external

   Creating library blib\arch\auto\List\Util\Util.lib and object blib\
+arch\auto\List\Util\Util.exp
Util.obj : error LNK2019: unresolved external symbol _Perl_seed refere
+nced in function _XS_List__Util_shuffle
blib\arch\auto\List\Util\Util.dll : fatal error LNK1120: 1 unresolved 
+externals

And that comes down to my having 5.8.6 rather the 5.8.8. A change 26054, was made to Scalar::Util::shuffle to address a problem whereby dprof would "break" shuffle. That change included calling Perl_seed(), which (apparently) was not previously exported. So change 26836 was required to export Perl_seed and fix the linker error.

My point is, this is standard version control ripples and I don't see how that would be fixed by using a different compiler?

Vanilla/Strawberry Perl may make it easier and simpler for developers to get set up to test code on Win32, but unless the vast majority of users also move to using it, the only benefit woudl be in the statistics of successful Win32 builds--which would be pretty useless unless users can benefit. It could even make things worse, as I could see the standard response to questions about build problems of packages on win32 being "it works with Vanilla Perl, so it must be Active State that is broken", which wouldn't help many users.

[xdg]

I could see the standard response to questions about build problems of packages on win32 being "it works with Vanilla Perl, so it must be Active State that is broken", which wouldn't help many users.

It wouldn't help many users of ActiveState Perl, you mean. If ActiveState is broken and something else works, isn't that an improvement for users in general? Maybe some friendly co-opetition would improve ActiveState, too.

As to the Scalar::Util thing -- it has more to do with the fact that ActiveState's auto-build procedure uses their earliest 5.8 Perl release as the the basis for PPM generation without any upgrades to any modules in core. 8xx PPM's are supposed to be ones that will work with the earliest 5.8 Perl released by ActiveState -- which was several years ago. So anyone depending on ActiveState-built PPM's has to accept a 3-year lag in compatibility.

It's compounded by the fact that Perl can't easily replace core modules on Windows if they are in-use. demerphq has recently patched ExtUtils::Install to use the Win32 API to do this on Windows (and the old files are removed at the next reboot). That's great news for VanillaPerl, but doesn't help ActiveState as, again, they're stuck with ExtUtils::Install from 5.8.0.

The inability to rely upon any upgrades to core modules in 5.8 is a major flaw in my opinion. I'm sure there have been substantial bug-fixes to numerous core modules from 5.8.0 to 5.8.8. But any distribution that depends on certain behavior of upgraded modules will fail a dependency version requirements checks for building a PPM or may not behavior properly if the dependency version is ignored.

A good example of that is File::Spec and company. Look at the Changes file and look at how many Win32 bug fixes there have been in the last year alone. There are a lot. Any any code which relies upon those bugs being fixed can't require a modern File::Spec and still pass ActiveState's PPM build process.

The point is that this is not "standard version control ripples" as you put it. This is a fundamental difference in the philosophy of module packaging, distribution and support. Both are valid and they just serve different "customers" who have different wants.

[adamk]

There is no reason why AS' autobuild process couldn't download the latest/required version of module dependancies. That's a procedural problem for AS, not a problem for users.

You assert there is no reason, and yet here we are without AS autobuild support for vast quantities of things. It is a problem for users because AS are the defacto gatekeepers for what is and is not supported on Windows. Lets look at just the CPAN packages that start with "P".

http://ppm.activestate.com/BuildStatus/5.8-P.html

Vast swaths of modules no longer work in the autobuild system, the grand irony being that Windows is the platform with the WORST support in the PPM autobuild system, and Mac OS X being the best, when one would think given their relative userbases, Windows would be the best supported.

Looking at some common things from that list...

PAR does not work, PDL does not work, PGP does not work, my new PITA testing project does not work, POE (which has NO non-core deps by default) does not work, PPI (which has almost no non-core deps) does not work, and thus most of Perl:: does not work, Plucene does not work, POSY doesn't, Process doesn't, Perl-GPS doesn't, plus dozens of less well used modules.

Many of these use plain ordinary pure Perl code with proper platform-neutral File::Spec usage.

Although it may APPEAR there is no reason, there most certainly is. And despite efforts to fix the autobuild system, it is increasingly appearing that these are systematic faults, inherent to the design of the PPM system itself, and may well be unfixable without breaking back-compatibility.

I and others have raised this problem a number of times, via the online forms, emails, and even in person at OSCON. I've spent a year in the rediculous position that as the AUTHOR of PPI I cannot install it on my own Windows laptop so I can do demonstrations during my PPI talk.

Or at least, I cannot without a vast amount of work, installing the compiler and make myself, then downloading the entire chain of 20 distributions manually and hand-installing all of them.

The point of installers is to install things.

And as long as everything Just Worked, the PPM system was a perfectly acceptable solution.

But despite gozer's and other people's efforts, it has now reached a point where the problems have become too much for me. When something like this goes wrong, I am in a position where there is nothing at all I can do to fix the problem, other than to bitch to ActiveState. And I hate to bitch about things which I can't fix myself if needed.

So if there are prodedural problems inside of ActiveState PPM system that I cannot fix, that even THEY might not be able to fix, then it is clear to me that what the Perl community needs is a way to at least have the OPTION to help itself in these situations.

The CamelPack has helped solve the short term problem by at least making it easy to set up a compiler and a make with AP.

Vanilla has provided an excellent research distribution so we can look at Win32 problems very close to and in the Perl core, and look at problems in the toolchain without creating an expectation that it is useful for authors or developers.

Strawberry is the next step, and is about making a minimal but complete and bug free distribution. It's the third step of an unknown number.

This is NOT about compilers, and this isn't about wanting to make Win32 Perl more like Unix. This is about wanting Win32 Perl to be like Perl on every other one of the 100+ platforms, from Plan 9 to Nokia mobile phones.

Windows Is No Special Flower.

The low level common Perl modules should be able to build and install portably on all platforms (or as many as possible), and from there normal Perl developers should not have to care which specific platform they are on if they are not doing anything exotic. It should Just Work.

Windows is not special, it just presents some challenges, much as every other platform has presented some challenges.

And I think the Perl community as a whole deserves at least the option to face those challenges themselves, rather than loading all the work and all the pressure, and all the cost onto one company.

It's not fair to ActiveState and it's not fair to the Perl community.

[rhesa]

adamk wrote:

The aim is to provide CPAN authors a Win32 Perl platform that is similar in use and style to the Unix Perls, to make it easier for them to port to Win32, and to provide a suitable basic Perl platform for experienced Perl developers that prefer to install from CPAN, instead of using binary packages.

I didn't read that as aiming for a replacement of ActivePerl. Having a sensible toolchain on Win32 would remove much of the perceived reluctance to develop on that platform, because it would be much easier to start working _right away_, without having to dig around for compilers, header files, and what not.

I'm also guessing this would benefit the AS distribution in the long run, since it would mean more modules will build on Win32, which means AS will be able to package those too.

You wrote:

I imagine that any *nix programmer that encountered this problem when trying to port their module to Win32 would probably take one look, blame it on a broken MS C compiler/CRT and understandably give up, but the problem lies within the the perl sources.

That might very well be true right now. If this project succeeds, that argument would no longer be true, since we'd be using the same tool chain on both platforms. The grant also asks for fixing bugs like these.

You wrote:

Maybe Strawberry is only the starting point and would enable/encourage more people to become actively involved in making Perl work better on Win32?

I believe that's the goal, yes, absolutely.

Even though I said goodbye to Win32 long ago (a personal and professional luxury, I know), I do see this as a valuable plan: it would improve perl for a very large user base, integrating their development practices with those of unixy platforms; making more modules available to them means an expanded user base, which means more eyeballs, which implies improved code. I feel that's a win-win situation for both camps. Having used that word, I even have hope we won't be speaking of "both camps" in the future.

[BrowserUk]

I'm also guessing this would benefit the AS distribution in the long run, since it would mean more modules will build on Win32, which means AS will be able to package those too.

Um. The problem is, will it?

Are the DLLs produced by compiling XS code with MinGW compatible with a Perl built with MS VC++? If not, how will allowing module authors to "get away" with making minimal changes to their modules in order to achieve successful tests under Vanilla/Strawberry enable those same packages to pass the AS autobuild process?

[demerphq]

Are the DLLs produced by compiling XS code with MinGW compatible with a Perl built with MS VC++?

Assuming that the VC++ is VC6 then yes, if its VC7 then yes, but not if you use C++, if its VC8 then no.

The rule is that it doesnt matter what compiler was used, it matters what CRT is in use. Since stuff built with both VC6 and MINGW link against MSVCRT60.dll they are able to coexist peacefully. Similarly MSVCRT60 and MSVCRT70 are compatible in terms of C linking, but C++ exceptions will cause a problem. Luckily the use of C++ is rare in Perl.

Also as of later AS Perl builds you can compile using either MINGW or VC. If VC isnt available and MINGW is then EUMM will automatically use the latter.

[rhesa]

Aren't you implying that the AS distribution is the reference to which all Perl module authors should conform? Why wouldn't it be the other way around?

I don't see why AS couldn't switch to gcc for their builds (theoretically speaking). At the very least, Strawberry will give _them_ something to build and test against.

In any case, Strawberry would give module authors a tool to test these things for themselves. The barriers to that are currently too high, in my opinion. Whether they will make use of that option is another question, but it seems obvious that they won't _without_ these tools.

[adamk]

Yes, it will absolutely benefit AS in the long fun, and in fact in the short run too.

Because Vanilla has allowed the feedback loop to be shortened, we've been able to fix more than a dozen bugs in various areas that afflict ActivePerl as well, in places like CPAN.pm, Term::ReadLine, Win32API::Registry and others.

There are not bugs in C code related, they are bugs in Perl code. In fact, in some cases, I cannot see how the module could possibly build AT ALL so that it could be bundled into AP.

The VAST majority of bugs encountered so far have nothing to do with compilers at all. They are ordinary bugs in ordinary code. But perhaps because they are bundled, or perhaps because the feedback loop is at 6 month intervals, they aren't getting fixed.

[syphilis]

Are the DLLs produced by compiling XS code with MinGW compatible with a Perl built with MS VC++?

Yes they are - and vice versa, too. I've often mixed things in this way (in both directions). Exceptions may exist - such as Inline::C and Inline::CPP, where you've built an extension that expects you to be using gcc (or cl) and run it with a perl that expects you to be using cl (or gcc). If the VC++ compiler in question is other than version 6.0 then you might strike an additional (very rare) problem because of different runtimes being used.

(I also wondered what Strawberry will achieve. Having read this thread, I can't see that it has anything to offer me ... but that's about all I've been able to deduce ...)

Cheers,
Rob

[adamk]

This is not about DLLs. The problems with compiler compatibility has been largely fixed.

95% of the problems are in non-C areas.

[demerphq]

My understanding of this is that Vanilla Perl is meant to be a core Win32 perl distro. But I think Adam is grudgingly starting to realize that such a distro isn't actually that useful as a Win32 production tool so he wants a "Strawberry Perl" that contains extra modules like the extended Win32 modules (lib-win etal).

Personally I find this all a moderate touch amusing as I have long argued that Win32 perl is less than it should be because core perl just doesnt support enough of the Windows API to work around the fact that Windows doesnt behave like UNIX.

If strawberry perl is what is required to get enough of a consensus on this particular issue that we actually see some better Win32 API support in core then ill be happy.

[adamk]

You are right insomuch as Vanilla Perl IS meant to be as close to being a naked Perl core as we can make it. In this regard it tries to take the purist line even if it creates difficulties. Because it's identifying these difficulties clearly that are a major cause of our current situation.

But, given that we are approaching the solution from some distance away, it's quite clear that there is zero chance of Vanilla being magically made into a useful and more importantly SIMPLE in only a few months.

Indeed, what is probably going to be needed is a full release cycle of Perl itself.

I think Vanilla Perl 5.8.9 is going to be a very different beat entirely, and potentially much more useful in and of it's own right. But that still doesn't invalidate the usefulness in having an experimental-only "Win32 Perl core only" distribution, so that we can identify further problems.

Strawberry is the answer to the question "what is the smallest and most self-contained Win32 Perl distribution we can make, that is as easy to install and start using as people are used to on Windows".

Rather than making people go through the effort of installing Bundle::CPAN and any other core and toolchain upgrades from Vanilla in the 5.8.8 release cycle, it means we can have a relatively slim but still quite usable Perl distribution.

Something we can point people at and say "If you want to write Perl programs that use CPAN modules, you can use that". And it will be as trivial to set up and get started as people are used to.

Now, while you are also right in that we might need some better support for the Win32API modules, I'm not willing to take any of that on face value. I think that the Win32API modules need to only be going in on the basis of necesity, on a one by one basis.

So I think we'll certainly see Win32API::Registry and Win32::TieRegistry in Strawberry, because they are needed by Bundle::CPAN. At this point I'm also about 90% sure we'll see Win32API::File in there as well.

Beyond those two I still don't see any evidence that other Win32 API modules are absolutely necesary to the Perl toolchain, and that they can't just be installed later like any other module.

Where Vanilla is "Core Perl", Strawberry is "Core Perl + Toolchain". It gets us past the current problems of bootstrapping the Perl toolchain in THIS release cycle, and lets us have a distribution you can actually write code on and have some expectation it will run, and that you can install any other CPAN modules you might need in a relatively straight forward manner.

Now if you want to schlurp in every single thing in Win32:: and bundle the lot, you might be a little more interested in the Chocolate Perl concept :)

But it is my style, and has been a successful strategy so far, to attack these problems in discrete steps. It creates a much richer and more diverse ecosystem of solutions, and doesn't expose you to the sort of risks you face trying to tackle the ENTIRE problem space in one hit.

[demerphq]

Beyond those two I still don't see any evidence that other Win32 API modules are absolutely necesary to the Perl toolchain, and that they can't just be installed later like any other module.

Seriously Adam, how well do you know the API? Ive read elsewhere that youve stated that you arent a C programmer and aren't a Win32 developer, so on what basis do you form that conclusion?

I mean I can think of a variety of places where proper API support would be useful. Consider a module I've seen you discuss elsewhere, Term::Readline::Perl, if Win32::Console was core then the possibility that TRP could be made fully functional on Win32 Perl would be much higher.

I think there is a clear reluctance in p5p to put dependencies on non-core modules into core modules, so im doubtful that route is very useful. Even if we assume people do go the "Win32 support is optional" route (they havent really so far) I reckon at some point there will be so much cruft in the core code checking for proper Win32 support that I reckon it will outweigh just bundling the modules into core anyway. And as far as I can see the only valid reason proper Win32 support is omitted from core is distribution size, which itself actually would only affect the handful of people regularly working on core. (I think the true reason is actually just that there is a certain amount of prejudice and ambivalence towards the platform in the p5p group).

Anyway, you seem to really like the "no special flower" comment, so I'll throw one back at you: One size does not fit all. Everything has special needs, and thats ok. The world would be a very bland and boring place if there were "no special flowers".

And as a last follow up, for all my comments, please dont think that I dont support what you are doing with Vanilla Perl and Strawbery Perl. I do support it¹ . I think its a breath of fresh air in the Win32 Perl scene and i think long term it will achieve a lot of good things. So thanks a lot.

1. except the name, I hate the name "Strawberry Perl", why not White-Perl and Blue-Perl or something that is at least shorter to type?

[BrowserUk]

Personally I find this all a moderate touch amusing as I have long argued that Win32 perl is less than it should be because core perl just doesnt support enough of the Windows API to work around the fact that Windows doesnt behave like UNIX.

I agree.

If strawberry perl is what is required to get enough of a consensus on this particular issue that we actually see some better Win32 API support in core then ill be happy.

I was basically asking "Is will acheive that"?

You mention elsewhere that MinGW isn't compatible with VC++v8. When I went looking for the fabled "free MS compiler", this was the one I found. Is v7 also available?. I still use v6, but if v8 is the only free one available, then maybe the discussion above about making Perl compile with that is also a valid area of discussion.