Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

[JOB] The Perl Foundation seeks Windows Developer

by adamk (Chaplain)
on Apr 01, 2006 at 12:13 UTC ( #540629=perlnews: print w/ replies, xml ) Need Help??

I couldn't resist the title given the date, but this isn't actually a joke :)

Yesterday Ovid put out a call for grant applications, and ideas for things that could be sponsored.

I wrote in mentioning a project that needs to be done, but which I don't have the time or qualifications to handle. The grant committee is interested, and so I've been asked to put out a call for anyone interested in spending a month or so on TPF's dime.

In response to my CPAN-capable Windows Installer challenge in January, Carl Franks responded with Vanilla Perl, which is an installer containing MinGW, dmake, and core Perl compiled with them.

Vanilla could be classed as a research distribution. It allows people working on the Perl toolchain itself to work on Win32 compatibility, it provides a way to examine how the Perl code interacts with Win32.

But it isn't really useful to anyone doing development, and is still loaded with bugs, although both David Golden and myself have been working in bits and pieces to slowly squash the bugs in it.

What would be much more useful would be a different flavour of Perl, a distribution tentatively code-named Strawberry Perl.

The purpose of Strawberry Perl is to expand on Vanilla, bundling in all of Bundle::CPAN, as well as bundling some strategic upgrades of some dual CPAN/core modules that have had Win32-related upgrades since 5.8.8 was released.

The aim is to provide CPAN authors a Win32 Perl platform that is similar in use and style to the Unix Perls, to make it easier for them to port to Win32, and to provide a suitable basic Perl platform for experienced Perl developers that prefer to install from CPAN, instead of using binary packages.

Because nobody currently working on Vanilla can contribute very much time, the grant would involve the TPF paying someone to take responsibility for the creation of delivery of the first production release of Strawberry Perl.

This means tracking and keeping an eye on ALL of the bugs that are preventing the release, hunting down and diagnosing bugs, creating patches to fix them when the module's author isn't very Win32 savvy or doesn't have time to fix them, taking co-maint bits to release new versions of modules where the author is awol or doesn't have time to do releases, and generally dealing with any bits and pieces holding up Strawberry Perl.

This is the direction Vanilla is moving anyway, but currently it's moving in dribs and drabs, and it would be great to have someone push it through to completion.

Interested parties should have experience with both Win32 and Perl, and given the nature of the task, will need to play well with others.

So how about it, want to be funded to work on a Perl distribution for a while?

Note this is a grant, not a job, and so the rules of the grant process still apply. But the grants committee would look highly favourably on anyone interested in this project.

Comment on [JOB] The Perl Foundation seeks Windows Developer
Re: [JOB] The Perl Foundation seeks Windows Developer
by xdg (Monsignor) on Apr 01, 2006 at 13:20 UTC

    Let me add that the Win32 experience needed includes both Perl and the Windows API/C. Some of the bigger challenges may be getting unix-y programs to compile or unravelling problems in distributions that may build OK with Visual C++ and nmake, but break with MinGW and dmake (the tools in VanillaPerl). E.g. Win32::API is in the latter category.

    -xdg

    Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

      I'll note that you aren't expected to do 100% of everything yourself.

      There are certainly other people around doing bits and pieces.

      A great deal of the work is just locating and diagnosing the problem, and reporting it to the author so they can do the quick fix.

      But what we lack most of all is someone to really manage the process. Keep an eye on _all_ the bugs and all the authors, and step in wherever there are gaps to get everything sorted out.

      So while you may well need to do a lot of different things, hopefully you won't have to do it alone.
Re: [JOB] The Perl Foundation seeks Windows Developer
by Steve_p (Priest) on Apr 01, 2006 at 13:26 UTC

    An important dependency of this project is the ability to compile Perl and XS-based modules with Visual C++ 8.0, a.k.a. Visual Studio 2005 or the .NET 2.0 SDK C++ compiler. Change made in how C executables and DLLs are compiled with .NET 2.0 have caused significatant problems in building a Perl with this compiler. As of now, Perl cannot be successfully built with this compiler.

      Hi

      could you provide more details on this? I thought the Vanilla/Strawberry Perls were bundled with a gcc compiler. Is this a requirement of Strawberry perl or of the grant?

      thanks, j

      Update:

      It's the date thing, right?

        No, it is not part of the grant to do any additional work to adapt to changes introduced by commercial compilers that may have broken existing code.

        Vanilla/Strawberry are built with gcc, and while some people may feel it is important to do the work needed to handle changes in commercial compilers, I don't personally feel it is the responsibility of the TPF to be funding compatability with non-open-source compilers, particularly where the blame for the code not working lies with those commercial compilers.

        So while obviously you shouldn't be introducing any NEW bugs that might break existing working scenarios, fixing problems caused by commercial compiler vendors will not be part of the grant, or at least part of THIS grant. The grant committee may feel it is worth seperately funding this work, but I'd prefer to deal with seperate problems seperately.

        In addition, and as an aside, Windows Vista is incompatible with nmake, and there is no obvious upgrade path for name, seeing as we have been using a very very old legacy build of nmake this entire time.

        So from Vista onwards, I would expect that all Perl distributions will need to transition away from nmake to something like dmake.
Re: [JOB] The Perl Foundation seeks Windows Developer
by BrowserUk (Pope) on Apr 02, 2006 at 16:33 UTC

    I keep reading this, and I'm still lost for an explanation of what the Vanilla and Strawberry projects are aiming to achieve?

    As a Win32 Perl user,

    1. I can already build my own perl

      I actively sic choose to use Active State Perl for various reasons.

      • They've already solved most of the problems.
      • They've added value in the form of all the must have modules they bundle with their distributions.
      • They've added value in the form of their html-ised documentation sets which are 1000% more usable that perldoc.
      • They've added value in the form of the PPM manager.

        Whilst I can build most modules myself, having a quick, simple and convenient 'just do it' install for the more complex or tricky modules is a huge time saver.

        Why expend time and energy re-discovering all the problems and solutions to installing packages on win32 for every installation, when the problems have already been solved and are available for one-step installation?

    2. I can already build my own modules, with or without the use of CPAN.pm

      I can use the CPAN shell. I choose not to, because it carries with it so much baggage that simply does nothing for me.

      For the vast majority of modules, I can manually download, build, test and install them--including 1 or 2 small dependencies before the CPAN shell has finished configuring itself, which it seems to need to do every time?

    On those occasions I fail to be able to build a module, (most recently Devel::FastProf), the root cause seems to be much more to do with the way the authors write their modules to make use of inherently *nix-only features of the Perl source code, or by directly accessing parts of the Perl runtime bypassing the good work done the developers to isolate OS dependencies by virtualising them.

    I guess what I am saying is, I certainly have the time, and (some of), the skills that would allow me to contribute to this project, but I am at a loss to understand what it is that you envisage that I, and other win32 users, would gain from this effort?

    Also, what does the overall Perl community gain from the TPF funding this project?

    The bottom line is, you are asking for someone to apply for a TPF grant, and therefore raise a grant application, but it's not at all clear to me what the objectives of the project are? Who is it intended to help and how?

    I also think that a much clearer brief is required. In this short thread we've already seen that there is more than one opinion as to the direction that should be pursued. Your brief justification of the motivation for Vanilla & Strawberry

    The aim is to provide CPAN authors a Win32 Perl platform that is similar in use and style to the Unix Perls, to make it easier for them to port to Win32, and to provide a suitable basic Perl platform for experienced Perl developers that prefer to install from CPAN, instead of using binary packages

    Leaves me uneasy that it wouldn't benefit Win32 Perl users much. Despite the almost obligatory "I don't/only/have to use Win32" disclaimer, the biggest problem facing Win32 Perl users is that the vast majority of Perl developers don't want to develop on Win32, and this project sounds like yet another attempt to make win32 look as much like *nix as possible to allow reluctant but willing CPAN authors to more easily test and bug fix their modules on the win32 platform. Maybe that is the right way to go, but based on my own limited experience of trying to build the 'awkward squad' of modules on Win32, the problems go deeper than the compiler, CRT or build utilities.

    As an example of what I mean, the problem I encountered with Devel::FastProf was that although the MSVC CRT has perfectly normal C getc() routine, once you've bypassed various errors to do with simple unixisms, the base Perl headers files are converting the getc() used in the source, to a call to

    FastProf.xs(97) : warning C4013: 'PerlSIO_getc' undefined; assuming ex +tern returning int

    And the PerlIO/SIO subsystem doesn't appear to provide this routine. I've tried to unwind the macros to work out why getc() is redefined as PerlSIO_getc(), when that routine does not appear to exist anywhere in the entire Perl source tree, not just the Win32 part, without success.

    I imagine that any *nix programmer that encountered this problem when trying to port their module to Win32 would probably take one look, blame it on a broken MS C compiler/CRT and understandably give up, but the problem lies within the the perl sources. Does anyone anywhere use the whole PerlSIO subsystem? What was the purpose of that layer of headers files? And was it ever achieved/completed?

    And this is not an isolated incident. Many of the other problems of compiling code written to run on GCC with the MS compiler come down to compiler settings. Things like declaring for loop variables inline, which gcc allows even though it isn't part of the C standard, have to be explicitly enabled using the MS compilers.

    My point is, I am not at all sure that simply moving to a perl distribution that builds with gcc is going to fix very much, or help Win32-reluctant module developers to more easily port their modules. And unless Strawberry also included all, (or at least most), of the value added features that come with the AS distributions, then Win32 Perl users, (as opposed to developers), are going to be very reluctant to adopt it. It would be of little benefit to anyone if the only people that made use of Vanilla/Strawberry Perl were only developers looking to simplify the porting of their modules to Win32, because it is quite likely that the results of their efforts would not work for the majority of Win32 users.

    Please don't misunderstand me, I would hate to be seen as dismissive of any attempt to make Perl more Win32 friendly. Indeed, I would love to be a part of such an effort and dog knows if there was a little income to be earnt from such efforts it certainly would not go amiss. My fear is that, as I understand the direction outlined here, (which of course my be a completely different thing from the intent), it may not, even probably would not, do much to help in this regard, and that any monies that the TPF feels are justified in making Perl more Win32 friendly/complete might be better directed.

    Maybe Strawberry is only the starting point and would enable/encourage more people to become actively involved in making Perl work better on Win32? I think I am the wrong person to make that judgement and so I would encourage an expanded discussion of the overall goals and the best way of achieving them, to help convince me, and others, that this is the right way to go.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      adamk wrote:
      The aim is to provide CPAN authors a Win32 Perl platform that is similar in use and style to the Unix Perls, to make it easier for them to port to Win32, and to provide a suitable basic Perl platform for experienced Perl developers that prefer to install from CPAN, instead of using binary packages.

      I didn't read that as aiming for a replacement of ActivePerl. Having a sensible toolchain on Win32 would remove much of the perceived reluctance to develop on that platform, because it would be much easier to start working _right away_, without having to dig around for compilers, header files, and what not.

      I'm also guessing this would benefit the AS distribution in the long run, since it would mean more modules will build on Win32, which means AS will be able to package those too.

      You wrote:

      I imagine that any *nix programmer that encountered this problem when trying to port their module to Win32 would probably take one look, blame it on a broken MS C compiler/CRT and understandably give up, but the problem lies within the the perl sources.

      That might very well be true right now. If this project succeeds, that argument would no longer be true, since we'd be using the same tool chain on both platforms. The grant also asks for fixing bugs like these.

      You wrote:

      Maybe Strawberry is only the starting point and would enable/encourage more people to become actively involved in making Perl work better on Win32?

      I believe that's the goal, yes, absolutely.

      Even though I said goodbye to Win32 long ago (a personal and professional luxury, I know), I do see this as a valuable plan: it would improve perl for a very large user base, integrating their development practices with those of unixy platforms; making more modules available to them means an expanded user base, which means more eyeballs, which implies improved code. I feel that's a win-win situation for both camps. Having used that word, I even have hope we won't be speaking of "both camps" in the future.

        I'm also guessing this would benefit the AS distribution in the long run, since it would mean more modules will build on Win32, which means AS will be able to package those too.

        Um. The problem is, will it?

        Are the DLLs produced by compiling XS code with MinGW compatible with a Perl built with MS VC++? If not, how will allowing module authors to "get away" with making minimal changes to their modules in order to achieve successful tests under Vanilla/Strawberry enable those same packages to pass the AS autobuild process?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
      Also, what does the overall Perl community gain from the TPF funding this project?

      In short: diversity and choice.

      I think the Perl community gains by opening up Perl usage on Windows to a wider audience. Until now, Perl on Win32 has really meant two choices: Cygwin or ActiveState, both of which bring a lot of baggage along with their benefits.

      One of the downsides of relying on, for example, ActiveState, is that while they solve some problems for you, you wind up held hostage to choices that they make. One of the goals of CamelPack and VanillaPerl, etc. is to make having your own Perl distribution on Win32 as easy as it is on unix-based platforms -- with direct reliance on CPAN rather than intermediate repositories to allow for access to the most up to date modules available.

      Here's a real-life example: Scalar::Util. ActiveState's PPM repository targets the first ActiveState release for a particular major version of Perl to maximize backwards compatibility and (presumably) minimize support costs. Because the version of Scalar::Util frozen into the first ActiveState Perl 8xx release didn't include the critical refaddr function, every CPAN module that requires a Scalar::Util with refaddr fails the automated build process for 8xx PPMS -- even though later releases of ActiveState Perl 8xx include more modern versions of Scalar::Util. That's a lot of modules -- including my own Class::InsideOut.

      With VanillaPerl, I and other users don't have to rely upon PPM repositories -- they just use CPAN directly and can rely on the dependency management code already built into the CPAN toolchain. And, because a compiler is bundled (unlike with ActiveState), that includes XS modules.

      I certainly don't think that's for everyone. Many end users may be perfectly happy with a PPM-dependent approach. But I think TPF and the community will benefit by creating more choices.

      -xdg

      Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

        I just went to CPAN and pulled Scalar::Util. It took less than 10 seconds to download via my dial-up connection. And doing the prayer-in-four-parts took almost exactly the same amount of time. And that included me mis-typing the name once. There is no reason why AS' autobuild process couldn't download the latest/required version of module dependancies. That's a procedural problem for AS, not a problem for users.

        It sucks that perfectly workable modules are getting flagged as unbuildable by the AS process, especially for want of the latest version of a module, (in this case developed by an AS developer?), that they already know builds correctly as they ship it with later versions of their own distributions.

        However, the version I just downloaded and built was Scalar::Util v1.17. The latest version available on cpan is v1.18. This fails to build under AS 5.8.6 because of an unresolved external

        Creating library blib\arch\auto\List\Util\Util.lib and object blib\ +arch\auto\List\Util\Util.exp Util.obj : error LNK2019: unresolved external symbol _Perl_seed refere +nced in function _XS_List__Util_shuffle blib\arch\auto\List\Util\Util.dll : fatal error LNK1120: 1 unresolved +externals

        And that comes down to my having 5.8.6 rather the 5.8.8. A change 26054, was made to Scalar::Util::shuffle to address a problem whereby dprof would "break" shuffle. That change included calling Perl_seed(), which (apparently) was not previously exported. So change 26836 was required to export Perl_seed and fix the linker error.

        My point is, this is standard version control ripples and I don't see how that would be fixed by using a different compiler?

        Vanilla/Strawberry Perl may make it easier and simpler for developers to get set up to test code on Win32, but unless the vast majority of users also move to using it, the only benefit woudl be in the statistics of successful Win32 builds--which would be pretty useless unless users can benefit. It could even make things worse, as I could see the standard response to questions about build problems of packages on win32 being "it works with Vanilla Perl, so it must be Active State that is broken", which wouldn't help many users.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

      My understanding of this is that Vanilla Perl is meant to be a core Win32 perl distro. But I think Adam is grudgingly starting to realize that such a distro isn't actually that useful as a Win32 production tool so he wants a "Strawberry Perl" that contains extra modules like the extended Win32 modules (lib-win etal).

      Personally I find this all a moderate touch amusing as I have long argued that Win32 perl is less than it should be because core perl just doesnt support enough of the Windows API to work around the fact that Windows doesnt behave like UNIX.

      If strawberry perl is what is required to get enough of a consensus on this particular issue that we actually see some better Win32 API support in core then ill be happy.

      ---
      $world=~s/war/peace/g

        Personally I find this all a moderate touch amusing as I have long argued that Win32 perl is less than it should be because core perl just doesnt support enough of the Windows API to work around the fact that Windows doesnt behave like UNIX.

        I agree.

        If strawberry perl is what is required to get enough of a consensus on this particular issue that we actually see some better Win32 API support in core then ill be happy.

        I was basically asking "Is will acheive that"?

        You mention elsewhere that MinGW isn't compatible with VC++v8. When I went looking for the fabled "free MS compiler", this was the one I found. Is v7 also available?. I still use v6, but if v8 is the only free one available, then maybe the discussion above about making Perl compile with that is also a valid area of discussion.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
        You are right insomuch as Vanilla Perl IS meant to be as close to being a naked Perl core as we can make it. In this regard it tries to take the purist line even if it creates difficulties. Because it's identifying these difficulties clearly that are a major cause of our current situation.

        But, given that we are approaching the solution from some distance away, it's quite clear that there is zero chance of Vanilla being magically made into a useful and more importantly SIMPLE in only a few months.

        Indeed, what is probably going to be needed is a full release cycle of Perl itself.

        I think Vanilla Perl 5.8.9 is going to be a very different beat entirely, and potentially much more useful in and of it's own right. But that still doesn't invalidate the usefulness in having an experimental-only "Win32 Perl core only" distribution, so that we can identify further problems.

        Strawberry is the answer to the question "what is the smallest and most self-contained Win32 Perl distribution we can make, that is as easy to install and start using as people are used to on Windows".

        Rather than making people go through the effort of installing Bundle::CPAN and any other core and toolchain upgrades from Vanilla in the 5.8.8 release cycle, it means we can have a relatively slim but still quite usable Perl distribution.

        Something we can point people at and say "If you want to write Perl programs that use CPAN modules, you can use that". And it will be as trivial to set up and get started as people are used to.

        Now, while you are also right in that we might need some better support for the Win32API modules, I'm not willing to take any of that on face value. I think that the Win32API modules need to only be going in on the basis of necesity, on a one by one basis.

        So I think we'll certainly see Win32API::Registry and Win32::TieRegistry in Strawberry, because they are needed by Bundle::CPAN. At this point I'm also about 90% sure we'll see Win32API::File in there as well.

        Beyond those two I still don't see any evidence that other Win32 API modules are absolutely necesary to the Perl toolchain, and that they can't just be installed later like any other module.

        Where Vanilla is "Core Perl", Strawberry is "Core Perl + Toolchain". It gets us past the current problems of bootstrapping the Perl toolchain in THIS release cycle, and lets us have a distribution you can actually write code on and have some expectation it will run, and that you can install any other CPAN modules you might need in a relatively straight forward manner.

        Now if you want to schlurp in every single thing in Win32:: and bundle the lot, you might be a little more interested in the Chocolate Perl concept :)

        But it is my style, and has been a successful strategy so far, to attack these problems in discrete steps. It creates a much richer and more diverse ecosystem of solutions, and doesn't expose you to the sort of risks you face trying to tackle the ENTIRE problem space in one hit.
Re: [JOB] The Perl Foundation seeks Windows Developer (*)
by BrowserUk (Pope) on Apr 04, 2006 at 15:10 UTC

    So, that should really read,

    Wanted: Tame Win32 developer who will keep quiet and just do what we clever Unix guys tell him to do; and not ask silly questions like who is this meant to benefit and how.

    Pretty much what I thought.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlnews [id://540629]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (7)
As of 2014-10-25 17:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (146 votes), past polls