Perl6 - Distribution builder/installer

[dragonchild]

(This is a continuation of Perl6 - Module::Build.)

After reading those replies, talking on #perl6, and emailing perl6-language, I've come to the following conclusions:

• There is a problem with how the problemspace is being divided up. There seems to be a call for having a separate distro builder and distro installer.
• Actually, there seem to be six separate components in the whole distribution management space:
  1. Distro builder (ExtUtils::MakeMaker / Module::Build)
  2. Distro installer / uninstaller (EU::MM / M::B)
  3. Distro metadata inspector (No P5 analog)
  4. Distro distribution center (C6PAN)
  5. Distro up-/down-loader (Think CPAN/CPANPLUS and the cpan_upload script)
  6. The 'use' and 'require' keywords
  I include the latter because all 6 components need to be aware of S11.
• Because of this, I suspect there's a seventh, shared component that controls how @*INC is searched and which of the competing options is selected.
• That shared component needs to be mixin-able, with its Roles overridable. Because the way authoring/versioning metadata is stored will vary by OS, there needs to be OS-specific code in there.
• A major issue, which apparently has not been discussed a lot, is how to implement the fact that Parrot (and any other VM P6 may target) will provide for multi-lingual distributions. The only comparison that exists is .NET, and that problem is solved by the fact that all the languages are under the same architecture team. We don't have that luxury.

Now, what I would like from the community:

• What actions would you like to see from the various components? What would those actions entail? Would there be dependencies between actions? If so, could those dependencies be overriden (a la "perl -MCPAN -e 'force install Foo::Bar'")
• Ideas regarding the Parrot dillema, particularly ideas crossposted to p6-language.
• Comments on anything and everything I've posted so far.

[Zaxo]

I think that many of the problems shared by those components could be relieved by putting some teeth into Config.pm. Presently in P5, Config.pm has lots of potentially useful details about the local system, but many have the disclaimer in pod that they shouldn't be relied on.

System utility paths are reset to some vanilla value after configuration (bash, bison, byacc, cat, . . .). Why not leave them intact? Why not have a localization script which can correct those entries?

Config.pm seems quite accurate to me, but I'm on linux and I always build my own perl from source. That makes most things Just Work for me. With the fresh start of P6, there is the opportunity to tell vendors that the configuration must be accurate as installed and used on the local system.

[Tanktalus]

I do understand that it's easier to be a nay-sayer than to contribute positively, so I kinda regret doing this.

For non-shared perls, that's great. For shared perls, it merely might be good. There is a lot of thought that needs to go into giving Config.pm the teeth it needs. Some of it is reasonable, some much more difficult.

There are at least a few different uses for Config.pm that I'm aware of, and thought needs to be applied to all uses (whether I've thought of them or not). That doesn't mean that they all need to stay in Config.pm in Perl6, only that these different "users" of Config.pm still can do what they need to. For lack of better terms, they are:

1. The IT guy. This is the guy whose task it is to compile a perl that is usable by one or more machines. This is the person who is creating the initial Config.pm (though s/he may not realise it) to compile perl itself, plus all the modules that come with Perl.
2. The global module installer. This is the guy whose task it is to install more modules over the rest of the use of this copy of perl.
3. The local module installer. This is someone who doesn't have full privileges to the system, and thus is installing some private modules to use with the global perl.
4. The user. Someone who is running perl code, not necessarily caring that the code is in perl.

Except for the first role, each role has a many-to-many relationship to people: more than one person may have sysadmin authority to install global modules, more than one person (possibly on more than one computer) will install local modules, and, hopefully, more than one actual end user, but even then there may be just one end user (and that user could be cron or apache).

At the other end, for example at home, all these roles are filled by me. The configuration must work for all these roles, or as many roles as are solvable and realistic (the user role must always work for all machines sharing the same perl).

That said, this may really work, with a good design. I just think it's a bit more complex than you're implying, and would look forward to some group coming up with a universal solution here.

[tilly]

Two immediate reactions.

The first is that you can insist all you want, but when vendors ship binary distributions they won't pay much attention to whether the configuration is accurate, and users won't be first to blame the vendor for those mistakes.

The second is that I've done several network installs of Perl. That's where I install Perl to a file server that a bunch of Windows machines all see, instead of installing to each machine. Yes, I know all of the ways that this is fragile. But it has worked very well in practice, and you'll find that people will do this whether you like it or not.

[dragonchild]

The shared item I was thinking of is the OS-dependent bit that navigates the modulename-specification that S11 lays out. I can imagine it being implemented with symlinks on *nix, but that's not going to work on VMS.

As for Config ... I actually don't know how much is going to be needed, frankly. Especially given the idea floating around that documentation for a module should be within that module's install. Maybe the manpages for Perl shouldn't be in $Config{man1}, but $Config{perlmanpages} and let the understanding of that be OS-dependent.

[BrowserUk]

A few thoughts for consideration.

• All files relating to a particular module(version/author) should exists in a directory tree below that module's place within the file system. Including, but not limited to:
  1. Distribution files (.tar.gz etc.)
  2. Build files (makefiles, source files)
  3. Built files including binary (dynamically linked) components.
  4. Test files.
  5. Build process log files.

  One exception might be generated docs like htmlized POD/man pages.

• Parrot can save/load bytecode files.

  At use time, the first thing looked for should be a pre-compiled Parrot bytecode file. If found, this can be loaded, bypassing the need for compilation. Some check for updates should be possible.

  When a module is installed and tested, the bytecode is written out to the appropriate place (if tests pass?) and that module need never be recompiled until a new version is downloaded.

• If I've been following things correctly, Parrot bytecode is platform independant?

  If so, it would sometimes be convenient to be able to download the bytecode only from CP6AN. Even better if this could be done (safetly) on the fly.

• As well as use implicitly loading the latest version of module where the version is unspecified, and the specified version when requested, the ability to hold debug and/or development versions within the same libtree and request they be loaded would be handy.
• A tool for taking a development version directly from the libtree and packaging it for distribution would be a nice convenience.
• An tool equivalent to Module::Starter that created a development skeleton in-situ would complete the toolset.
• Random thought. I wonder if there is any scope/merit in adding svn source control into the mix?

[dragonchild]

A few responses I already have the answers to:
• As well as use implicitly loading the latest version of module where the version is unspecified, and the specified version when requested, the ability to hold debug and/or development versions within the same libtree and request they be loaded would be handy.

  According to S11, there are three items that determine what gets loaded:

  1. Module name
  2. Module version range
  3. Module "authorizing authority URI" (usually author).
  So, you can have the C6PAN version and the debug version and it will "Just Work".
• An tool equivalent to Module::Starter that created a development skeleton in-situ would complete the toolset.

  The distro builder would have this action. There's no reason why it should be separated.

• Random thought. I wonder if there is any scope/merit in adding svn source control into the mix?

  I don't think so. For one thing, SourceForge is still CVS. Another is that some prefer svk over svn and (as I discoverd on feather) they're not identical.

• Parrot loading bytecode

  This seems to indicate that Parrot is the thing we're installing modules for, not Perl6. So, we would have a module installer for each VM Perl6 targets? We have to reinstall eveyr time Parrot is upgraded? I'm not sure I like that ...

  Maybe, instead, we should be looking at caching the bytecode from the module and it's used unless either Perl6 or Parrot or the module has changed? That would seem to be something for "use"/"require" to do ...

[BrowserUk]

So, you can have the C6PAN version and the debug version and it will "Just Work".

So, just make the debug version's author 'debug' or similar. Makes sense.

I don't think so. For one thing, SourceForge is still CVS. Another is that some prefer svk over svn

I guess I should have just said source control (omitting the 'svn'), but most everyone is seems to be settling upon svn recently. It would be kind of nice to have CP6AN be the master repository and then authors could just branch locally for the next version and commit to CP6AN when they're ready. Also, svn/svk will talk to each others repositories won't they? At least for the 'co' & 'up' commands?

And the download tool could use the intelligent diffing to update installed packages rather than downloading the whole thing. And just checkout new modules? Not sure if that could work, but the thought crossed my mind so I mention it.

I'm not sure where the sourceforge stuff comes in here? Is CP6AN likely to be on SF?

This seems to indicate that Parrot is the thing we're installing modules for, not Perl6.

I know that Parrot is intended to support multiple languages but I don't recall reading that Perl6 was intending to target multiple VMs?

In any case, when the builder runs the test suite, it could (optionally) issue a

perl6 --cache-bytecode -MNew::Module

as a final action if the test passed. Or something...

I agree that it is probably a function of use/require to actually load the bytecode, but I thing the Perl6 executable could be asked to load/save/igonore cahced bytecode and the builder module could support thiose options on a case-by-case basis as well as it being a configurable option for an installation?

Anyway. Just thoughts for the pot.

[adrianh]

Comments on anything and everything I've posted so far.

Random comments:

• I don't think separating out the module builder and installer will be a big issue. There's gonna be a whole lot of common code between the two so you'll probably end up with them in the same distribution anyway..
• I wouldn't worry about C6PAN/CPANPLUS. These should be independent of the module builder/installler. People are always going to want to build/install modules outside of the context of CPAN. The only issue I would have is to make sure that there is a way to get at your META.yml equivalent without having to execute code.
• Might be worth looking at Ruby Gems (one of Ruby's solution to this problem). The chapter on Gems from the Pickaxe book is available for download if you want a overview. There are some interesting design decisions.
• Dig out the Module::Build mailing list archives :-)

[salva]

modules should be easy to repack as deb, rpm, etc.