Truly reusable software components

[vrk]

Perl programmers have CPAN. It's a repository of modules for all sorts of problems and solutions, interfacing with various technologies, network protocols, and file formats, adding features to the language, and a very long list of other ready components. In a way, it fulfills the dream of object-oriented programming people, and really any programmer, of having a generic and complete library of all written modules so far that can then be used in solving new problems.

This is an ideal situation. How many Perl progammers realize it? It's not only about saving time and effort, or enabling you to use technologies that you don't have time or interest to learn, or the ability to create well-nigh trivial solutions with just the right choice of modules, or having the largest repository of modules to brag about. It's all this and a realization of an old ideal. With sugar on top.

There's only one problem with it: it's Perl only.

Other programming languages do have "comprehensive archive networks" -- frequently modelled after CPAN itself (which, in turn, was modelled after CTAN). With some googling, I was able to locate the following archives of reusable modules for different programming languages:

Network	Archived technology	Year founded	Approximate size (Oct 17th, 2007)
CTAN	TeX	1992	7.5 GB (probably including different versions)
CPAN	Perl. Of course.	1994	4 GB, 12345 modules
CRAN	R	2004?	1197 packages
CEAN	Erlang	2006?	212 packages
CKAN	Open Knowledge Definition	2005	112 packages
JSAN	JavaScript	2005	A few dozen?
CSAN	Scheme	2004	0 packages(?)
cCLan	Lisp	Some time after 2000?	?
PLaneT	Scheme	?	Around 270 packages
ASDF-Install	Lisp	?	Around 360 packages

(Well, Open Knowledge Definition is not a programming language. TeX is, in a way. Is it Turing complete?)

The main problem is duplication of effort, something these archives try, by definition, to reduce! But there doesn't seem to way a out of the dilemma: how do we create a repository of generic, truly reusable software modules that are language-agnostic? There is no inclination of the world converging to one single programming language, so picking one "universal" programming language is not an option.

Perhaps there could be an archive targeting a universal virtual machine. Although there would have to be consensus on the machine architecture first (I propose the SECD machine), imagine that all programming languages had compilers that could target the virtual machine. The packages in this imaginary archive would have two versions: the source code and binary machine code for the virtual machine. Then, depending on if you have the compiler, or the language runtime environment, installed or not, an automatic install tool would pick either the source code of the program or the binary version compiled to the virtual machine.

Benefits: as long as source languages had compilers, modules could be written in any language, in principle. Even if you had never heard of the language before, you could still use the virtual machine code bundled with the module.

The obvious drawbacks are having to rely on machine language again, even if it is for a virtual machine. Another is the complexity involved, and duplication of effort in terms of compilers.

Any other ideas? I don't really accept "but Perl is a universal language!" as an answer, at least not until Perl 6 exists.

Update: omouse pointed out more Scheme and Common Lisp archives, which were added to the table: PlaneT and ASDF-Install. The latter seems to even be hosted at CLiki, which is where cCLan can be found too... Sorry for the oversight.

[moritz]

The idea of a common virtual machine is used for .NET, and, more popular here, parrot ;-).

The big difference is that .NET is optimized for statically typed languages, and parrot for "dynamic" languages.

The parrot developers are working hard on implementing all kinds of languages, and once they have released version 1.0 I'm sure they will gain much more interest. Especially if they manage to make it very fast. I believe they will ;-)

That will make sharing of libraries much easier, so you'll be able to code in "Perl 6 on Rails" if you want, or Python with Catalyst - fill in your favorite language and library.

The big problem, as always, is to convince enough people to focus on one platform (be it a VM or something else).

[kyle]

Well, Open Knowledge Definition is not a programming language. TeX is, in a way. Is it Turing complete?

Yes, for what it's worth, TeX is Turing complete.

[bradenshep]

While it's by no means a universal VM, or even a VM, x86 32-bit architectures fulfill some of this, in the sense that it's commonly and widely used.

With a more general and uniform way of calling libraries written in other languages from any of the languages (see .NET) the virtual machine and the machine itself can be one and the same. Give every language a compiler for the native architecture (or an interpreter running natively on same) and it's your VM.

The "general and uniform way of calling libraries written in other languages" is the tricky part.

[doom]

The main problem is duplication of effort, something these archives try, by definition, to reduce! But there doesn't seem to way a out of the dilemma: how do we create a repository of generic, truly reusable software modules that are language-agnostic? There is no inclination of the world converging to one single programming language, so picking one "universal" programming language is not an option.

Writing code for a VM isn't a solution, because in general the real problem here is not solvable by technical means: it's fundamentally a social problem.

Groups of programmer's are uncomfortable using tools that are not written in their language of choice. This is for a combination of reasons ranging from mindless snobbery to intelligent conservation of effort.

This subject also comes up when discussing extension languages for programmer's development environments. To quote what I've said before on the emacswiki, ExtensionLanguageAdvocacy: "But embracing multiple languages necessarily involves sub-dividing the community. Even if there’s no actual fork in the elisp code-base, there would be splits in the emacs community – you need to be a perl programmer to further extend an extension written in perl, and conversely most perl-programmer’s will have little interest in assisting with work on extensions written in Java."

[omouse]

I'm really amazed that you didn't find the right Lisp and Scheme CPAN-clones. The one for PLT/MzScheme is PLaneT which is here: http://planet.plt-scheme.org/

and there is one for Common Lisp here: http://www.cliki.net/ASDF-Install
(ASDF-Install can install and download packages from the cliki.net website)