Announce: Pinto-0.026 -- Curate Your Own CPAN Repository

[jthalhammer]

Hello esteemed Monks,

A few months ago, I was tasked with building a private CPAN for a new client. I had already done this a couple times before using CPAN::Site or CPAN::Mini, but I was never really happy with the results. So this time, I started from scratch. The result is called Pinto, and it is now available on CPAN. Pinto is inspired by CPAN::Mini, CPAN::Mini::Inject, and OrePAN, but adds several interesting features (listed below).

So I invite you to kick the tires on Pinto and give me your feedback. Beware that Pinto has a lot of dependencies, and I'm still not sure what the minimum versions should be. So I recommend brewing a fresh perl or using the -L option with cpanm(1) to install Pinto into a clean (and separate) sandbox. Pinto is still alpha code and subject to change in incompatible ways. And make sure you get the 0.026 version.

• Pinto supports several usage patterns: With Pinto, you can create a repository that mirrors all the latest distributions from CPAN. Or you can create a repository with just your own private distributions. Or you can create a repository that has all the distributions required for a particular project. Or you can combine any of the above patterns.

• Pinto supports adding and removing distributions in the repository: Pinto gives you the power to precisely tune the contents of your repository. For example, you can add your own locally patched CPAN distribution, and then remove it when the original author releases a fixed version.

• Pinto helps you avoid breakage due to upgrades: Pinto lets you pin your repository index to a particular version of a package, so that you can hold that package at the last know working version, while allowing other packages to be upgraded.

• Pinto can be integrated with your version control system: Pinto can automatically commit to your version control system whenever the contents of the repository changes. This gives you reproducible and identifiable snapshots of your dependencies, and a mechanism for rollback when things go wrong.

• Pinto makes it easier to build several local repositories: Creating new Pinto repositories is easy, and each has its own configuration. So you can have different repositories for each department, or each project, or each version of perl, or each customer, or whatever you want.

• Pinto can pull distributions from multiple remote repositories: Pinto can mirror or import distributions from multiple sources, so you can create private (or public) networks of repositories that enable separate teams or individuals to share distributions.

• Pinto supports team development: Pinto is suitable for small to medium-sized development teams, where several developers might contribute new distributions at the same time. Pinto ensures that concurrent users don't clobber each other.

• Pinto has a robust command line interface: The pinto-admin command line tool has options to control every aspect of your Pinto repository. It is well documented and behaves in a DWIM fashion.

• Pinto can be extended: You can extend Pinto by creating Pinto::Action subclasses to perform new operations on your repository, such as extracting documentation from a distribution, or grepping the source code of several distributions.

Pinto might be easiest to explain with a couple command-line demonstrations. In this first demonstration, suppose our application depends on Plack-0.98, which happens to be the latest version at that moment:

# Create an empty CPAN-like repository in directory ~/mypan
$ pinto-admin -r ~/mypan create

# Get the current version of Plack (and all its dependencies) into the
+ repository
$ pinto-admin -r ~/mypan import Plack

# Suppose HTTP::Simple has a bug and you made your own patch as HTTP-S
+imple-0.23-PATCHED.tar.gz
$ pinto-admin -r ~/mypan add HTTP-Simple-0.23-PATCHED.tar.gz

# Point cpan(1) or cpanm(1) at your repository, to install Plack with 
+your patched HTTP::Simple
# cpanm --mirror file:///home/you/mypan --mirror-only Plack

# Time passes...and now we decide we want to upgrade the latest Plack,
+ which is now Plack-2.4
$ pinto-admin -r ~/mypan import Plack-2.4

# It seems that bug in HTTP::Simple has now been fixed.  So we can rem
+ove our patched version
$ pinto-admin -r ~/mypan remove HTTP-Simple-0.23-PATCHED.tar.gz

For this next demonstration, suppose we want to have the latest version of everything in CPAN:

# Create a new repository (or you can use an existing one)
$ pinto-admin -r ~/mypan create

# Get the latest version of ALL packages from a CPAN (may take a coupl
+e hours)
$ pinto-admin -r ~/mypan mirror

# Now we have a stable snapshot of CPAN we can install from
$ cpanm --mirror file:///home/you/mypan --mirror-only Catalyst Moose P
+erl::Critic

# Time passes...now we want to bring our snapshot up to date with the 
+tip of CPAN
$ pinto-admin -r ~/mypan mirror

# But wait!  The new Moose-2.1 breaks our code.  We want to stay with 
+our old Moose-1.4
$ pinto-admin -r ~/mypan pin Moose-1.4

# More time passes...We've fixed our code to work with Moose-2.1
$ pinto-admin -r ~/mypan unpin Moose

You can actually combine these scenarios in various ways too. At the end of the day, Pinto provides you with a stable, cherry-picked slice of CPAN that you can systematically evolve over time. I'm still figuring out how best to apply that capability. Your insights are much appreciated.

[DrHyde]

I confess to having not read the whole post, but ...

Pinto helps you avoid breakage due to upgrades ... if you "pin" a module to a particular version, have you solved the dependencies problem - ie, that that version might break with a newer version of one of its dependencies, or that pinning it might make future releases of stuff that depends on it break? I gave quite a lot of thought to this when coming up with cpXXXan, but decided that it was Far Too Hard to actually solve properly in my spare time!

[jthalhammer]

have you solved the dependencies problem - ie, that that version might break with a newer version of one of its dependencies, or that pinning it might make future releases of stuff that depends on it break?

No, I'm not that smart. Pinto doesn't figure out the optimal dependency graph for you. It just helps you tune the index so that you can (eventually) discover the optimal graph yourself.

Pinning a package merely holds the index to a particular version of that package. That version could break with newer dependencies (although I certainly could implement a recursive "pin" that would also pin dependencies). And pinning a package could make it impossible for cpan(1) or cpanm(1) to satisfy a prerequisite. In either case, you won't know until you build.

So finding the optimal dependency graph is still a trial-and-error process. Manually solving this problem for all of the CPAN is probably not feasible. But I'm hopeful that it is feasible for the average project which only uses a small subset of the CPAN.

Pinto could be used in conjunction with cpXXXan. For example, I could create a Pinto repository to track CP5.12.0AN, and then pin the Pinto repository index to those versions that are compatible with my application. So I'll end up with an index that is most likely to yield a working build of my application on perl 5.12.0.

Come to think of it, you probably could implement cpXXXan itself with Pinto. Basically, Pinto enables you to quickly and easily create whole repositories with arbitrarily (or systematically) defined sets of packages. I'm still figuring out how to actually apply that capability in the real world.

[DrHyde]

Implementing this sort of "pinning" has been on my to-do list for some time - I'll cross it off now and just recommend that people use Pinto :-)

[Khen1950fx]

Excellent work! Just out of curiosity, I looked at the number of dependencies. I counted 204, and these were just on my short list:). Here they are:

#!/usr/bin/perl

use strict;
use warnings;
use CPAN;

CPAN::Shell->install(qw(
MIME::Base64
URI
ExtUtils::ParseXS
Text::Abbrev
Module::CoreList
Module::Load
Params::Check
Module::Load::Conditional
Locale::Maketext::Simple
IPC::Cmd
ExtUtils::CBuilder
Perl::OSType
IO::Dir
Version::Requirements
CPAN::Meta::YAML
JSON::PP
Parse::CPAN::Meta
CPAN::Meta
Exporter
Carp
File::Spec
File::Temp
version
aliased
Declare::Constraints::Simple
Module::Metadata
Attribute::Handlers
Scalar::Util
Params::Validate
Test::Harness
Test::More
URI::FromHash
Test::use::ok
Path::Class
Sub::Uplevel
Test::Exception
Carp::Clan
MooseX::Types
Test::XML::Compare
Any::Moose
IO::Moose
Set::Object
Test::Moose
Devel::GlobalDestruction
Sub::Name
Eval::Closure
Params::Classify
parent
Module::Runtime
Class::Load
Class::Load::XS
Algorithm::C3
Class::C3
MRO::Compat
Task::Weaken
Package::Stash
Package::Stash::XS
Class::MOP
Locale::BR
MooseX::Types::Path::Class
Data::OptList
Sub::Exporter
Dist::CheckConflicts
Try::Tiny
Test::Fatal
Sub::Install
Params::Util
Package::DeprecationManager
Encode
Test
Text::Wrap
Pod::Escapes
Pod::Simple
Pod::Man
ExtUtils::MakeMaker
namespace::clean
namespace::autoclean
MooseX::Types::URI
WWW::RobotRules
HTTP::Cookies
Net::FTP
Net::HTTP
HTTP::Daemon
Digest::base
Digest::MD5
HTTP::Negotiate
File::Listing
HTML::Tagset
HTML::Parser
Time::Local
HTTP::Date
Compress::Raw::Zlib
Compress::Raw::Bzip2
IO::Uncompress::Inflate
HTTP::Status
Encode::Locale
LWP::MediaTypes
LWP::UserAgent
base
XSLoader
Variable::Magic
B::Hooks::EndOfScope
PerlIO::gzip
Test::Fatal
CPAN::DistnameInfo
URI::Escape
Package::Locator
Class::MakeMethods
Text::Balanced
Parse::RecDescent
Class::Method::Modifiers
strictures
Moo
Digest::SHA1
Class::Base
Class::Inspector
File::ShareDir
IO::Scalar
Filter::Util::Call
YAML
Algorithm::Diff
Text::Diff
Test::Differences
Pod::Usage
List::AllUtils
Text::Reform
Text::Autoformat
Mixin::Linewise::Readers
Config::INI
MooseX::Configuration
String::Format
MooseX::NonMoose
autobox
Perl6::Junction
Moose::Autobox
LockFile::Simple
Module::Find
Class::C3::Componentised
Class::Accessor
Class::Accessor::Chained::Fast
Data::Page
Context::Preserve
Storable
Clone
Hash::Merge
Class::Accessor::Grouped
SQL::Abstract
Config::Any
Data::Dumper
Data::Dumper::Concise
DBI
DBD::SQLite
Scope::Guard
DBIx::Class::Schema
Term::ANSIColor
File::Spec::Native
UNIVERSAL::can
UNIVERSAL::isa
Test::MockObject
Package::Constants
IO::Zlib
Archive::Tar
Archive::Zip
Dist::Metadata
Archive::Extract
Test::Tester
Test::NoWarnings
Test::Deep
Tree::DAG_Node
Devel::StackTrace
Test::Warn
Test::Most
Dist::Requires
IO::Interactive
autodie
Module::Pluggable::Object
Getopt::Long
Getopt::Long::Descriptive
String::RewritePrefix
IO::TieCombine
App::Cmd::Setup
Number::Compare
Text::Glob
CPAN::Checksums
Module::Signature
Readonly
Readonly::XS
IO::String
File::Path
Time::HiRes
IPC::Run3
Probe::Perl
Test::Script
File::Which
File::HomeDir
Path::Class::File
Pinto::Types
MooseX::Types::Moose
IO::Pty
Math::Round
DateTime::Locale
Test::Requires
Class::Singleton
SQL::Translator
Pinto));