Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Announce: Pinto-0.026 -- Curate Your Own CPAN Repository

by jthalhammer (Friar)
on Dec 07, 2011 at 20:31 UTC ( #942322=perlmeditation: print w/replies, xml ) Need Help??

Hello esteemed Monks,

A few months ago, I was tasked with building a private CPAN for a new client. I had already done this a couple times before using CPAN::Site or CPAN::Mini, but I was never really happy with the results. So this time, I started from scratch. The result is called Pinto, and it is now available on CPAN. Pinto is inspired by CPAN::Mini, CPAN::Mini::Inject, and OrePAN, but adds several interesting features (listed below).

So I invite you to kick the tires on Pinto and give me your feedback. Beware that Pinto has a lot of dependencies, and I'm still not sure what the minimum versions should be. So I recommend brewing a fresh perl or using the -L option with cpanm(1) to install Pinto into a clean (and separate) sandbox. Pinto is still alpha code and subject to change in incompatible ways. And make sure you get the 0.026 version.

  • Pinto supports several usage patterns: With Pinto, you can create a repository that mirrors all the latest distributions from CPAN. Or you can create a repository with just your own private distributions. Or you can create a repository that has all the distributions required for a particular project. Or you can combine any of the above patterns.

  • Pinto supports adding and removing distributions in the repository: Pinto gives you the power to precisely tune the contents of your repository. For example, you can add your own locally patched CPAN distribution, and then remove it when the original author releases a fixed version.

  • Pinto helps you avoid breakage due to upgrades: Pinto lets you pin your repository index to a particular version of a package, so that you can hold that package at the last know working version, while allowing other packages to be upgraded.

  • Pinto can be integrated with your version control system: Pinto can automatically commit to your version control system whenever the contents of the repository changes. This gives you reproducible and identifiable snapshots of your dependencies, and a mechanism for rollback when things go wrong.

  • Pinto makes it easier to build several local repositories: Creating new Pinto repositories is easy, and each has its own configuration. So you can have different repositories for each department, or each project, or each version of perl, or each customer, or whatever you want.

  • Pinto can pull distributions from multiple remote repositories: Pinto can mirror or import distributions from multiple sources, so you can create private (or public) networks of repositories that enable separate teams or individuals to share distributions.

  • Pinto supports team development: Pinto is suitable for small to medium-sized development teams, where several developers might contribute new distributions at the same time. Pinto ensures that concurrent users don't clobber each other.

  • Pinto has a robust command line interface: The pinto-admin command line tool has options to control every aspect of your Pinto repository. It is well documented and behaves in a DWIM fashion.

  • Pinto can be extended: You can extend Pinto by creating Pinto::Action subclasses to perform new operations on your repository, such as extracting documentation from a distribution, or grepping the source code of several distributions.

Pinto might be easiest to explain with a couple command-line demonstrations. In this first demonstration, suppose our application depends on Plack-0.98, which happens to be the latest version at that moment:

# Create an empty CPAN-like repository in directory ~/mypan $ pinto-admin -r ~/mypan create # Get the current version of Plack (and all its dependencies) into the + repository $ pinto-admin -r ~/mypan import Plack # Suppose HTTP::Simple has a bug and you made your own patch as HTTP-S +imple-0.23-PATCHED.tar.gz $ pinto-admin -r ~/mypan add HTTP-Simple-0.23-PATCHED.tar.gz # Point cpan(1) or cpanm(1) at your repository, to install Plack with +your patched HTTP::Simple # cpanm --mirror file:///home/you/mypan --mirror-only Plack # Time passes...and now we decide we want to upgrade the latest Plack, + which is now Plack-2.4 $ pinto-admin -r ~/mypan import Plack-2.4 # It seems that bug in HTTP::Simple has now been fixed. So we can rem +ove our patched version $ pinto-admin -r ~/mypan remove HTTP-Simple-0.23-PATCHED.tar.gz

For this next demonstration, suppose we want to have the latest version of everything in CPAN:

# Create a new repository (or you can use an existing one) $ pinto-admin -r ~/mypan create # Get the latest version of ALL packages from a CPAN (may take a coupl +e hours) $ pinto-admin -r ~/mypan mirror # Now we have a stable snapshot of CPAN we can install from $ cpanm --mirror file:///home/you/mypan --mirror-only Catalyst Moose P +erl::Critic # Time we want to bring our snapshot up to date with the +tip of CPAN $ pinto-admin -r ~/mypan mirror # But wait! The new Moose-2.1 breaks our code. We want to stay with +our old Moose-1.4 $ pinto-admin -r ~/mypan pin Moose-1.4 # More time passes...We've fixed our code to work with Moose-2.1 $ pinto-admin -r ~/mypan unpin Moose

You can actually combine these scenarios in various ways too. At the end of the day, Pinto provides you with a stable, cherry-picked slice of CPAN that you can systematically evolve over time. I'm still figuring out how best to apply that capability. Your insights are much appreciated.

Jeffrey Thalhammer
Imaginative Software Systems

Replies are listed 'Best First'.
Re: Announce: Pinto-0.026 -- Curate Your Own CPAN Repository
by DrHyde (Prior) on Dec 08, 2011 at 14:39 UTC

    I confess to having not read the whole post, but ...

    Pinto helps you avoid breakage due to upgrades ... if you "pin" a module to a particular version, have you solved the dependencies problem - ie, that that version might break with a newer version of one of its dependencies, or that pinning it might make future releases of stuff that depends on it break? I gave quite a lot of thought to this when coming up with cpXXXan, but decided that it was Far Too Hard to actually solve properly in my spare time!

      have you solved the dependencies problem - ie, that that version might break with a newer version of one of its dependencies, or that pinning it might make future releases of stuff that depends on it break?

      No, I'm not that smart. Pinto doesn't figure out the optimal dependency graph for you. It just helps you tune the index so that you can (eventually) discover the optimal graph yourself.

      Pinning a package merely holds the index to a particular version of that package. That version could break with newer dependencies (although I certainly could implement a recursive "pin" that would also pin dependencies). And pinning a package could make it impossible for cpan(1) or cpanm(1) to satisfy a prerequisite. In either case, you won't know until you build.

      So finding the optimal dependency graph is still a trial-and-error process. Manually solving this problem for all of the CPAN is probably not feasible. But I'm hopeful that it is feasible for the average project which only uses a small subset of the CPAN.

      Pinto could be used in conjunction with cpXXXan. For example, I could create a Pinto repository to track CP5.12.0AN, and then pin the Pinto repository index to those versions that are compatible with my application. So I'll end up with an index that is most likely to yield a working build of my application on perl 5.12.0.

      Come to think of it, you probably could implement cpXXXan itself with Pinto. Basically, Pinto enables you to quickly and easily create whole repositories with arbitrarily (or systematically) defined sets of packages. I'm still figuring out how to actually apply that capability in the real world.

      Jeffrey Thalhammer
      Imaginative Software Systems

        Implementing this sort of "pinning" has been on my to-do list for some time - I'll cross it off now and just recommend that people use Pinto :-)
Re: Announce: Pinto-0.026 -- Curate Your Own CPAN Repository
by Khen1950fx (Canon) on Dec 10, 2011 at 22:12 UTC
    Excellent work! Just out of curiosity, I looked at the number of dependencies. I counted 204, and these were just on my short list:). Here they are:
    #!/usr/bin/perl use strict; use warnings; use CPAN; CPAN::Shell->install(qw( MIME::Base64 URI ExtUtils::ParseXS Text::Abbrev Module::CoreList Module::Load Params::Check Module::Load::Conditional Locale::Maketext::Simple IPC::Cmd ExtUtils::CBuilder Perl::OSType IO::Dir Version::Requirements CPAN::Meta::YAML JSON::PP Parse::CPAN::Meta CPAN::Meta Exporter Carp File::Spec File::Temp version aliased Declare::Constraints::Simple Module::Metadata Attribute::Handlers Scalar::Util Params::Validate Test::Harness Test::More URI::FromHash Test::use::ok Path::Class Sub::Uplevel Test::Exception Carp::Clan MooseX::Types Test::XML::Compare Any::Moose IO::Moose Set::Object Test::Moose Devel::GlobalDestruction Sub::Name Eval::Closure Params::Classify parent Module::Runtime Class::Load Class::Load::XS Algorithm::C3 Class::C3 MRO::Compat Task::Weaken Package::Stash Package::Stash::XS Class::MOP Locale::BR MooseX::Types::Path::Class Data::OptList Sub::Exporter Dist::CheckConflicts Try::Tiny Test::Fatal Sub::Install Params::Util Package::DeprecationManager Encode Test Text::Wrap Pod::Escapes Pod::Simple Pod::Man ExtUtils::MakeMaker namespace::clean namespace::autoclean MooseX::Types::URI WWW::RobotRules HTTP::Cookies Net::FTP Net::HTTP HTTP::Daemon Digest::base Digest::MD5 HTTP::Negotiate File::Listing HTML::Tagset HTML::Parser Time::Local HTTP::Date Compress::Raw::Zlib Compress::Raw::Bzip2 IO::Uncompress::Inflate HTTP::Status Encode::Locale LWP::MediaTypes LWP::UserAgent base XSLoader Variable::Magic B::Hooks::EndOfScope PerlIO::gzip Test::Fatal CPAN::DistnameInfo URI::Escape Package::Locator Class::MakeMethods Text::Balanced Parse::RecDescent Class::Method::Modifiers strictures Moo Digest::SHA1 Class::Base Class::Inspector File::ShareDir IO::Scalar Filter::Util::Call YAML Algorithm::Diff Text::Diff Test::Differences Pod::Usage List::AllUtils Text::Reform Text::Autoformat Mixin::Linewise::Readers Config::INI MooseX::Configuration String::Format MooseX::NonMoose autobox Perl6::Junction Moose::Autobox LockFile::Simple Module::Find Class::C3::Componentised Class::Accessor Class::Accessor::Chained::Fast Data::Page Context::Preserve Storable Clone Hash::Merge Class::Accessor::Grouped SQL::Abstract Config::Any Data::Dumper Data::Dumper::Concise DBI DBD::SQLite Scope::Guard DBIx::Class::Schema Term::ANSIColor File::Spec::Native UNIVERSAL::can UNIVERSAL::isa Test::MockObject Package::Constants IO::Zlib Archive::Tar Archive::Zip Dist::Metadata Archive::Extract Test::Tester Test::NoWarnings Test::Deep Tree::DAG_Node Devel::StackTrace Test::Warn Test::Most Dist::Requires IO::Interactive autodie Module::Pluggable::Object Getopt::Long Getopt::Long::Descriptive String::RewritePrefix IO::TieCombine App::Cmd::Setup Number::Compare Text::Glob CPAN::Checksums Module::Signature Readonly Readonly::XS IO::String File::Path Time::HiRes IPC::Run3 Probe::Perl Test::Script File::Which File::HomeDir Path::Class::File Pinto::Types MooseX::Types::Moose IO::Pty Math::Round DateTime::Locale Test::Requires Class::Singleton SQL::Translator Pinto));

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://942322]
Approved by Argel
Front-paged by Old_Gray_Bear
[MidLifeXis]: (5.22 at that time)
[MidLifeXis]: Yeah, I know, really helpful :-)
[Corion]: If all else fails, Spreadsheet::XLSX should be "installable" by manually copying the files, provided that the prerequisites are met
[thezip]: It fails in the "formatted 2-digit numeric" tests
[thezip]: Okay Corion, that's what I wanted to hear. Thanks!
[Corion]: Oh, a test failure... Just force-install it then? --force and potentially --notest to skip the tests alltogether ?
[ambrus]: thezip: is Activestate Perl usable for your scenario instead?
[MidLifeXis]: Yeah, --notest is good, not sure I would do --force.
[thezip]: It fails 2/10 tests in that group. What are the ramifications if I force it? What should I look for for?
[thezip]: I've migrated from ActiveState to Strawberry. No going back...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (13)
As of 2017-03-23 17:14 GMT
Find Nodes?
    Voting Booth?
    Should Pluto Get Its Planethood Back?

    Results (290 votes). Check out past polls.