Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Building a local 'ppm' repository (Windows)

by LittleGreyCat (Scribe)
on Jul 05, 2006 at 10:35 UTC ( #559293=perlquestion: print w/replies, xml ) Need Help??

LittleGreyCat has asked for the wisdom of the Perl Monks concerning the following question:

Following on (because I have seen other similar queries) from Installing CPAN modules without using 'ppm' I concluded that CPAN and Windows don't mix so I needed a local 'ppm' repository:

I was pointed to http://aspn.activestate.com/ASPN/Downloads/ActivePerl/PPM/Zips

I used a download manager to download all the ZIP archives intending to unzip them in one massive WinZip exercise and create my own offline repository.

For some obscure reason:
. the ZIP files available from the ActiveState site are version numbered
. the 'tar.gz' files within the ZIP are version numbered
. the 'ppd' files within the ZIP are NOT version numbered.

e.g.
ABI.zip contains
. ABI.ppd
. ABI.tar.gz

ABI-0.01.zip contains
. ABI.ppd
. ABI-0.01.tar.gz

Because both versions have the same name for the 'ppd' file you can't just download all the ZIP files then unzip them to create your repository. (Been there, done that, no T shirt)

I ended up having to write a Perl ditty to unzip each archive to a work area and rename the 'ppd' file to match the 'zip' file name before adding it to the repository.

Oh, and I had to get '7-zip' to be able to unzip from the command line.

All in all, what should have been a download and a quick unzip turned into a major exercise.
I still have **NO IDEA** why the 'ppd' files are not version numbered.

Can any worthy Monk shed light on this curious situation?
Or perhaps point me to a consolidated 'ppm' archive (which I have so far failed to find).

  • Comment on Building a local 'ppm' repository (Windows)

Replies are listed 'Best First'.
Re: Building a local 'ppm' repository (Windows)
by Herkum (Parson) on Jul 05, 2006 at 13:57 UTC

    To answer your question about the ppd file lets look at the content of one.

    <?xml version="1.0" encoding="UTF-8"?> <SOFTPKG NAME="ABI" VERSION="0,01,0,0"> <ABSTRACT>Perl module to parse chromatogram files generated by Appli +ed Biosystems (ABI) automated DNA sequencing machine.</ABSTRACT> <AUTHOR>Malay Kumar Basu (mbasu@ncbi.nlm.nih.gov)</AUTHOR> <IMPLEMENTATION> <ARCHITECTURE NAME="MSWin32-x86-multi-thread-5.8" /> <CODEBASE HREF="MSWin32-x86-multi-thread-5.8/ABI-0.01.tar.gz" /> <OS NAME="MSWin32" /> </IMPLEMENTATION> <TITLE>ABI</TITLE> </SOFTPKG>

    The ppd file is similiar to CPAN's meta.yml file. It is used by ppm to get information about the package, what platform the package has been bundled for (since different OS's use different compilers) and the version of the software you want to install.

    A don't know of a ppm utility for making sure that you can make yourself a personal repository with only the most current modules. Personally I don't like ppm at all, it has too many issues and it drives me nuts. It has driven me more towards Linux (and CPAN) more than anything else has been able too.

    Edited by planetscape - changed pre tags to code tags

      Thanks - but I don't think this answers my question (or at least the one I thought I asked).
      A full online repository holds several versions of some packages.
      To replicate this in an offline repository you would need to hold several versions of the same package.
      To do this you would need a unique 'ppd' file for each version.

      For some reason the 'ppd' files are not version specific, so you can't directly build an offline repository from the ActiveState ZIP archive.

      I am not looking for a '.ppm' repository with only the most recent modules, I am looking for as near as possible a 'mirror' of the online repository.
      This requires the holding of several versions of modules.

      You can build a CPAN mirror with 'CPAN::Mini'; I just want the same for 'ppm'.

        I am not looking for a '.ppm' repository with only the most recent modules, I am looking for as near as possible a 'mirror' of the online repository.
        I suppose my question would be - why? Okay, I understand the air-gap issue, but why do you eed to have the entire repository on your own machine?

        At present the ppm's available from ActiveState are for those modules which will build succesfully, but even that does not guarantee that they will work flawlessly on Win32. However using CPAN currently gives you an even lower chance of success on Win32 - lots and lots of modules will not build or will not pass their tests.

        A full blown PPM repository is a SOAP server. If you want to make things a little easier you can have a look at this thread http://aspn.activestate.com/ASPN/Mail/Message/ppm/2851563 on the ActiveState ppm mailing list which discusses how to 'fake' a repository. Although I have never had to do it, you can create some other files and the PPM client will look at those and make the indexing and searching easier apparently.

        jdtoronto

        On my local mini-CPAN-mirror I keep only the most recent versions of the modules and have no problems with that, so I'm not sure why you would want to have all versions of the ppm-zips.

        CountZero

        "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Re: Building a local 'ppm' repository (Windows)
by xdg (Monsignor) on Jul 06, 2006 at 01:34 UTC
    Or perhaps point me to a consolidated 'ppm' archive (which I have so far failed to find).

    There's a list of repositories at win32.perl.org. One or more of them might prove useful.

    And if you want to live life on the bleeding edge of win32 and direct compilation from CPAN, I just posted a new release of Vanilla Perl. (Perl 5.8.8 plus bundled compiler for Win32).

    -xdg

    Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Re: Building a local 'ppm' repository (Windows)
by CountZero (Bishop) on Jul 05, 2006 at 21:28 UTC
    The best solution I found is first to install MinGW, then try to use CPANPLUS or --if that doesn't work for some modules-- CPAN and as a last resource, ppm. This allowed me until now to install any Perl module I needed.

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Re: Building a local 'ppm' repository (Windows)
by ldln (Pilgrim) on Jul 05, 2006 at 20:30 UTC
    LittleGreyCat, your idea works nicely if you decide to only to unzip the latest version of each module and make that available in your local disk based ppm repo.

    Do you really need (or even want) to have every version of every module in your repository?

    In the rare case that you need a special version of a module you have that available in .zip file format on disk.

Re: Building a local 'ppm' repository (Windows)
by Cody Pendant (Prior) on Jul 05, 2006 at 21:33 UTC
    Why did you conclude that CPAN doesn't work on Windows?


    ($_='kkvvttuu bbooppuuiiffss qqffssmm iibbddllffss')
    =~y~b-v~a-z~s; print
      He means "most modules in CPAN" rather than the client.

      The client works ok on Windows now, but lots of modules have problems still.
Re: Building a local 'ppm' repository (Windows)
by aplonis (Pilgrim) on Jul 09, 2006 at 02:10 UTC

    When trapped behind a firewall that for some reason blocks PPM, I did two things. First I bought the ActiveState CDROM which is, in itself, a local PPM repository.

    But of course, that only holds for the ActiveState repository, which is by no means complete. For certain other packages I built a small, local PPM by downloading select items from Randy Kobes PPM repository at U Winnipeg. To get what I needed from there I just browsed with MSIE and clicked to download each *.ppd and the *.zip. Then I had to edit the paths on the *.ppd files.

    It took a few trys at building packages to know what all I needed. And that part was mildly tedious. But it was only for a few crypto items not in ActiveState's PPM. So it's not so many. Then I did likewise to get the GD package from Lincoln D Stein's PPM.

    Those few steps in combination (CDROM and manual aquisition) were fairly painless and did all that I needed.

    Gan Uesli Starling
    Kalamazoo MI USA

Re: Building a local 'ppm' repository (Windows)
by Scott7477 (Chaplain) on Apr 04, 2007 at 22:30 UTC
    Your post reminded me of a problem which I have been trying to solve involving extracting URL's pointing to a specific filetype (say a gz archive) from a web page. It turns out that at CPAN there is a page which contains an alphabetical list of all modules, with a hyperlink to the tar.gz file of each module.

    The following code (given appropriate substitution of the command line input; ie gz for pdf) will create a text file with all of the hyperlinks to the tar.gz files:
    use strict; use LWP::Simple; use HTML::SimpleLinkExtor; #usage getfileoftype http://www.example.com pdf > urllist.txt my $url = shift; my $filetype = shift; my $filetypelen = length($filetype); my $offset = -$filetypelen; #print $filetypelen."\n"; #print $offset."\n"; my $fileget = getstore($url,"tempfile.html"); my $extor = HTML::SimpleLinkExtor->new(); $extor->parse_file("tempfile.html"); my @a_hrefs = $extor->a; for my $element (@a_hrefs) { # print $element; # print "\n"; my $suffix = substr($element,$offset,$filetypelen); # print $suffix; # print "\n"; if ($suffix =~ m/$filetype/){ print $element; print "\n"; } }
    Once you have that, you can then use the following code to automatically download all of the modules if you so choose, or whatever subset of the modules you wish to extract from the text file created by the above code:
    use strict; use LWP::Simple; use File::Basename; open (DATA, "urllist.txt") || die "File open failure!"; while (my $downloadurl = <DATA>){ (my $name, my $path, my $suffix) = fileparse($downloadurl); my $finurl = $downloadurl; print $finurl."\n"; my $savefilename = $name.$suffix; print $savefilename; print "\n"; my $status = getstore($finurl,$savefilename); print $status."\n" }
    Both pieces of code work nicely on my WinXP box. Yes, I know that "tempfile.html" gets clobbered, but I was just glad to get this code working, and WinXP doesn't seem to care. In any case, you can now generate a local repository of modules. Hope this helps. Suggestions for improvement in my code are welcome!
      "Your post reminded me of a problem which I have been trying to solve involving extracting URL's pointing to a specific filetype (say a gz archive) from a web page. It turns out that at CPAN there is a page which contains an alphabetical list of all modules, with a hyperlink to the tar.gz file of each module. "

      Back on the subject after a long break!

      Unfortunately, the repository for use by 'ppm' requires a layer above the '.tar.gz' files - a directory of '.ppd' files which describe the packages.

      Therefore I have to create this structure locally to have a local repository.

      I haven't yet found a simpler way than downloading the '.zip' files each of which which contains a '.ppd' file plus a '.tar.gz' file with a subsidiary path name.

      Thanks for the code, though - it could form a basis for downloading all the '.zip' files from the ActiveState website.

      This would in turn allow the creation of an 'all_in_one' utility to download all the '.zip' files from the website, unpack them into the repository, and create the 'package.lst' file.

      The local repository could then be copied to CD and distributed as an off-line 'ppm' repository.

      Cheers

      Dave R

      Nothing succeeds like a budgie with no teeth.
Re: Building a local 'ppm' repository (Windows)
by glasswalk3r (Friar) on Oct 18, 2007 at 15:32 UTC

    This post is quite late, but anyway... it would help any other guy who would like to have a local PPM reposity also...

    Because both versions have the same name for the 'ppd' file you can't just download all the ZIP files then unzip them to create your repository. (Been there, done that, no T shirt)

    Looks like you're just going to the wrong side... you should do the same PPM tool does. For instance, instead of fetching ZIP files you could use the URL http://ppm.activestate.com/PPMPackages/zips/8xx-builds-only/Windows/, which will give you all the PPD's. After that, is just a matter to parse the PPD (which are simple XML files), create a list of tarballs to download. The PPD also gives the architecture, so I just tried http://ppm4.activestate.com/MSWin32-x86/5.8/820/MSWin32-x86-multi-thread-5.8/ and it worked.

    This should solve the issues you had. The major problem, as I see, is combining the repository of ActiveState with others, since ActiveState's does not hold all available modules in PPM format. Winnipeg's repository, for example, has many additional modules plus the same available on ActiveState's.

    Oh, and I had to get '7-zip' to be able to unzip from the command line.

    CPAN has utilities to deal with ZIP files, so using a binary is not necessary.

    Alceu Rodrigues de Freitas Junior
    ---------------------------------
    "You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill
      "you could use the URL http://ppm.activestate.com/PPMPackages/zips/8xx-builds-only/Windows/, which will give you all the PPD's"

      Actually, that is the URL which gave me all my zip files (please see original post).

      I now have some combined code which builds the repository from zip files and then builds the 'package.lst' file from the repository.
      See What is the format of a .packlist file for a PPM repository?.
      This works fine with 5.8 ppm; about to test with 5.10 ppm.
      The code is really nasty, but I probably don't have time to clean it up.
      The code was developed at least partially on work time, so I will have to check to see if there are any issues with posting it.
      If not, I will post it as an example of how not to write Perl (but still get the job done in a slow, rude, crude and disgusting way).

      Please note that one of the 'rude, crude and disgusting' design decisions was to use DOS command line utilities invoked via Perl where possible to minimise coding time.
      A future project (when I have some spare time) is to replace all the DOS dross with Perl code including handling Zip files.
      Until then I will use 7-zip because I can do what I want with a single line of Perl :-).

      Cheers

      Dave R

      Nothing succeeds like a budgie with no teeth.

        I am facing the same firewall issue, and had exactly the same thoughts about setting up a local repo. Did you seek/get permission to post the code? It would save me a whole load of time - and most likely the same for others in our situation.

        Cheers,
        Anthony.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://559293]
Approved by Corion
Front-paged by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (4)
As of 2021-06-13 09:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (54 votes). Check out past polls.

    Notices?