Re^4: Searching for Modules and Descriptions on CPAN Remotely

[ghenry]

I am searching for all the Catalyst::Plugin::* modules.

They all appear, but because most of the authors haven't registered the namespace, the descriptions don't show:

use strict;
use warnings;
use Carp;
use Regexp::DefaultFlags;
use CPANPLUS::Backend;

my $cb = CPANPLUS::Backend->new
   or croak "Can't create new CPANPLUS::Backend object";

my @cat_plugins = $cb->search( type => 'module', allow => [ qr/\A Cata
+lyst::Plugin/ ], );

for my $plugin (@cat_plugins) {
    print $plugin->name, "\n";
    print $plugin->description, "\n";
}

[adrianh]

They all appear, but because most of the authors haven't registered the namespace, the descriptions don't show

Ah. Light dawns. Apologies for being dense.

The description field is part of the metadata associated with registered modules - so it won't exist for non registered ones.

Can you give an example of the text that search.cpan is showing for non-registered modules that you thought was the module description?

My guess is that it's something that's been extracted from the SYNOPSIS or DESCRIPTION sections of the POD - in which case you're going to be screen scraping search.cpan, or downloading and extracting the POD from the distribution.

[ghenry]

Yeah, I've now figured that out.

If you search for Catalyst::Plugin, you can see the results.

I just want the module name and the text directly below the link result.

Looks like I'll be turning to WWW::Mechanize after all :-(

[randyk]

At search.cpan.org, as well as within CPAN-Search-Lite, attempts are made to obtain module and package descriptions from the pod, if the CPAN indices don't have them. An alternative to downloading and extracting the pod, or using WWW::Mechanize, is to use the csl_soap script that comes with CPAN-Search-Lite to query, via a soap interface, the database that cpan.uwinnipeg.ca uses. If you set, within the script,

my $soap_uri = 
   'http://theoryx5.uwinnipeg.ca/Apache/InfoServer';
my $soap_proxy = 
   'http://theoryx5.uwinnipeg.ca/cgi-bin/ppminfo.cgi';

then calling the script as:

perl csl_soap --module Catalyst::Plugin::Geography

will fetch the available information from the database, including the description.

The mod_search function of PPM::Make::Util indicates how this soap service can accept an array reference of module names. Currently there isn't support for wild-card searches, but that could easily be added, if there was interest.

[adrianh]

Try this:

#! /usr/bin/perl

use strict;
use warnings;
use CPANPLUS::Backend;

# this will need to be more clever in the general case
# (cross platform paths, odd locations, generated code, etc.)
sub find_pod {
    my ( $dir, $module_name ) = @_;
    my $module_path = join '/', split /::/, $module_name;
    foreach my $extension ( qw( pod pm ) ) {
        my $file = "$dir/lib/$module_path.$extension";
        return $file if -e $file;
    }
    return;
}

sub get_description_from_pod {
    my ( $filename, $module_name ) = @_;
    open my $fh, "<", $filename or die "open failed ($!)\n";
    while ( my $line = <$fh> ) {
        chomp $line;
        return $1 if $line =~ m/^$module_name\s+-\s+(.*)$/s;
    }
    return;
}

sub description {
    my $module = shift;
    return $module->description if $module->description;
    $module->fetch or die CPANPLUS::Error->stack_as_string;
    my $dir = $module->extract or die CPANPLUS::Error->stack_as_string
+;
    return 'unknown' unless my $pod_file = find_pod( $dir, $module->na
+me );
    return get_description_from_pod( $pod_file, $module->name ) 
        or 'unknown';
}

my $cb = CPANPLUS::Backend->new;
my @modules = $cb->search( type => 'module', allow => [ qr/\ACatalyst:
+:Plugin/ ] );
print $_->name, " : ", description( $_ ), "\n" foreach @modules