Can a Perl module "inspect" itself and provide details about methods and parameters?

[Br'er Rabbit]

I'm suffering through an exercise to do some image processing with ImageMagick by way of PerlMagick (the Image::Magick Perl module).

The documentation is maddening (notwithstanding RLS's praise of it as "unimpressive"), and anyone who wants to do anything other than a simple example, will find theirself roundtripping from pillar to post.

Until reading his comment, I hadn't thought to look at the--shudder--source code,

...but my thoughts were running a different direction.

I seem to remember that Perl modules are "objects" or collections of them and I remember from the many attacks on my comprehension by "object-orientation" that "objects" can be ASKED this-or-that. (It's supposed to be a "feature.")

I also recall that they implement METHODS which can be public or private to make answer to queries.

And so...

...SINCE "objects" and "Perl modules" are way smarter than me, knowing way much more about their innard-parts than me, I would like to ask them:

"Hey, could you please TELL me what PUBLIC methods you have, and WHAT parameters they use, including their DEFAULTS?"

Kinda like: "Hey, why don't you read your own damn source code and highlight the most important bits for me?"

If they would just do that, I would write their answers down and tape them to the stupid monitor and stop all this hunting around.

Thank you for your time. Peace.

[stevieb]

I think it'd be a futile effort to try to sort out what params each subroutine accepts because they can come in and be handled in a myriad of ways.

However, using my Devel::Examine::Subs, you can at least get info about subroutines within a file, module or directory.

Here's a code snip. In this case, I'm in a distribution directory (my Mock::Sub repository):

use warnings;
use strict;

use Devel::Examine::Subs;

my $des = Devel::Examine::Subs->new(file => '.');

my $data = $des->all;

for my $file (keys %$data){
    print "$file:\n";
    for my $sub (@{ $data->{$file} }){
        print "\t$sub\n";
    }
}

Output:

lib/Mock/Sub/Child.pm:
    new
    _mock
    remock
    unmock
    called
    called_count
    called_with
    name
    reset
    return_value
    side_effect
    _check_side_effect
    mocked_state
    _no_warn
    DESTROY
    _end
lib/Mock/Sub.pm:
    import
    new
    mock
    mocked_subs
    mocked_objects
    mocked_state
    DESTROY
    __end

You can even get info about each sub. This example is reading just a single file (but it does work with directories and modules, it's just a bit different to extract the data. Docs contain this info.

use warnings;
use strict;
use feature 'say';

use Devel::Examine::Subs;

my $des = Devel::Examine::Subs->new(file => 'lib/Mock/Sub.pm');

my $subs = $des->objects;

for my $sub (@$subs){
    say $sub->name;
    say "\tstart line:   ". $sub->start;
    say "\tend line:     " . $sub->end;
    say "\tline count:   " . $sub->line_count;

    # the following would list out the entire code block
    # for the sub. Commented out for output brevity
    
    #say "\tsub code:";
    #say "\t\t$_" for @{ $sub->code };
}

Snip of output:

import
    start line:   14
    end line:     17
    line count:   4
mocked_objects
    start line:   74
    end line:     82
    line count:   9
new
    start line:   18
    end line:     26
    line count:   9
DESTROY
    start line:   100
    end line:     101
    line count:   2
__end
    start line:   102
    end line:     102
    line count:   1
mocked_state
    start line:   83
    end line:     99
    line count:   17
mocked_subs
    start line:   62
    end line:     73
    line count:   12
mock
    start line:   27
    end line:     61

Works on in-memory modules as well. Send the module name to the file parameter. If the module is installed, we'll load it and read from there:

use warnings;
use strict;

use Devel::Examine::Subs;

my $des = Devel::Examine::Subs->new(file => 'Logging::Simple');

my $subs = $des->all;

print "$_\n" for @$subs;

Output from my Logging::Simple distribution module:

BEGIN
_sub_names
new
level
file
name
timestamp
levels
labels
display
print
child
custom_display
fatal
_generate_entry
_levels
_log_only

Note that the _end() methods are simply placeholders; they allow my editor to fold all of my POD up into the last subroutine so when I go to bottom of the current file, it doesn't take me to the end of my POD.

You could also use PPI to do the same thing. That's what my distribution uses, and added a whole slew of things around it.

[Br'er Rabbit]

@Stevieb

> I think it'd be a futile effort to try to sort out what params each subroutine accepts because they can come in and be handled in a myriad of ways.

This part is just sad: if objects were self-documenting, then they wouldn't have to rely on the existence of (good) external documentation.

"If I ever write one", (he wrote, pompously) "I'll sure gladly provide everything everybody would want to know."

Your code will nevertheless be a Big Help in filling in some holes about What's Actually Callable.

Thank you.

[stevieb]

"This part is just sad: if objects were self-documenting, then they wouldn't have to rely on the existence of (good) external documentation."

When I'm writing against an object, my IDE pops up with a list of valid methods to choose from, and autocompletes if I desire. To boot, once I've typed in the method name, it'll display to me what arguments the method (or function) accepts. Now, if the function/method is accepting a single hash for example, there's no way to know at all what key/value pairs are used within that hash. However, if all of the parameters are individual scalars in a specific order, it will list those (with their names included).

Here's an image showing the IDE displaying a list of available methods to an external class (works on objects all the same).

...and here's an image showing the parameters for an object's method.

A good IDE can help tremendously, especially when you're working with numerous classes at the same time. You can CTRL-click on any variable, method, function or anything, and you can make one click to go to the definition of it immediately.

Rarely do I need to view documentation online, or the command line. Typically in Perl, we gather up all of our parameters at the top of each subroutine, so being able to go directly to the definition of a sub makes it trivial to see exactly how params are being used.

[Your Mother]

With ImageMagic—since that was the genesis of the thread(s)—you have an added complication. You have to document, introspect the Perl interface/bindings and the C libs beneath. Documentation and code are not, and can never be, the same any more than even simple math could be. I don’t think it’s particularly sad. It is what it is. Perl’s deep flexibility, including lack of types, also precludes a perfect solution, maybe even a consistently decent one, to this kind of introspection. Temporarily suspending the excellent, at least interesting, ideas of affordance and literacy in code; a tool that is also its own manual will end up cumbersome and awkward as both. XML and XSLT come to mind for me here.

[Br'er Rabbit]

@AllMonksWhoTookTheTime

Thank you, esp. for explaining what must seem obvious to you by now. (I hope this thread survives: at my age, I'm sure I'll be back to read it... ;^) )

Note @Stevieb: Re: Devel::Examine::Subs had to install a good number of other modules first, so out-of-touch was my CPAN, and now am only failing t/42 and t/43. Clearly, from you own observation, presaging others', about the "uncertainty" associated with retrieved parameters, the pressure to have and use the tool is off. I'll give it a few more good tries, but will otherwise take the hint and move on and look for my answers in the source. Thank you.

[hippo]

I seem to remember that Perl modules are "objects" or collections of them

This isn't necessarily the case. Some Perl modules are object-orientated and are classes (not "objects" which are things instantiated from the classes) but equally some Perl modules are not object-orientated at all. Some (like the venerable CGI.pm) can be treated either as a class or not, depending on the calling code by having both an OO interface and a procedural interface within the same module.

[stevieb]

WiringPi::API is exactly like that. It has both an Object Oriented interface, as well as a procedural interface, and both interfaces can be used at the same time, if desired.

[GrandFather]

Out of the box Perl provides very light weight support for OO. As with most things Perl you can use it to create pretty much whatever you need. So in this specific case it comes down to "how was Image::Magick written" and for the most part the answer is "not in Perl". The Perl code is a fairly thin wrapper around the underlying library so even in the best case introspection of the Perl code isn't going to be a great help.

But it's worse than that. Even languages that provide introspection can't tell you much about semantics. Sure, you may be able to find out about methods that can be called and maybe even about the parameters they are expecting, but there is a large gap between that an knowing what those parameters mean, or how they interact, or the order methods should be called or a huge range of other vital stuff you need to know to use an API - OO or otherwise.

You may find it more helpful to ask for help than bemoaning a lack of good documentation and trying to run an endgame around the documentation issue.

[stevieb]

Good point on not all Perl code is actually written in Perl. That's definitely something that I should have clued in on considering half the "Perl" code I've written the last two years has been C, C++ and XS :)

[swl]

Just a couple of data points, as others have discussed the overall issues with XS, AUTOLOAD and the like. Neither will process argument lists, so they only address part of the question.

Devel::Symdump will give a huge amount of information about packages, but I prefer Class::Inspector these days as it is simpler to use.

use Devel::Symdump;
my $obj = Devel::Symdump->rnew( $package_name ); 
my @methods = $obj->functions();
#  if you are sure private methods only start with underscore...
my @public_methods = grep {$_ !~ /^_/} @methods;

and

use Class::Inspector;
my $methods = Class::Inspector->methods( $package_name );
my $public_methods = Class::Inspector->methods( $package_name, 'public
+' );

[Your Mother]

Fun. Play stub–

use 5.10.0;
package DoReMe {
    use Moo;
    use namespace::autoclean; # Redacts introspection output significa
+ntly!
    has "solfege" => is => "lazy";
}

use Class::Inspector;
my $methods = Class::Inspector->methods("DoReMe");
say join $/, @{$methods};

say "--";

use Devel::Symdump;
my $obj = Devel::Symdump->rnew("DoReMe");
say join $/, $obj->functions;

BUILDALL
BUILDARGS
DEMOLISHALL
does
meta
new
solfege
--
DoReMe::new
DoReMe::solfege

[Anonymous Monk]

Cool! Using a regex stolen from Class::Inspector:

CORE functions:

perl '-MData::Dumper()' -e '@_=sort grep {/\A[^\W\d]\w*\z/o} keys %{"$
+{CORE}::"};print Data::Dumper::Dumper\@_'

CORE + Cwd import:

perl '-MData::Dumper()' -MCwd -e '@_=sort grep {/\A[^\W\d]\w*\z/o} key
+s %{"${CORE}::"};print Data::Dumper::Dumper\@_'

Everything:

perl '-MData::Dumper()' -e '@_=sort keys %{"${CORE}::"};print Data::Du
+mper::Dumper\@_'

[LanX]

> CORE functions:

nope these are some plain named symbols in the %main:: STASH since your var $CORE is undef.

Anyway you need to check the {CODE} slot of the glob-values to be sure that they hold a code-ref.

  DB<30> sub test { "bla" }

  DB<31> x map { \&$_ } grep {defined &$_ } values %::
0  CODE(0x3477178)
   -> &main::test in (eval 40)[C:/Perl_524/lib/perl5db.pl:737]:2-2
1  CODE(0x3463838)
   -> &main::dumpvar in C:/Perl_524/lib/dumpvar.pl:474-501
2  CODE(0x3457a78)
   -> &main::dumpValue in C:/Perl_524/lib/dumpvar.pl:33-40
  DB<32>

dumpvar and dumpValue are internal debugger routines

[haukex]

I'm suffering through an exercise to do some image processing with ImageMagick by way of PerlMagick (the Image::Magick Perl module). The documentation is maddening (notwithstanding RLS's praise of it as "unimpressive"), and anyone who wants to do anything other than a simple example, will find theirself roundtripping from pillar to post.

Just as a note, you might be interested in How Geizhals took out ImageMagick before ImageMagick took *us* out.

[Your Mother]

Sidebar on the same topic: GraphicsMagick is, or at least was, a safer and more performant fork of ImageMagick. It’s been several years since I knew that to be a fact however so take it with a pinch of kosher.

[LanX]

Yes and no.

No ...Perl is so dynamic that you will hardly automatically parse all subs without an AI which is more clever than you.

But Yes ... you might built a solution which fits in 80-90 percent of the cases.

For instance is it allowed to dynamically proxy subs with AUTOLOAD and the way args are read from @_ can vary in many ways.

In the case of OO classes you'd also need to check the inheritance tree in @ISA.

And I don't even know which backdoors XS modules are allowed to use.

Saying so,after using a modul it's possible to dynamically introspect its Symbol Table Hash aka STASH.

And it's possible to list all lines involving the @_ array.

Something standard like my ($self,$name) = @_ might be easy to decipher.

But this will only show you the internal variable name and not the description of an argument $name.

Hence this would only be a semi automatic help often requiring human intervention.

But in your case, no way

From my experience with Image Magick this won't help much though, because the underlying C library is pretty inconsistent and Perl might only provide a thin layer.

[Discipulus]

Hello Br'er Rabbit,

> Can a Perl module "inspect" itself and provide details about methods..

Yes! You can parse a perl document with PPI two oneliners just to get started:

# a bare package provided as string

perl -MPPI -e "print $_->content for @{ PPI::Document->new(\'package M
+yPack{ sub new{ bless{}, shift} }')->find('PPI::Statement::Sub') }"

#output
sub new{ bless{}, shift}


# inspecting Data::Dump module (getting its path from %INC)

perl -MPPI -MData::Dump -e "print $_->content for @{ PPI::Document->ne
+w( $INC{'Data/Dump.pm'} )->find('PPI::Statement::Sub') }; "

#output

sub dump
{
    local %seen;
    local %refcnt;
    local %require;
    local @fixup;
...

You can go further trying to extract arguments received (good luck ;) and maybe you have to learn what PPI PDOM Tree is:

perl -MPPI -MPPI::Dumper -e "PPI::Dumper->new( PPI::Document->new(\'pa
+ckage MyPack{ sub new{my class shift; return bless{},$class}  }') )->
+print"

PPI::Document
  PPI::Statement::Package
    PPI::Token::Word    'package'
    PPI::Token::Whitespace      ' '
    PPI::Token::Word    'MyPack'
    PPI::Structure::Block       { ... }
      PPI::Token::Whitespace    ' '
      PPI::Statement::Sub
        PPI::Token::Word        'sub'
        PPI::Token::Whitespace          ' '
        PPI::Token::Word        'new'
        PPI::Structure::Block   { ... }
          PPI::Statement::Variable
            PPI::Token::Word    'my'
            PPI::Token::Whitespace      ' '
            PPI::Token::Word    'class'
            PPI::Token::Whitespace      ' '
            PPI::Token::Word    'shift'
            PPI::Token::Structure       ';'
          PPI::Token::Whitespace        ' '
          PPI::Statement::Break
            PPI::Token::Word    'return'
            PPI::Token::Whitespace      ' '
            PPI::Token::Word    'bless'
            PPI::Structure::Constructor         { ... }
            PPI::Token::Operator        ','
            PPI::Token::Symbol          '$class'
      PPI::Token::Whitespace    '  '

L*

[Anonymous Monk]

I seem to remember that Perl modules are "objects" or collections of them and I remember from the many attacks on my comprehension by "object-orientation" that "objects" can be ASKED this-or-that. (It's supposed to be a "feature.")

Sometimes you can learn about objects by dumping them:

perl -MHTTP::Tiny -MData::Dumper -le '$o=HTTP::Tiny->new;print Dumper 
+$o'
$VAR1 = bless( {
                 'timeout' => 60,
                 'keep_alive' => 1,
                 'max_redirect' => 5,
                 'verify_SSL' => 0,
                 'agent' => 'HTTP-Tiny/0.070',
                 'no_proxy' => []
               }, 'HTTP::Tiny' );

Ish:

perl -MHTTP::Tiny -le '$o=HTTP::Tiny->new;print "$_ = $o->{$_}" for so
+rt keys %$o'
agent = HTTP-Tiny/0.070
keep_alive = 1
max_redirect = 5
no_proxy = ARRAY(0x7fc821802fc8)
timeout = 60
verify_SSL = 0

Sometimes, there's no params:

perl -MImage::Magick -MData::Dumper -le '$o=Image::Magick->new;print D
+umper $o'
$VAR1 = bless( [], 'Image::Magick' );

[Fletch]

This "works", but be very aware you're breaking a fundamental tenet of OO (encapsulation) by doing so.

And by so sinning you're lining yourself up for the inevitable great vengance and furious anger which will rain down upon you (c.f. Ezekiel 25:17) when the module author decides that mauve has more RAM and changes to inside-out objects in a subsequent release and whatever hack you came up with depending on the prior internal implementation breaks.

Fine for pedagogical exploration; never do this in real code.