Filepointer of source file inside source filter

[liz]

Executive summary: find out the original file pointer position of a line being filtered in a source filter

One of my projects is load.pm which allow ondemand loading of subroutines, much like AutoLoader.pm. But with the advantage that it doesn't need to autosplit source files first, which means development is much easier (you don't need to install before you can test). This is achieved by recording the seek() location of the start of the subroutine and the number of bytes of source code. Whenever a routine needs to be loaded, the original source-file is opened, seek(ed) to the correct position and then read for the number of bytes. This is then evalled with the right stuff around it.

However, what I dislike about AutoLoader c.s modules, is that the source of the subroutines that need to be loaded, is located after __END__ in the source code. This has the following disadvantages in my opinion:

• It disrupts the "natural" order of the subroutines in the source, whatever "natural" order you may favour.
• You can't immediately see whether a subroutine is loaded on demand: you would have to know whether it is located after an __END__.
• It looks like it is part of the external documentation.

So I would like to implement a module that would allow a subroutine attribute to indicate whether the subroutine should be loaded on demand or not. Something like:

sub foo : ondemand { }

The way I would like to implement this, is with a source filter (for now anyway). And I think I got it all figured out, except for one bit. How to record the file pointer's position in the source file where the subroutine in question starts. I need to have this to allow that part of the source code to be evalled when needed.

I realize that this is basically impossible with stacked source filters, but I think I can make a case that my source filter must to be first one to be activated.

One yucky way to do this, would be to read the original source file for the N lines until where the use load is located (because I can know the line number from the call stak, but I would need the byte offest).

Other suggestions, anyone?

Liz

[bart]

One yucky way to do this, would be to read the original source file for the N lines until where the use load is located (because I can know the line number from the call stak, but I would need the byte offst).

Other suggestions, anyone?

Eh... create an index file, next to the original source file, mapping the line number to the index? It's simple enough — error checking has been dropped, as this is just an example, and error checking distracts from the code that actually matters:

open SRC, "$source";
open IDX, ">$source.offset";
binmode IDX;
while(<SRC>) {
    print IDX pack 'N', tell SRC;
}

I did try it with CGI.pm this way:

use CGI;
$source = $INC{'CGI.pm'};

Now, when you want the start of line 2990 (first line at is at offset 0, and its start offset is not stored):

my $line = 2990;
open IDX, "$source.offset";
binmode IDX;
seek IDX, 4 * ($line - 2), 0;
read IDX, my($packed), 4;
$offset = unpack 'N', $packed;

open SRC, "$source";
seek SRC, $offset, 0;
print "Line $line is:\n", scalar <SRC>;

which, for me, prints

Line 2990 is:
# Globals and stubs for other packages that we use.

This is indeed the line #2990 for my CGI.pm ($CGI::VERSION='2.752').

Note that this is Windows, and I did not bother to use binmode() on the source file. You must be consistent: either use binmode() both on creating the index file and seeking in the source file, or for neither.

My idea is that you have a make-like mechanism, much like Inline, and compare the modification dates on the index file and the module source file, and recreate the index file if it doesn't exist, or is out of date.

[liz]

Hmmm... I guess I should have mentioned I want to do all of this at compile time without the need of any external file store, because otherwise I might as well use AutoLoader ;-).

I guess it will come down to reading the source file until the invoking line is encountered and recording that file pointer. That shouldn't be too bad as the load.pm source filter should be the first for that module, and consequently is located somewhere near the beginning of the source.

Wish someone would have a better idea, though (apart from hacking it into Perl itself, which might be an option).

Liz

[cbraga]

You see, you don't need to split your modules when developing with Autoloader. Your program will work just fine. You only have to split the modules when creating a production version.

Also, your own solution suffers from some of the ills you place on Autoloader, namely, the need for preprocessing the file and somewhere to store its metadata.

You make a point for Autoloader's uncleanliness with having the code put after the __END__ label but your module is also unclean sice you'll need it to be the first source filter to be loaded.

[liz]

Hmmm... I may be missing something. But if I have a module Foo:

package Foo;
use AutoLoader qw(AUTOLOAD);
1;
__END__
sub a {print "a\n" }

and a program:

use Foo;
a();

I get the following error:

Can't locate auto/Foo/autosplit.ix in @INC (@INC contains:
 /usr/local/lib/perl5/5.8.1/darwin-thread-multi
 /usr/local/lib/perl5/5.8.1
 /usr/local/lib/perl5/site_perl/5.8.1/darwin-thread-multi
 /usr/local/lib/perl5/site_perl/5.8.1
 /usr/local/lib/perl5/site_perl
 .) at /usr/local/lib/perl5/5.8.1/AutoLoader.pm line 160.
 at Foo.pm line 3
Undefined subroutine &main::a called at x line 2.

which indicates to me you must install your module before you can test it. Maybe you don't get an error if you've already installed a module that way, but then you're not running the development version, but the installed version!

Or are we talking about different things, as you are talking about "Autoload" and I'm talking about AutoLoader?

Liz

[cbraga]

I guess you're right, I should've typed "Autoloader" and typed "Autoload". I also didn't express myself correctly on the first paragraph. I guess I rushed out and forgot to mention I meant to comment out Autoloader and don't use it when developing the module. Thanks for the -1 though.

[adrianh]

One yucky way to do this, would be to read the original source file for the N lines until where the use load is located (because I can know the line number from the call stak, but I would need the byte offest).

If you're willing to go that far could you do something like (untested):

package MyFilter;
use Filter::Simple;

FILTER {
    while (m/sub \s+ (\w+) \s+ : \s+ method/xg) {
        my ($name, $offset) = ($1, pos);
        $offset += $BYTE_OFFSET_FROM_WHERE_FILTER_WAS_INVOKED;
        warn "ondemand subroutine $name at byte offset $offset\n"
    };
};

.. or am I missing something obvious (probably - since I think source filters are evil and avoid them like the plague ;-)