Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Filepointer of source file inside source filter

by liz (Monsignor)
on Oct 05, 2003 at 09:25 UTC ( #296678=perlquestion: print w/replies, xml ) Need Help??
liz has asked for the wisdom of the Perl Monks concerning the following question:

Executive summary: find out the original file pointer position of a line being filtered in a source filter

One of my projects is which allow ondemand loading of subroutines, much like But with the advantage that it doesn't need to autosplit source files first, which means development is much easier (you don't need to install before you can test). This is achieved by recording the seek() location of the start of the subroutine and the number of bytes of source code. Whenever a routine needs to be loaded, the original source-file is opened, seek(ed) to the correct position and then read for the number of bytes. This is then evalled with the right stuff around it.

However, what I dislike about AutoLoader c.s modules, is that the source of the subroutines that need to be loaded, is located after __END__ in the source code. This has the following disadvantages in my opinion:

  • It disrupts the "natural" order of the subroutines in the source, whatever "natural" order you may favour.
  • You can't immediately see whether a subroutine is loaded on demand: you would have to know whether it is located after an __END__.
  • It looks like it is part of the external documentation.

So I would like to implement a module that would allow a subroutine attribute to indicate whether the subroutine should be loaded on demand or not. Something like:

sub foo : ondemand { }

The way I would like to implement this, is with a source filter (for now anyway). And I think I got it all figured out, except for one bit. How to record the file pointer's position in the source file where the subroutine in question starts. I need to have this to allow that part of the source code to be evalled when needed.

I realize that this is basically impossible with stacked source filters, but I think I can make a case that my source filter must to be first one to be activated.

One yucky way to do this, would be to read the original source file for the N lines until where the use load is located (because I can know the line number from the call stak, but I would need the byte offest).

Other suggestions, anyone?


Replies are listed 'Best First'.
Re: Filepointer of source file inside source filter
by bart (Canon) on Oct 05, 2003 at 12:16 UTC
    One yucky way to do this, would be to read the original source file for the N lines until where the use load is located (because I can know the line number from the call stak, but I would need the byte offst).

    Other suggestions, anyone?

    Eh... create an index file, next to the original source file, mapping the line number to the index? It's simple enough — error checking has been dropped, as this is just an example, and error checking distracts from the code that actually matters:
    open SRC, "$source"; open IDX, ">$source.offset"; binmode IDX; while(<SRC>) { print IDX pack 'N', tell SRC; }
    I did try it with this way:
    use CGI; $source = $INC{''};
    Now, when you want the start of line 2990 (first line at is at offset 0, and its start offset is not stored):
    my $line = 2990; open IDX, "$source.offset"; binmode IDX; seek IDX, 4 * ($line - 2), 0; read IDX, my($packed), 4; $offset = unpack 'N', $packed; open SRC, "$source"; seek SRC, $offset, 0; print "Line $line is:\n", scalar <SRC>;
    which, for me, prints
    Line 2990 is:
    # Globals and stubs for other packages that we use.
    This is indeed the line #2990 for my ($CGI::VERSION='2.752').

    Note that this is Windows, and I did not bother to use binmode() on the source file. You must be consistent: either use binmode() both on creating the index file and seeking in the source file, or for neither.

    My idea is that you have a make-like mechanism, much like Inline, and compare the modification dates on the index file and the module source file, and recreate the index file if it doesn't exist, or is out of date.

      Hmmm... I guess I should have mentioned I want to do all of this at compile time without the need of any external file store, because otherwise I might as well use AutoLoader ;-).

      I guess it will come down to reading the source file until the invoking line is encountered and recording that file pointer. That shouldn't be too bad as the source filter should be the first for that module, and consequently is located somewhere near the beginning of the source.

      Wish someone would have a better idea, though (apart from hacking it into Perl itself, which might be an option).


Development with Autoloader
by cbraga (Pilgrim) on Oct 05, 2003 at 18:24 UTC
    You see, you don't need to split your modules when developing with Autoloader. Your program will work just fine. You only have to split the modules when creating a production version.

    Also, your own solution suffers from some of the ills you place on Autoloader, namely, the need for preprocessing the file and somewhere to store its metadata.

    You make a point for Autoloader's uncleanliness with having the code put after the __END__ label but your module is also unclean sice you'll need it to be the first source filter to be loaded.

      Hmmm... I may be missing something. But if I have a module Foo:
      package Foo; use AutoLoader qw(AUTOLOAD); 1; __END__ sub a {print "a\n" }
      and a program:
      use Foo; a();
      I get the following error:
      Can't locate auto/Foo/autosplit.ix in @INC (@INC contains: /usr/local/lib/perl5/5.8.1/darwin-thread-multi /usr/local/lib/perl5/5.8.1 /usr/local/lib/perl5/site_perl/5.8.1/darwin-thread-multi /usr/local/lib/perl5/site_perl/5.8.1 /usr/local/lib/perl5/site_perl .) at /usr/local/lib/perl5/5.8.1/ line 160. at line 3 Undefined subroutine &main::a called at x line 2.
      which indicates to me you must install your module before you can test it. Maybe you don't get an error if you've already installed a module that way, but then you're not running the development version, but the installed version!

      Or are we talking about different things, as you are talking about "Autoload" and I'm talking about AutoLoader?


        I guess you're right, I should've typed "Autoloader" and typed "Autoload". I also didn't express myself correctly on the first paragraph. I guess I rushed out and forgot to mention I meant to comment out Autoloader and don't use it when developing the module. Thanks for the -1 though.
Re: Filepointer of source file inside source filter
by adrianh (Chancellor) on Oct 05, 2003 at 21:06 UTC
    One yucky way to do this, would be to read the original source file for the N lines until where the use load is located (because I can know the line number from the call stak, but I would need the byte offest).

    If you're willing to go that far could you do something like (untested):

    package MyFilter; use Filter::Simple; FILTER { while (m/sub \s+ (\w+) \s+ : \s+ method/xg) { my ($name, $offset) = ($1, pos); $offset += $BYTE_OFFSET_FROM_WHERE_FILTER_WAS_INVOKED; warn "ondemand subroutine $name at byte offset $offset\n" }; };

    .. or am I missing something obvious (probably - since I think source filters are evil and avoid them like the plague ;-)

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://296678]
Approved by Courage
Front-paged by Courage
and snow settles gently...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (6)
As of 2017-01-19 07:01 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (168 votes). Check out past polls.