Filter::Simple bug with here documents

[Ovid]

Though I've already reported this bug, along with a fix, I have no idea how the fix works. I'm completely mystified. Here's a small test case:

package NoFilter;

use Filter::Simple;

FILTER_ONLY code => sub {};

1;

In other words, it filters nothing.

Here's my test code:

#!/usr/bin/perl

use NoFilter;

my $buf = <<HERE;
/
HERE

my $buf2 = <<HERE;
HERE

This generates the error:

substr outside of string at /usr/local/lib/perl5/5.8.7/Text/Balanced.pm line 70.

At first blush it looks like Text::Balanced has a problem with the HERE docs (and historically it has had problems with HERE docs). However, there's a relatively easy fix for this. Opening up Filter::Simple and adding [$_] as the first line of the anonymous subroutine which Filter::Simple::gen_std_filter_for returns:

sub gen_std_filter_for {
    my ($type, $transform) = @_;
    return sub {
        [$_];   # XXX this is the "voodoo" line
        my $instr;

And the problem mysteriously goes away. I have no frickin' idea why. Any clues? I've emailed thedamian about this but frankly, that's one heck of an embarrassing bug fix since it's "voodoo". Is there something bizarelly simple that I'm overlooking?

Side note: you can also have fun by deleting the "/" line in the first here doc. That generates other fun error messages.

This is perl, v5.8.7 built for darwin-2level. I'm using the latest Text::Balanced and Filter::Simple. I can replicate this problem on OS X and Fedora Core 2. A friend of mine can't replicate this on Perl 5.8.5 Redhat 4.0.

[tye]

Don't do that!

FILTER_ONLY will always have bugs. Just don't use it! It is really just that simple.

And the problem mysteriously goes away.

Yes. Didn't you learn anything from people's attempt to use Switch.pm?? Things mysteriously break (and mysteriously unbreak). Just STOP IT!

You actually can write reasonable source filters for Perl. You just need to avoid trying to not have your filter apply to quoted things (or other subsets of Perl syntax). So define your own way to escape things for places when the filter would be a pain. And define your custom syntax so you don't need to escape away from it much.

And never, never use FILTER_ONLY!

[Ovid]

Can you provide some context for these dire warnings?

[tye]

FILTER_ONLY tries to parse Perl code. Did you really not realize this or not realize that such is doomed and thus something quite foolish to include in production code?

[GrandFather]

Curiosity only but, what lead you to try [$_]; there?

[Ovid]

What? You don't just randomly insert [$_] throughout your code to see what happens? :)

After much debugger work (and using the debugger with source filters is not fun), I found the error appeared to be coming from &Text::Balanced::extract_multiple. Since I couldn't get it to generate the error directly, I fell back to using an empty source filter and that kicked up the error. &Filter::Simple::gen_std_filter_for is the only place in Filter::Simple where extract multiple is called:

sub gen_std_filter_for {
    my ($type, $transform) = @_;
    return sub {
        my $instr;
        local @components;
        for (extract_multiple($_,$extractor_for{$type})) {

So naturally, right before that line I added this:

use Data::Dumper;
warn Dumper( [ $_, $extractor_for{$type} ] );

At that point, the error went away, I sneezed and my head imploded (how Britney Spear's head fails to implode when she sneezes is another mystery I haven't solved). More playing around got that down to the "fix" which I posted.

[nexion]

This seems to be an issue with 5.8. I ran in to this issue myself on 5.8 and found that 5.10, at least on my system, does not trigger the bug.