Scraping with Treebuilder

[lv211]

Hi,

I picked up Spidering Hacks and I've had a problem with hack #19.

I keep getting the errors:

"my" variable @perlbooks masks earlier declaration in same scope at treebuilder.pl line 50.

Bareword "parent" not allowed while "strict subs" in use at treebuilder.pl line 15.

syntax error at treebuilder.pl line 34, near ")

Here is the script:

#!/usr/bin/perl
use strict;
use LWP::Simple;
use HTML::TreeBuilder;

my $url = 'http:www.oreilly.com/catalog/prindex.html';
my $page = get( $url ) or die $!;
my $p = HTML::TreeBuilder->new_from_content( $page );

my @links = $p->look_down(
    _tag => 'a',
        href => qr{^ \Qhttp://www.oreilly.com/catalog/\E \w+$}x
);

my @rows = map { $_->parent-parent } @links;

my @books;
for my $row (@rows) {
    my %book;
    my @cells = $row->look_down( _tag => 'td' );
    $book{title}  = $cells[0]->as_trimmed_text;
    $book{isbn}  = $cells[1]->as_trimmed_text;
    $book{price}  = $cells[2]->as_trimmed_text;
    $book{price} =~ s/^\$//;
    $book{url} = get_url( $cells[0] );
    $book{safari} = get_url( $cells[3] );
    $book{examples} = get_url( $cells[4] );
    push @books, \%book;
}

sub get_url {
    my $node = shift;
        my @hrefs = $node->look_down( _tag => 'a' )
        return unless @hrefs;
        my $url =  $hrefs[0]->attr('href');
        $url =~ s/\s+$//;
        return $url;
}

$p = $p->delete;

{
    my $count = 1;
        my @perlbooks = sort { $a->{price} <=> $b->{price} }
                grep { $_->{title} =~ /perl/i } @books;
        print $count++, "\t", $_->{price}, "\t", $_->{title} for @perl
+books;
}

{
    my @perlbooks = grep { $_->{title} =~ /perl/i } @books;
        my @javabooks = grep { $_->{title} =~ /java/i } @books;
        my $diff = @javabooks - @perlbooks;
        print "There are " .@perlbooks." Perl books and ".@javabooks.
              " Java books. $diff more java than Perl."
}

[liverpole]

Hi lv211,

Yes, it's because you have a couple of errors in your code:

    my @rows = map { $_->parent-parent } @links;

should be parent->parent.  And ...

    my @hrefs = $node->look_down( _tag => 'a' )

is missing a semi-colon ';' at the end.

That should make your code compile cleanly, anyway.  Good luck!

[lv211]

Thanks liverpole.

[liverpole]

My pleasure :)

By the way, I did notice, while trying to run your program, that I was getting an error with the URL you have:

my $url = 'http:www.oreilly.com/catalog/prindex.html';

It doesn't work for me in a browser, either (it's a missing or unavailable page).

The reason I mention it is that the error message from Perl was somewhat cryptic ... so if you get an error retrieving that page, you may want to try another page which you're positive is accessible.

[HuckinFappy]

The first error liverpole identified is easy to find if you turn on warnings (which you are not using in your code):

[10] perl -Mwarnings /tmp/testit.pl
"my" variable @perlbooks masks earlier declaration in same scope at /t
+mp/testit.pl line 50.
Bareword "parent" not allowed while "strict subs" in use at /tmp/testi
+t.pl line 15.
syntax error at /tmp/testit.pl line 34, near ")
        return"
Global symbol "@hrefs" requires explicit package name at /tmp/testit.p
+l line 34.
Execution of /tmp/testit.pl aborted due to compilation errors.

The error stopping you from even compiling though, is the missing semicolon. 13 years of writing perl, and I still find missing semicolons, parens and braces are the hardest errors to find sometimes. I use perltidy now to try and help. For example, running your code through perltidy, I end up with:

sub get_url {
   my $node  = shift;
   my @hrefs = $node->look_down(_tag => 'a') return unless @hrefs;
   my $url   = $hrefs[0]->attr('href');
   $url =~ s/\s+$//;
   return $url;
}

Well, that long line jumped right out at me as being seriously wrong, and it was easy to figure out the fix then.

HTH,
~Jeff