comment on

hello again starbolin, many many thanks for the reply - since i am new to 1. perl 2. this board - i will need some help & time to get involved with both. here the code - and my question - can i apply this code to the whole board. In order to get a "Copy" of the board with category 17 and category 3 .... http://www.nukeforums.com/forums/viewforum.php?f=17 http://www.nukeforums.com/forums/viewforum.php?f=3 thx for the help and the reply! regards perl lover girl (the girl that loves perl so much and so hard - it is soo sweeet to get into this ;-) you cannot imagine ]


#!/usr/bin/perl
use strict;
use warnings;

use LWP::RobotUA;
use HTML::LinkExtor;
use HTML::TokeParser;
use URI::URL;

use Data::Dumper; # for show and troubleshooting

my $url = "http://www.nukeforums.com/forums/viewforum.php?f=17";
my $ua = LWP::RobotUA->new;
my $lp = HTML::LinkExtor->new(\&wanted_links);

my @links;
get_threads($url);

foreach my $page (@links) { # this loops over each link collected from
+ the index
    my $r = $ua->get($page);
    if ($r->is_success) {
        my $stream = HTML::TokeParser->new(\$r->content) or die "Parse
+ error in $page: $!";
        # just printing what was collected
        print Dumper get_thread($stream);
        # would instead have database insert statement at this point
     } else {
        warn $r->status_line;
     }
}

sub get_thread {
    my $p = shift;
    my ($title, $name, @thread);
    while (my $tag = $p->get_tag('a','span')) {
        if (exists $tag->[1]{'class'}) {
            if ($tag->[0] eq 'span') {
                if ($tag->[1]{'class'} eq 'name') {
                    $name = $p->get_trimmed_text('/span');
                } elsif ($tag->[1]{'class'} eq 'postbody') {
                    my $post = $p->get_trimmed_text('/span');
                    push @thread, {'name'=>$name, 'post'=>$post};
                }
            } else {
                if ($tag->[1]{'class'} eq 'maintitle') {
                    $title = $p->get_trimmed_text('/a');
                }
            }
        }
    }
    return {'title'=>$title, 'thread'=>\@thread};
}

sub get_threads {
    my $page = shift;
    my $r = $ua->request(HTTP::Request->new(GET => $url), sub {$lp->pa
+rse($_[0])});
    # Expand URLs to absolute ones
    my $base = $r->base;
    return [map { $_ = url($_, $base)->abs; } @links];
}

sub wanted_links {
    my($tag, %attr) = @_;
    return unless exists $attr{'href'};
    return if $attr{'href'} !~ /^viewtopic\.php\?t=/;
    push @links, values %attr;
}
[download]

again - my question is - can i apply the code on the part of the board. In order to get a "Copy" of the board with category 17 and category 3 .... http://www.nukeforums.com/forums/viewforum.php?f=17 http://www.nukeforums.com/forums/viewforum.php?f=3 look forward to hear from you regards

In reply to Re^3: proof of concept how to run this code by perl_lover_girl
in thread proof of concept how to run this code by perl_lover_girl

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Problems? Is your data what you think it is?
	PerlMonks