• You have repeated the code twice in your post, can you clean that up please.
• What are you trying to do? This looks like some sort of forum ripping robot?
• As submitted the initialisation of the LWP::RobotUA with the  new method seems to go into lala land and just get lost, but then the invocation of the method does not appear to be in accordance with the documentation.

jdtoronto

I'd suggest reading How (Not) To Ask A Question. There isn't much we can do to help unless you explain what it is you are trying to accomplish rather than just throwing a pile of code at us

hi Cowboy - many many thanks for your input.

i appreciate your idea and any and all help.

i have fixed my message and corrected all the mess.

now i look forward to your help.

my quesiton is; Can i do the grabbing and harvesting of the whole forum here



http://www.nukeforums.com/forums/viewforum.php?f=17
http://www.nukeforums.com/forums/viewforum.php?f=3


with the script that is shown above. is this doable

many many thanks fro all your help

thanks

perl_lover girl

Do you have permission from the forum owner to do what you are attempting to do? Most forum owners would not appreciate the kind of load your script would put on their system. Many forum owners have filters in place to prevent robots like these. You would most likely get your account revoked and a nasty curse placed on your children.

If you can convince the forum owners of a legitimate need they can probably give you direct access to their database. This puts a lesser load on their servers and offers you more options for searching/retrieving.

I am sorry to say that your code is a mess. I don't know where to start. I think that you are biteing off too much at once. I would start with some simpler tasks. Like, just grab one page and print it. You should also write separate little programs to test your subroutines before using them, so that you know if the parameters are passed correctly.

The text you posted here contained double breaks which made the post display funny. Use paragraphs <p> instead.

In spite of the rain I put on your parade we appreciate your participation in our site and hope you continue to visit.

hello again starbolin, many many thanks for the reply - since i am new to 1. perl 2. this board - i will need some help & time to get involved with both. here the code - and my question - can i apply this code to the whole board. In order to get a "Copy" of the board with category 17 and category 3 .... http://www.nukeforums.com/forums/viewforum.php?f=17 http://www.nukeforums.com/forums/viewforum.php?f=3 thx for the help and the reply! regards perl lover girl (the girl that loves perl so much and so hard - it is soo sweeet to get into this ;-) you cannot imagine ]

#!/usr/bin/perl
use strict;
use warnings;

use LWP::RobotUA;
use HTML::LinkExtor;
use HTML::TokeParser;
use URI::URL;

use Data::Dumper; # for show and troubleshooting

my $url = "http://www.nukeforums.com/forums/viewforum.php?f=17";
my $ua = LWP::RobotUA->new;
my $lp = HTML::LinkExtor->new(\&wanted_links);

my @links;
get_threads($url);

foreach my $page (@links) { # this loops over each link collected from
+ the index
    my $r = $ua->get($page);
    if ($r->is_success) {
        my $stream = HTML::TokeParser->new(\$r->content) or die "Parse
+ error in $page: $!";
        # just printing what was collected
        print Dumper get_thread($stream);
        # would instead have database insert statement at this point
     } else {
        warn $r->status_line;
     }
}

sub get_thread {
    my $p = shift;
    my ($title, $name, @thread);
    while (my $tag = $p->get_tag('a','span')) {
        if (exists $tag->[1]{'class'}) {
            if ($tag->[0] eq 'span') {
                if ($tag->[1]{'class'} eq 'name') {
                    $name = $p->get_trimmed_text('/span');
                } elsif ($tag->[1]{'class'} eq 'postbody') {
                    my $post = $p->get_trimmed_text('/span');
                    push @thread, {'name'=>$name, 'post'=>$post};
                }
            } else {
                if ($tag->[1]{'class'} eq 'maintitle') {
                    $title = $p->get_trimmed_text('/a');
                }
            }
        }
    }
    return {'title'=>$title, 'thread'=>\@thread};
}

sub get_threads {
    my $page = shift;
    my $r = $ua->request(HTTP::Request->new(GET => $url), sub {$lp->pa
+rse($_[0])});
    # Expand URLs to absolute ones
    my $base = $r->base;
    return [map { $_ = url($_, $base)->abs; } @links];
}

sub wanted_links {
    my($tag, %attr) = @_;
    return unless exists $attr{'href'};
    return if $attr{'href'} !~ /^viewtopic\.php\?t=/;
    push @links, values %attr;
}

[download]

again - my question is - can i apply the code on the part of the board. In order to get a "Copy" of the board with category 17 and category 3 .... http://www.nukeforums.com/forums/viewforum.php?f=17 http://www.nukeforums.com/forums/viewforum.php?f=3 look forward to hear from you regards

The minimal change consists of changing

my $url = "http://www.nukeforums.com/forums/viewforum.php?f=17";
my $ua = LWP::RobotUA->new;
my $lp = HTML::LinkExtor->new(\&wanted_links);

my @links;
get_threads($url);

foreach my $page (@links) {
    ...
}
[download]

to

my $ua = LWP::RobotUA->new;
my $lp = HTML::LinkExtor->new(\&wanted_links);

my @links;

foreach my $forum_id (17, 3) {
    my $url = "http://www.nukeforums.com/forums/viewforum.php?f=$forum
+_id";
    @links = ();  # yuck!
    my $links = get_threads($url);
    foreach my $page (@$links) {
        ...
    }
}
[download]

As you can see, I don't like your use of the global variable @links. We're forced to provide and initialize a variable that should be local to get_threads. Here's the fix:

#!/usr/bin/perl
use strict;
use warnings;

use LWP::RobotUA;
use HTML::LinkExtor;
use HTML::TokeParser;
use URI::URL;

use Data::Dumper; # for show and troubleshooting

my $ua = LWP::RobotUA->new();

foreach my $forum_id (17, 3) {
    my $url = "http://www.nukeforums.com/forums/viewforum.php?f=$forum
+_id";
    my $links = get_threads($url);
    foreach my $page (@$links) {
        ...
    }
}

sub get_thread {
    ...
}

sub get_threads {
    my $page = shift;

    my @links;
    my $lp = HTML::LinkExtor->new(sub {
        my($tag, %attr) = @_;
        return unless exists $attr{'href'};
        return if $attr{'href'} !~ /^viewtopic\.php\?t=/;
        push @links, values %attr;
    });

    my $request = HTTP::Request->new(GET => $url);
    my $response = $ua->request($request, sub {$lp->parse($_[0])});

    # Expand URLs to absolute ones
    my $base = $response->base;
    return [ map { url($_, $base)->abs } @links ];
}
[download]

Update: Added the minimal change.

Edited by planetscape - Reparented from Reaped: LWP & HTML::LinkExtor running recursively against a bulletin board to Re^3: proof of concept how to run this code

Read more... view votes (25 Bytes)

You don't seem to have answered the questions asked by the other monks above, so I'm not sure if you're actually having permission to do this or not.

If you have permission then it seems that your goal of inserting all the thread data into a database is pointless - it is already in a database on the server!

So, assuming you have permission to copy the details, etc, then you should simply ask the host to export the database for you.

If you don't have permission then spidering the site is probably your best option - if you do it slowly and carefully.

However down that path lies madnes..

i there, since i have no permission, i have to go the way down to Madness. - i count on you - Folks ineed your help. otherwise i really would get mad.. [would oyu do the job with regexpress or with the script above. ths for an answer. regard the perl lover girl love perl so hard

BTW - plz take another forum -. this forum drives me mad - it does so - AND I have other things that make me mad also your s lovergirl

I have gone through your question and follow ups from you and looks to me you are expecting somebody to write a perl code for you

programming doesn't work like that..this is where software development life cycle comes into picture

you got to put down requirements and analyse those requirements and once requirements are confirmed then based on that you got to design the project. Then comes the coding part where one who is developing will decide to use perl or some other to develop then testing implementation and maintainence..

And also please specify when you run this code where is the point you are getting errors and what errors

its very hard to read the code and tell you will it work if I run type of questions

Edited by planetscape - removed unnecessary br tags and replaced with p tags