Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^5: How to extract links from a webpage and store them in a mysql database

by chargrill (Parson)
on Dec 18, 2006 at 18:58 UTC ( [id://590526]=note: print w/replies, xml ) Need Help??


in reply to Re^4: How to extract links from a webpage and store them in a mysql database
in thread How to extract links from a webpage and store them in a mysql database

Here's your first bit of help: I've added use strict; and use warnings; near the top and fixed the resultant errors/warnings. I've run your code through perl tidy (a few times) and fixed up some other errors introduced. I fixed a few quoting and commenting issues introduced possibly by cut-n-paste errors, but definitely by your broken database connection string, which I fixed. I've also terminated your while and for loops (near the end) which may or may not be the right spot to terminate them - I can't actually fully run your code since I don't have all the modules installed nor a database handy at the moment.

I have run it once and passed it a base URL, it spit out a few of the links on the page, so I suppose it's doing something properly.

Try running this modified version and see what happens. If you make any changes, please format it for readability before posting again.

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use HTML::LinkExtor; use URI::URL; use DBI(); my $url = <>; # for instance #my $depth = 0; my @link = (); my $ua = LWP::UserAgent->new; # Set up a callback that collect li +nks my @a = (); sub callback { my( $tag, %attr ) = @_; return if $tag ne 'a'; push( @a, values %attr ); } # Make the parser.Unfortunately, we don't know the base yet (it migh +t be #diffent from $url) my $p = HTML::LinkExtor->new( \&callback ); my $res = $ua->request( HTTP::Request->new( GET => $url ), sub { $p->parse( $_[0] ) } ) ; # Expand all image URLs to absolute ones my $base = $res->base; @a = map { $_ = url( $_, $base )->abs; } @a; # Print them out print join( "\n", @a ), "\n"; my $dbh = DBI->connect( "DBI:mysql:database=gatxp;host=\"\"", "", "" ) +; #$dbh->do(" CREATE TABLE newlinks( md5 INTEGER(100) not null " # ."primary key, webpage VARCHAR(80) not null) "); $dbh->do(" INSERT INTO newlinks VALUES( 'MD5', '0', '$base', '1' ) "); foreach $a (@a) { $dbh->do(" INSERT INTO newlinks VALUES( '', '1', '$a', '0' ) "); } my $sth = $dbh->prepare('SELECT * FROM newlinks') or die " Couldn't prepare statement : " . $dbh->errstr; $sth->execute(); while( my $ref = $sth->fetchrow_hashref() ) { my $link = $ref->{'webpage'}; foreach $link (@link) { my $usa = LWP::UserAgent->new; $p = HTML::LinkExtor->new( \&callback ); my $res = $usa->request( HTTP::Request->new( GET => $link ), sub { $p->parse( $_[0] ) } ); $base = $res->base; @link = map { $_ = url( $_, $base )->abs; } @link; # Print them + out print "$$link \n "; } } $sth->finish(); $dbh->disconnect();

HTH



--chargrill
s**lil*; $*=join'',sort split q**; s;.*;grr; &&s+(.(.)).+$2$1+; $; = qq-$_-;s,.*,ahc,;$,.=chop for split q,,,reverse;print for($,,$;,$*,$/)

Replies are listed 'Best First'.
A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://590526]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others learning in the Monastery: (4)
As of 2024-03-19 06:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found