Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: How to extract links from a webpage and store them in a mysql database

by g0n (Priest)
on Dec 05, 2006 at 12:53 UTC ( #587856=note: print w/replies, xml ) Need Help??


in reply to How to extract links from a webpage and store them in a mysql database

Hi syedahmed.uos,

Running your code and passing it a single URL, I get a single row in the table, with the base in column 'webpage', and a space separated set of fully qualified links in the 'links' column.

You say that you want to have each link in a separate row of the table. To do that, you'll need to iterate over the list of links. Something like this:

for my $link (@a) { $dbh->do("INSERT INTO htmllinks VALUES ('$base',' $link')"); }

Just making that change won't work because you've set 'webpage' to be the primary key, so you'll get an error because the value for 'webpage' is the same for every row. If you take the 'primary key' out of your table create, and put the loop in as above, you'll get something like this:

+---------------------------------------------------------+ | webpage | htmllinks | +---------------------------------------------------------+ |http://myurl.com | http://myurl.com/link_one/ | |http://myurl.com | http://myurl.com/link_two/ | +---------------------------------------------------------+

Is that what you're looking for?

--------------------------------------------------------------

"If there is such a phenomenon as absolute evil, it consists in treating another human being as a thing."
John Brunner, "The Shockwave Rider".

Replies are listed 'Best First'.
A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://587856]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2020-06-02 17:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?



    Results (19 votes). Check out past polls.

    Notices?