Right, i have made the neccessary changes and i think the code works fine now. The problem is i don't quite think the content( format => "text" ); function in the WWW::Mechanize http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize.pm module works. I have used it with google and perlmonks.com and it gives me the whole content. Does anyone else have the same problem or is it something with my code?
Updated code:
#!/usr/bin/perl
use strict;
#Module used to go through the web pages, Can extract links, save them
+ and also strip the HTML of its contents
use WWW::Mechanize;
use URI;
print "WEB CRAWLER AND HTML EXTRACTOR \n";
print "Please input the URL of the site to be searched \n";
print "Please use a full URL (eg. http://www.dcs.shef.ac.uk/) \n";
#Create an instance of the webcrawler
my $webcrawler = WWW::Mechanize->new();
my $url_name = <STDIN>; # The user inputs the URL to be searched
my $uri = URI->new($url_name); # Process the URL and make it a URI
#Grab the contents of the URL given by the user
$webcrawler->get($uri);
# Put the links that exist in the HTML of the URL given by the user i
+n an array
my @website_links = $webcrawler->links($uri);
# The HTML is stripped off the contents and the text is stored in an
+array of strings
my $x = 0;
my @stripped_html;
$stripped_html[$x] = join ' ', $webcrawler->content( format => "text"
+ );
print $stripped_html[$x];
$x = $x + 1;
exit;
Thanks
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|