Count number of words on a web page?

[jdlev]

Is there a simple way to count the number of times a word appears on a web page?

[JSchmitz]

#!/usr/bin/perl

$/ = ""; 
$* = 1; 

while (<>) {
#s/-\n//g; 
tr/A-Z/a-z/; 
@words = split(/\W*\s+\W*/, $_); # split into words
foreach $word (@words) {
$wordcount{$word}++; # count the words
}
}
foreach $word (sort keys(%wordcount)) {
printf "%8d\t\t%s\n", $wordcount{$word}, $word;
}


Hope that helps

Jeffery

[cdarke]

Use word boundaries (\b).

$_ ='The,quick,brown;foxy.  Does a lot,of:jumping!';

my @words = split(/\W*\s+\W*/, $_); # split into words

print 'Number of words: '.@words."\n";

gives 4 words. However

my @words = split(/\b\W*/, $_); # split into words

gives 9 words.

[jdlev]

Thanks!

[zentara]

I would add that you might want to extract pure text from the html page, before word counting.... otherwise you will get html artifacts in your text count, like brackets and slashes. Read perldoc -q 'remove html'

Other ideas:

#!/usr/bin/perl
#You could use HTML::TokeParser::Simple and only print text tags.
#almost straight from the TokeParser::Simple POD

use HTML::TokeParser::Simple;
my $p = HTML::TokeParser::Simple->new( $somefile );
while ( my $token = $p->get_token ) {
    print $token->as_is if $token->is_text;
  }

###################################################################
#HTML::Strip - Perl extension for stripping HTML markup from text.

use HTML::Strip;

my $hs = HTML::Strip->new();

my $clean_text = $hs->parse( $raw_html );
 $hs->eof;

###################################################################


sub strip {
   my $html = shift;
   my $p    = HTML::PullParser->new(
       doc  => $html,
       text => 'text',
   );
   my $result = '';
   while ( my $t = $p->get_token ) {
       $result .= $t->[0];
   }
   return $result;
}

##############################################################

#If you just need to strip all the html tags from a page, 
#and are on a platform with lynx, you can use:

#! /usr/bin/perl
use strict;
use warnings;
my $text=`lynx -dump htmlDocument.html`;
print "$text";

##################################################################
#or
lynx -dump htmlDocument.html > htmlDocument.txt

[planetscape]

Depending on your needs, you may wish to take a look at the Ngram Statistics Package, by Ted Pedersen.