Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Extracting the sentences from a webpage

by Anonymous Monk
on Dec 07, 2009 at 22:05 UTC ( #811632=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:


I want to extract the sentences from a webpage and store them in an array. My attempt is below. I'm sure this can be done in a better way and that I haven't thought of all the possibilities. Any advices or tips?

use strict; use warnings; use LWP::Simple; use HTML::Strip; my $url = " +ar-nya-sjukregler"; my $html = get($url); defined $html or die "Can't fetch HTML from: ",$url; my $hs = HTML::Strip->new(); my $clean_text = $hs->parse( $html ); $hs->eof; #sentences spread across multiple lines $clean_text =~ s/(\n)/ /g; $clean_text =~ s/(\r)/ /g; #compact double white space into dots $clean_text =~ s/(\s){2,}/./g; #compact dots into single dots $clean_text =~ s/(\.){2,}/\./g; #split sentances on dots my @sentences = split(/\./, $clean_text); my @realSentences = (); #ignore sentences with single words foreach my $sentence (@sentences) { if(($sentence =~ tr/ //)>1) { push @realSentences, $sentence; } } foreach (@realSentences) { print "$_\n"; }

Replies are listed 'Best First'.
Re: Extracting the sentences from a webpage
by kyle (Abbot) on Dec 07, 2009 at 22:31 UTC

    I've never used Text::Sentence or Lingua::EN::Sentence, but they look like good places to start.

    It looks as if you're trying to take a double space as the end of a sentence. Is that a special feature of your input text?

    I'd do your @realSentences filtering with grep.

    my @realSentence = grep { / / } @sentences;

    In fact, from the split on down could be reduced to:

    print "$_\n" for grep { / / } split /\./, $clean_text;
Re: Extracting the sentences from a webpage
by Lu. (Hermit) on Dec 07, 2009 at 22:35 UTC
    Hi, there First, not all dots are at the end of sentences. Example : "P. Howard was back in the U.S.S.R, with only 52.4 roubles in his pockets..." Second, why reinvent the wheel ?
      a little more accurate is: split /\. /, $text

      but yeah, don't re-invent...

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://811632]
Approved by ww
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2021-05-18 14:27 GMT
Find Nodes?
    Voting Booth?
    Perl 7 will be out ...

    Results (180 votes). Check out past polls.