Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Unpredictable output form Text::ExtractWords

by Roger (Parson)
on Aug 16, 2005 at 14:44 UTC ( #484163=note: print w/ replies, xml ) Need Help??


in reply to Unpredictable output from Text::ExtractWords

I had a quick look at the code out of interest. The problem occurs in the str_normalize function. The code is buggy and it does not check boundary conditions properly.

Below is the line of the code that caused the problem.

if(isalpha(*(s-1)) && strchr(chrsep, *s) && isalpha(*(s+1))) { if(space_words(s, *s)) { char c = *s; while(*s) { if(*s == c) s++; else if(!isalpha(*s)) break; *p = *s; s++; p++; } } }
The code detects a possible space character, and tries to validate it. But the single character 'I' and the spaces surrounding it caused logical malfunction in the space_words function. Thus <space> is not recognized as separator for the words.

My advise is to ditch this module, and use simple Perl implementation. The single line below does pretty much what the ExtractWords.xs code does in hundred lines of C.
#!/usr/bin/perl use strict; use warnings; use Data::Dumper; my @strings = ( q|one two 'three' one|, q|yes i am 'sme'|, q|x21 y z|, ); for my $str (@strings){ my %hash; for ( map { length >= 3 ? $_ : () } $str =~ /(\w+)/g ) { $hash{$_}++; } print Dumper(\%hash); }
And the output is
$VAR1 = { 'three' => 1, 'one' => 2, 'two' => 1 }; $VAR1 = { 'yes' => 1, 'sme' => 1 }; $VAR1 = { 'x21' => 1 };


Comment on Re: Unpredictable output form Text::ExtractWords
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://484163]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (6)
As of 2015-08-02 14:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The oldest computer book still on my shelves (or on my digital media) is ...













    Results (2 votes), past polls