Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
I had a quick look at the code out of interest. The problem occurs in the str_normalize function. The code is buggy and it does not check boundary conditions properly.

Below is the line of the code that caused the problem.

if(isalpha(*(s-1)) && strchr(chrsep, *s) && isalpha(*(s+1))) { if(space_words(s, *s)) { char c = *s; while(*s) { if(*s == c) s++; else if(!isalpha(*s)) break; *p = *s; s++; p++; } } }
The code detects a possible space character, and tries to validate it. But the single character 'I' and the spaces surrounding it caused logical malfunction in the space_words function. Thus <space> is not recognized as separator for the words.

My advise is to ditch this module, and use simple Perl implementation. The single line below does pretty much what the ExtractWords.xs code does in hundred lines of C.
#!/usr/bin/perl use strict; use warnings; use Data::Dumper; my @strings = ( q|one two 'three' one|, q|yes i am 'sme'|, q|x21 y z|, ); for my $str (@strings){ my %hash; for ( map { length >= 3 ? $_ : () } $str =~ /(\w+)/g ) { $hash{$_}++; } print Dumper(\%hash); }
And the output is
$VAR1 = { 'three' => 1, 'one' => 2, 'two' => 1 }; $VAR1 = { 'yes' => 1, 'sme' => 1 }; $VAR1 = { 'x21' => 1 };


In reply to Re: Unpredictable output form Text::ExtractWords by Roger
in thread Unpredictable output from Text::ExtractWords by wfsp

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (16)
    As of 2014-09-02 15:32 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      My favorite cookbook is:










      Results (25 votes), past polls