Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: stripped punctuation

by thundergnat (Deacon)
on Oct 07, 2005 at 01:31 UTC ( #498103=note: print w/replies, xml ) Need Help??


in reply to stripped punctuation

Some things which often fail to get taken into account:

Words with internal apostrophes. (don't won't, can't shouldn't, you'll, it's, etc.)

Words with non ASCII characters. á, ń, ˙, etc.

This does:

########################################################## #! /usr/bin/perl use warnings; use strict; my $word = qr/(?<!\p{Alnum})\p{Alnum}+(?!\p{Alnum})/; my %count; while (<DATA>) { my $line = lc $_; while ($line =~ /($word('$word)?)/g){ $count{$1}++; } } printf "%15s %5d\n", $_, $count{$_} for sort keys %count; __DATA__ "Hello World!" "Oh poor Yorick, his world I knew well yes I did" "Words with internal apostrophes. (don't won't, can't shouldn't, you'l +l, it's, etc.)" "Seńor Montóya's resüme isn't ápropos."

Replies are listed 'Best First'.
Re^2: stripped punctuation
by thealienz1 (Pilgrim) on Oct 07, 2005 at 05:34 UTC

    The ultimate case would be to handle a word count in multiple languages, btu I believe that there are too many factors to take into account. Thanks for the suggestion though.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://498103]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (8)
As of 2019-11-19 10:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (95 votes). Check out past polls.

    Notices?