Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

regex, find words that occur more than once.

by Anonymous Monk
on Sep 14, 2020 at 12:21 UTC ( #11121725=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, i have a piece of text and want to find all words that occur more than once by using a regex. Here is what i have so far.

use strict; use warnings; my $text = "and him him lad has him done and john has has"; while($text=~/(\w\w\w).*?\g1/g){ print "$1\n"; }

Something is wrong because he can't find the word 'him'. I hope you can help me find the right regex. Ty

Replies are listed 'Best First'.
Re: regex, find words that occur more than once.
by GrandFather (Sage) on Sep 14, 2020 at 12:32 UTC

    You don't tell us why which leaves plenty of room for guessing. If you don't need to use a regex you can get a count of word use like this:

    use strict; use warnings; my $text = "and him him lad has him done and john has has"; my %words; ++$words{$_} for split /\s+/, $text; print "$_: $words{$_}\n" for sort keys %words;

    Prints:

    and: 2 done: 1 has: 3 him: 3 john: 1 lad: 1

    If that's not helpful for your immediate problem you might like to give us some more context.

    Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: regex, find words that occur more than once.
by haukex (Bishop) on Sep 14, 2020 at 12:28 UTC

    The problem is that on the first match, the \w\w\w matches the first and, but the .*? matches " him him lad has him done ", and the \g1 then matches the second and. The regex engine then continues matching after this entire match, i.e. at  john, so it only finds has has, and then it reaches the end of the string so there are no more matches. I assume the point of this exercise is to learn about zero-width lookarounds - read up on them at perlre and try the regex / (\w\w\w) (?= .*? \1 ) /gx.

Re: regex, find words that occur more than once.
by tybalt89 (Prior) on Sep 14, 2020 at 15:13 UTC
    #!/usr/bin/perl use strict; # https://perlmonks.org/?node_id=11121725 use warnings; use List::Util qw( uniq ); my $text = "and him him lad has him done and john has has"; print "$_\n" for uniq $text =~ /(\b\w+\b)(?=.*?\b\1\b)/g;

    Outputs:

    and him has
Re: regex, find words that occur more than once.
by Anonymous Monk on Sep 14, 2020 at 23:34 UTC

    I finished my script like so:

    use strict; use warnings; my %words; my $text = "and him him lad has him done and john has has"; while($text=~/\b(\w+)\b(?=.*\b\1\b)/g){ $words{$1}++; } foreach my $key(keys %words){ my $count = $words{$key} + 1; print "$key: $count\n"; }

    Again thank you for the advice.

      Nice synthesis of the advice you got!

      In production code dealing with many searches on large strings you might want to make the .* match in the look ahead assertion non-greedy (.*?) so a search to the end of the string isn't required. Also, adding sort to (keys %words) gets the words in consistent order making it easier to find particular results if there are a lot of words.

      Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
      Good synthesis, as GrandFather says. A couple of minor stylistic elements you might consider:
      • You should put spaces around your Binding Operators. Regular expressions already look confusing to a lot of people. Same for before the parentheses of your for loop, but that's less of an issue.
      • One might naively expect that the hash in question contains the counts, and you correct that right before output. That could be confusing if the two loops were more complex or separated by a lot of code. Rather than fixing it at the end, you could initialize the value to 1:
        while($text=~/\b(\w+)\b(?=.*\b\1\b)/g){ $words{$1} //= 1; # initialize empty value to one $words{$1}++; }
        See Logical Defined Or and Assignment Operators for details on what I did there.
      • I usually name my hashes based upon how the key is related to the value. In this case, for example, %hits could be natural because when you look at it on the screen, you see the number of hits you got -- $hits{$word} reads to me as hits for word in English. On the other hand, %words sounds like I'm going to get words back, not a number. With that first change, it might be %count, %repetitions or %reps. Perl more than most programming languages is supposed to read following common grammar.
      Full version with suggested modifications:
      use strict; use warnings; use 5.10.0; my %reps; my $text = "and him him lad has him done and john has has"; while($text =~ /\b(\w+)\b(?=.*\b\1\b)/g){ $reps{$1} //= 1; $reps{$1}++; } foreach my $key (keys %words){ say "$key: $reps{$key}"; }

      #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.

        I completely agree with you on the use of white space.

        The defined or initialization ($reps{$1} //= 1;) is completely unnecessary code bloat, partly because the values in the hash are only marginally important. For purposes of answering the question "Has this word repeated" $words{$1} = undef; works just as well and the following increment is not required.

        The post increment ($words{$1}++;)I should have picked up in the OP. It is mindless cargo culting with no redeeming qualities. Unless logic requires otherwise always pre-increment (++$words{$1};. Pre-increment is never slower (and often faster) than post-increment - which is of trivial importance almost always. Much more important is that sticking the operator out the front makes it easier to be seen making the code easier to understand and maintain.

        To my mind the most important aspect of the words hash is that it is a list of words - the keys are more important than the values. The name of the hash tells you that. Especially where a hash is being used to collect unique instances of things, realizing that keys can be the important data and the values are incidental is a liberating break through.

        Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
Re: regex, find words that occur more than once.
by Anonymous Monk on Sep 14, 2020 at 20:06 UTC

    Thank you

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11121725]
Approved by GrandFather
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2020-09-20 08:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If at first I donít succeed, I Ö










    Results (120 votes). Check out past polls.

    Notices?