Re: Splitting compound (concatenated) words )

[BrowserUk]

in reply to Splitting compound (concatenated) words )

If all the words are correctly spelt and are in your dictionary, then this appears to make a good attempt at many inputs:

#! perl -slw
use strict;

my @w = do{ local @ARGV = 'words.txt'; <> };
chomp @w;

my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwor
+ds';

my @subset = grep{ $s =~ /$_/ } 'a', 'perl', @w;

my $re1 = join '|', sort{ length( $b ) <=> length( $a ) }@subset;
my $re2 = "($re1)?" x 11;

print for grep defined(), $s =~ /^$re2$/;

__END__
C:\test>junk
could
someone
recommend
a
working
perl
module
to
split
concatenated
words

But note: I had to add 'a' & 'perl' which don't appear in my dictionary; and I cheated by hardwiring the number of words (11) to look for.

If I change that to a larger number (say 100), then the results are less good:

C:\test>junk
could
someone
recommend
aw
or
king
perl
module
to
split
concatenated
words

However, if I removed the iffy non-word 'aw' from my rather permissive dictionary, it once again produces the right output, but that just goes to prove how sensitive and dependent the results would be on a good dictionary and correct spelling.

[vit]

Thanks, great code. This is really something and I will probably take advantage of this.

However I would prefer to have some probabilistic approach so that no need in guessing between 11 or 100. There should be some NLP related module, I think.

[BrowserUk]

However I would prefer to have some probabilistic approach so that no need in guessing between 11 or 100.

The value of 100 is just an arbitrary value that just needs to be large enough to encompass the number of words in the input.

I'm not sure you'd get much better results with NLP, but its been a long time since I did anything with that and the art moves on. Good luck.

[vit]

This is not working for me.
I do not understand how

$s =~ /^$re2$/;

can match from the beginning to the end for

my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwor
+ds';

and

## $re2 = (recommend|someone|working|module|words|split|comme|could|..
+.

[BrowserUk]

I do not understand how $s =~ /^$re2$/; can match from the beginning to the end for my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwords'; and ## $re2 = (recommend|someone|working|module|words|split|comme|could|...

If that's what $re2 looks like, then you are not running the code I posted!

It should look like:

(concatenated|concatenate|catenated|recommend|catenate|commend|someone
+|working|catena|module|could|enate|split|words|perl|cate|king|mend|so
+me|word|work|ate|cat|con|daw|end|eon|ere|kin|let|lit|men|mod|one|per|
+rec|som|ted|ten|at|aw|ed|en|er|et|in|it|ki|li|me|mm|mo|na|ne|od|om|on
+|or|os|pe|re|so|to|wo|a)?
... repeat 11 (or more) times.

The way the code:

#! perl -slw
use strict;

my @w = do{ local @ARGV = 'words.txt'; <> };
chomp @w;

my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwor
+ds';
my @subset = grep{ $s =~ /$_/ } 'a', 'perl', @w;
my $re1 = join '|', sort{ length( $b ) <=> length( $a ) }@subset;
my $re2 = "($re1)?" x 11;

print for grep defined(), $s =~ /^$re2$/;

Works is:

1. First it loads the dictionary into @w.
2. Then it filters that array against the input string with grep, to produce @subset

   This is the subset of all words in the dictionary that appear somewhere in the input string.

3. It then build $re1, which is an alternation of all those words, longest first.

   So $re1 looks like this: longestword|longword|shorter|short

4. It then builds $re2 which looks like this:

   (longestword|longword|shorter|short)?(longestword|longword|shorter|sho
   +rt)?(longestword|longword|shorter|short)?...
   

   The effect is that (provided every word in your input is spelt correctly, and appears in the dictionary), is that it will match each longest word in turn.

   Because the repeated regex is conditional ($re1)?, if the number of repeated elements is longer than the number of words in the string, it will just stop matching when it reaches the end of the string, and the redundant captures will return the null string.

   Hence the grep defined().

The results will rarely be perfect, but it depending upon the source of your strings, it might form the basis for further, perhaps statistical, analysis.

I'm curious as hell about the source of the data and the purpose of the exercise?

[vit]

It actually works. I created a dictionary of 50000+ words sorted by popularity. The problem was that I ran your example but I did not have "concatenated" in there.
Thanks for your explanation. I will spend some time to understand this part

print for grep defined(), $s =~ /^$re2$/;

in details. Looks like genius. In terms of both, Perl flexibility and your implementation. Actually it needs

print "$_\n" for grep defined(), $s =~ /^$re2$/;

to print line-wise.

I'm curious as hell about the source of the data and the purpose of the exercise?

The dictionary is the side affect of phrases I retrieved from crawling for my applications. The purpose of the exercise is the following. I created a keyword generator which is a web application. From the logs I found that some people merge words in a seed phrase and my resulted keywords filter fails to establish similarity in these cases. So I need to split. But usually it is only a two reasonable words split so that looks like your algorithm with my dictionary works well.

I can send you a link to the tool, but I am not sure it's a right way to do it through the forum.