Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Re^2: Finding multiword units in a corpus

by veg_running (Initiate)
on Nov 17, 2022 at 12:07 UTC ( #11148220=note: print w/replies, xml ) Need Help??

in reply to Re: Finding multiword units in a corpus
in thread Finding multiword units in a corpus

Hi Ken, thank you very much for the suggested improvements. I am going to have a look at all of them.

I since realised I have another problem to solve though: My code only printed the last tag found for a token and not all of them, so if the same token were to appear more than once with a different ID for each occurence in the tagset, I would only get the last one in my output and not a list of all possible IDs assigned to that token.

Any suggestions on how to handle this would be most appreciated.

  • Comment on Re^2: Finding multiword units in a corpus

Replies are listed 'Best First'.
Re^3: Finding multiword units in a corpus
by kcott (Archbishop) on Nov 17, 2022 at 16:19 UTC

    Instead of %words2ids having key-value pairs like

    word => 'id'

    they could perhaps be more like

    word => [qw{id1 id2}]

    I'm not really across the specifics of what you want. You should supply a sample tagset file, a sample input file, and the expected output using those two files.

    My best guess would be changing these lines:

    ... $words2ids{fc $text} = $token; ... s/$re/++$found{fc $1}, "$1 $words2ids{fc $1}"/eg; ...


    ... push @{$words2ids{fc $text}}, $token; ... s/$re/++$found{fc $1}, "$1 @{$words2ids{fc $1}}"/eg; ...

    You should try this for yourself. If you run into difficulties, put together an SSCCE and post it: along with the two sample files and the expected output, this will give us a much better chance of quickly resolving whatever problems you're encountering.

    — Ken

Re^3: Finding multiword units in a corpus
by tybalt89 (Monsignor) on Nov 18, 2022 at 16:55 UTC


    #!/usr/bin/perl use strict; # use warnings; use List::AllUtils qw( rev_nsort_by ); my $corpusfile = '/tmp/d.11148202.corpus'; # FIXME filename my $wordfile = '/tmp/d.11148202.words'; # FIXME filename my %words2ids; { local @ARGV = $wordfile; while( <> ) { my ($key, $value) = split /[\t\n]/; $words2ids{lc $key} .= " $value"; } } my $pat = do { local $" = '|'; qr/(@{[ map quotemeta, rev_nsort_by { length } keys %words2ids ]})/i}; my %found; { local @ARGV = $corpusfile; print s/\b$pat\K/ $found{lc $1}++; $words2ids{lc $1} /ger while <>; } delete @words2ids{ keys %found }; # not found local $, = "\n"; print '',"---------------- Not Found:", sort(keys %words2ids), '';


    Lokho udebe <ZUL-SIL-0016-n> kukwenze isilomo. Ukuzihlola izinyo <ZUL-SIL-0018-n> <ZUL-SIL-0018-n-other> kungahlenga +izinyo lomhlathi <ZUL-SIL-0019-n> yakho. Amakhala agxiza amafinyila. Ulimi <ZUL-SIL-0017-n> amafutha ulimi <ZUL-SIL-0017-n> wonke ULIMI <ZU +L-SIL-0017-n> amabheringi. Sebenzisa amafutha ulimi <ZUL-SIL-0017-n>. Zama ukugwema ukudla okuncinca udebe <ZUL-SIL-0016-n>. ---------------- Not Found: ingemuva lomqala umphimbo

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11148220]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (7)
As of 2023-06-05 10:33 GMT
Find Nodes?
    Voting Booth?
    How often do you go to conferences?

    Results (24 votes). Check out past polls.