Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

Good day! I'm trying to write a script that takes a list of words and checks to see if each word exists within another word in the list. The specifics are as follows:

1) The word list contains 640,000 entries

2) A word cannot match itself (ie: "a" cannot match "a")

3) A word cannot match itself as a plural even if it makes a different word (ie: "a" cannot match "as")

4) A word cannot match itself with an apostrophe s "'s" (ie: "a" cannot match "a's" but "a" can match "aa's")

Coming from a mainframe background I am trying to use loops but the performance is horrible. From what i've read, hashing seems to be the way to go but I think I am still implementing this as a loop and getting very poor performance (100 records in 20 seconds).

Here is what i've tried so far:

use warnings; use strict; #define constants my $datapath="F:\\wordsinwords\\"; my $wordfile= $datapath."wordlist.txt"; #define variables my $outrecs=0; my $word; open LOG, ">".$datapath."wordsinwords_LOG.txt" or die $!; select LOG; $|=1; #read the wordlist file into an array open (WORDFILE, $wordfile); chomp (@words = (<WORDFILE>)); close (WORDFILE); #main #coerce the array into a hash %hash = map { $_ => 1 } @words; #search for matches #I have no idea how to put a single #regex together that could meet all of the criteria so #I was going to run this multiple times to target specific #criteria until I found all permutations. foreach $word (keys %hash) { $outrecs++ if /.+$searchword/ ~~ %hash; } #=============================================== # This was an attempt at using an array # but it was also very slow #=============================================== #foreach $word (@words) { # $outrecs++ if ($found) = grep (/.+?$word/, @words); #} #close files & write out completion log print LOG "Created output file with: ".$outrecs." records.\n"; close LOG;
Any suggestions would be greatly appreciated. Thank you!

In reply to Words in Words by sarchasm

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others cooling their heels in the Monastery: (6)
    As of 2015-07-03 17:20 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









      Results (54 votes), past polls