Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: list of unique strings, also eliminating matching substrings

by GrandFather (Sage)
on May 21, 2011 at 06:02 UTC ( #906045=note: print w/replies, xml ) Need Help??


in reply to list of unique strings, also eliminating matching substrings

When you say "about 300 characters long", what is the actual range? Are there any constraints on where a substring may match a larger string? Can there be exact matches within a set of strings and if so should duplicates be removed?

Update: length question already answered I see.

Update: and the key question I didn't ask: how many strings of the original 100,000 do you expect you might end up with after duplicates and substrings are removed?

True laziness is hard work
  • Comment on Re: list of unique strings, also eliminating matching substrings

Replies are listed 'Best First'.
Re^2: list of unique strings, also eliminating matching substrings
by lindsay_grey (Novice) on May 30, 2011 at 22:42 UTC

    i don't think there are any constraints on where a substring may match a larger string.

    there can be exact matches. we are removing those using the unique function.

    for the current set of sequences (assuming the program is working correctly), we go from 206,737 sequences to 166,089. not sure how many are removed because they are exact matches and how many are removed because they match a substring of a larger sequence. i will add that check.

    i think the program is working correctly, but it takes a long time, longer than it seems it should take. (~5 hours on a pretty fast computer) getting the unique sequences is fast, but we have an additional step where we go back and retrieve the sequence name and then do the substring comparison. it is these two steps that are so slow.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://906045]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (7)
As of 2020-12-03 19:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How often do you use taint mode?





    Results (57 votes). Check out past polls.

    Notices?