Re: Comparing a list to a tab delimited text file

[Laurent_R]

It depends how large exactly a "very large file" is.

The typical way to do this type of work is to read your tab delimited text file and to store words you're looking for in a hash, and then to look for the word in the hash. The reason this is the good way to do that is that hash lookup is extremely fast (and does not depend on the size of the hash). Now, of course, this works only if the hash does not grow too large to fit in memory. So please provide more information about the size of your files and what you would need to store in addition to the key.

It would be good if you could provide a small extract of both files, showing cases matching your searches.

[Azaghal]

I apologise for the delay, and hope you're still willing to help me.

I understand you need more details about my script and resources, here they are :

- 8Go RAM (that is a problem for me now, as I tend to write quite greedy code, I'm getting out of memory errors)

- a 70Ko XML file, almost 2M lines long, each line corresponding to one word / node, like this :

<DocumentSet>
<document>
<w lemma="appeler" type="VER:pres">appelle</w>
<w lemma="quand" type="KON">quand</w>
<w lemma="gronder" type="VER:infi">gronder</w>
</document>
</DocumentSet>

- a 10 Ko tabulation separated text file, 150k lines long, looking like this :

tunisiennes    tynizjEn    tunisien    ADJ    f    p    0,3    3,51   
+ 0    0,2    0,2    undef
remplît    R@pli    remplir    VER    undef    undef    61,21    81,42
+    0    0,2    0,2    "sub:imp:3s;"
remuons    R°my§    remuer    VER    undef    undef    24,42    62,84 
+   0,2    0    0,2    "imp:pre:1p;ind:pre:1p;"
remuât    R°m8a    remuer    VER    undef    undef    24,42    62,84  
+  0    0,2    0,2    "sub:imp:3s;"
renaudant    R°nod@    renauder    VER    undef    undef    0    2,64 
+   0    0,2    0,2    "par:pre;"
ébouriffées    ebuRife    ébouriffé    ADJ    f    p    0,22    3,45  
+  0    0,2    0,2    undef
rendissent    R@dis    rendre    VER    undef    undef    508,81    46
+8,11    0    0,2    0,2    "sub:imp:3p;"

I'm using the XML::Twig module to go through the XML tree and modify nodes. I use a foreach $w instruction to loop through each <w> node and then check if its content matches a word from the first column of the tab document. If so, I want to add some attributes from the other columns to the XML node for a result like this :

<w conjugaison="imp:pre:2s;ind:pre:1s;ind:pre:3s;sub:pre:1s;sub:pre:3s
+;" genre="" lemma="appeler" nombre="" type="VER:pres">appelle</w>

    <w genre="" lemma="quand" nombre="" type="KON">quand</w>

    <w conjugaison="inf;" genre="" lemma="gronder" nombre="" type="VER
+:infi">gronder</w>

Ask me for more info if needed.

[Laurent_R]

OK, if I understand well, what you need to do is to store the content of your tab separated file into a hash where the key would be the first column of each line of this file and the value the rest of the line (or the full line, or an array of the fields, whatever suits best your needs). Your tab separated file is relatively small and should fit into memory.

After having done that, you read your XML file and, for each node of interest, you look up the hash to see if the word exists. If it does, you retrieve the information from the corresponding value and update your XML content.

Pseudo-code for building the hash:

my %hash;
open my $TAB, "<", $tab_file or die "Cannot open $tabfile $!";
while (my $line = <$TAB>) {
    my $key = (split /\t/, $line)[0];
    $hash[$key] = $line;
}
close $TAB;

Then, when reading the XML file, you lookup the word of interest in the hash:

if (exists $hash{$xml_word}) {
    my $line = $hash{$xml_word};
    # now do what you want with $line
}

You could also preprocess the $line when reading the tab separated file and store in the hash not the full line, but only the part of the data that you need, in a format that makes it easier to use (perhaps as an array ref, for example). Whether this is a good idea depends on your data (are you likely to user almost all entries of the TSV file, or only a few of them? Are you likely to use the TSV entries several times? and so on).

[Azaghal]

Thanks a lot for this answer, it works perfectly and my script now runs fast enough, no need to preprocess $line.

For future reference there's just one typo in your pseudo code, $hash[$key] should be $hash{$key}.