Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

File comparison: not diff

by rschuler (Beadle)
on Apr 09, 2002 at 16:16 UTC ( #157760=perlquestion: print w/replies, xml ) Need Help??

rschuler has asked for the wisdom of the Perl Monks concerning the following question:

I need to compare two files. These files have a key and a value on each line. I need to note when a key is in one file, and not in the other. I also need to note when the keys match, but the values are different. ex. given:
left-file right-file 1,a 1,a 2,b 3,c 4,f 4,g 5,h 5,h
I need 2 absent from right-file, 3 absent from left-file and 4 is different. diff works so long as difference are separated by one or mores lines with a matching key and value. With the above input files diff produces:
< 2,b < 4,f --- > 3,c > 4,g
Which is not quite right. -Bass has the beginnings of the shell of what I need. Being lazy, I was hoping the CPAN would have the algorithm I need, but I have found nothing. Note that the files are big, 1/2 million lines or so and the key and value are actually multiple fields.

Replies are listed 'Best First'.
Re: File comparison: not diff
by thelenm (Vicar) on Apr 09, 2002 at 16:30 UTC
    Here's a little program I whipped up that should do about what you want. Hopefully it should be easy to modify so that you can plug it into whatever your project is. I tested it on the input you gave, but not extensively.
    #/usr/bin/perl use strict; use warnings; @ARGV == 2 or die "Must specify 2 files!\n"; my $afile = shift; my $bfile = shift; my $ahash = make_hash($afile); my $bhash = make_hash($bfile); sub make_hash { my $file = shift; my %hash = (); open IN, "<$file" or die "Can't open '$file': $!\n"; while (<IN>) { chomp; my ($key,$val) = split(/,/,$_); $hash{$key} = $val; } return \%hash; } print "In A hash but not B hash:\n", map {"$_\n"} grep {not exists $bhash->{$_}} keys %$ahash; print "In B hash but not A hash:\n", map {"$_\n"} grep {not exists $ahash->{$_}} keys %$bhash; print "In A hash and B hash but different:\n", map {"$_\n"} grep {exists $bhash->{$_} and $ahash->{$_} ne $bhash->{$_ +}} keys %$ahash;
      Thanks thelenm I like your answer. I did not think to use hashes.

      But I see a problem, the files are large, about 500000 lines, each line is up to 1275 characters (avg about 250). That's about 500000 x 250 x 2 = 238 Mbytes of data in the hashes. The data will grow in the future. I think that will be rather stressful for the machine it's running on:) I think I'll try sorting the files beforehand, and keep only two lines in memory. Something like:

      #code fragment leaving out parseing, reporting, use strict etc. my ($leftkey,$leftvalue) = split_line(scalar <LEFTFILE>); my ($rightkey,$rightvalue) = split_line(scalar <RIGHTFILE>); # work through both files sequentially matching advances by key while(defined($leftkey) and defined($rightkey)) { my $compare = $leftkey cmp $rightkey; if($compare == 0) { if ($leftvalue ne $rightvalue) { value_diff($leftkey, $leftvalue, $rightkey, $rightvalue); } ($leftkey,$leftvalue) = split_line(scalar <LEFTFILE>); ($rightkey,$rightvalue) = split_line(scalar <RIGHTFILE>); next; } elsif($compare > 0) { missing_left($rightkey,$rightvalue); ($rightkey,$rightvalue) = split_line(scalar <RIGHTFILE>); next; } else { missing_right($leftkey,$leftvalue); ($leftkey,$leftvalue) = split_line(scalar <LEFTFILE>); next; } } # are there missing items at end of the files? while(defined($leftkey)) { missing_right($leftkey,$leftvalue); ($leftkey,$leftvalue) = split_line(scalar <LEFTFILE>); } while(defined($rightkey)) { missing_left($rightkey,$rightvalue); ($rightkey,$rightvalue) = split_line(scalar <RIGHTFILE>); }
Re: File comparison: not diff
by graff (Chancellor) on Apr 20, 2002 at 05:47 UTC
    I just uploaded a utility I wrote some time ago to handle this sort of thing on the command line. Look for "cmpcol" by graff in the code catacombs (new posting).

    It probably doesn't do exactly what you want in your particular case, but it does tend to be very handy for a lot of things...

    Here are some sample usages, given your left and right files as input:

    # intersection (-i) of key fields: # (default delimiter is \s+, so use "-d ," here) $ cmpcol -i -d , left right 1 4 5 # print those records with the combined line # content from both files: $ cmpcol -i -d , -lb left right 1,a:<>:1,a 4,f:<>:4,g 5,h:<>:5,h # keys only in "left" file, which is "#1" on the # command line (-x1), showing full line (-l1): $ cmpcol -x1 -l1 -d , left right 2,b # lines with identical content (don't use # the comma delimiter in this case): $ cmpcol -i left right 1,a 5,h # all lines that are not identical (this # always shows which file contained each line): $ cmpcol -x left right 2,b <1 3,c <2 4,f <1 4,g <2 # and so on. I think the "usage" message is # the best part (at least I hope so): $ cmpcol Usage: cmpcol {-i|-u|-us|-x|-x1|-x2} [options] file1[:col#] file2[:col +#] Comparison modes: -i : produce intersection of file1[:col#] and file2[:col#] -u(s) : produce union (and identify sources) -x : produce exclusive-or, identifying sources -x1 (-x2) : produce items unique to file1 (or file2) Options: -l{1,2,b} : print whole lines from file1,file2 or both (default: pri +nt column) -g(v) ptn : grep (-v) -- only compare lines that (don't) contain /pt +n/ -c cchar : ignore material following cchar -d delim : use /delim/ as column separator (default is white-space) use "-d tab" for tab-delimited, "-d dot" for period-deli +mited file1 or file2 may be 'stdin' default ':col#' for comparison is first column from each file, which +is ':1'

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://157760]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2021-04-20 05:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?