Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re^2: average a column in tab-delimited file

by garyboyd (Acolyte)
on Jan 31, 2012 at 12:10 UTC ( #950963=note: print w/replies, xml ) Need Help??


in reply to Re: average a column in tab-delimited file
in thread average a column in tab-delimited file

once I have my values in an array, how do I find average of all similar values for column 3? eg how do I find all of those that have 34.9 and average the result in column 6?

  • Comment on Re^2: average a column in tab-delimited file

Replies are listed 'Best First'.
Re^3: average a column in tab-delimited file
by Anonymous Monk on Jan 31, 2012 at 14:51 UTC

    Please explain in words, how you would do it, using paper and pencil

      well I've written something that almost does the job, except it goes wrong on the very last line of the file. I read the file into an array and calculate the averages which I then put into a hash and then iterate through the array and lookup values from the hashes. I'm sure there's a more elegant way to do it though....

      #!/usr/bin/perl # Usage: perl average_table.pl -i <input file> use strict; use warnings; use Getopt::Long; my $infile; my @fields; my @array; my @values; my %hash; my $add_fst; my @out; my @count; my $i=1; my @final; my %pval; my %fst; my $prev_loc; my $prev_pval; my $prev_fst; my $prev_join; GetOptions ( "infile=s" => \$infile, ); open INFILE, "<$infile" or die $!; @fields = <INFILE>; foreach (@fields){ @array = split(/\t+/, $_); chomp (@array); my ($name, $chr, $location, $gen, $dom, $pval, $fst) = @array[ +0..6]; my $join = "$chr-$location"; push (@values, $join, $pval); if ($location == $prev_loc){ $pval = $pval + $prev_pval; $fst = $fst + $prev_fst; $i++; } elsif ($location != $prev_loc) { my $pval_average = ($prev_pval / $i); my $fst_average = ($prev_fst / $i); my $join2 = "$prev_join\t$pval_average\t$fst_average"; $pval{$prev_join} = $pval_average; $fst{$prev_join} = $fst_average; push (@final, $join2); $i = 1; } $prev_loc = $location; $prev_pval = $pval; $prev_fst = $fst; $prev_join = $join; } foreach (@fields){ @out = split(/\t+/, $_); chomp (@out); my ($name, $chr, $location, $gen, $dom, $pval, $fst) = @ou +t[0..6]; my $join = "$chr-$location"; my $new_pval = $pval{$join}; my $new_fst = $fst{$join}; print $name."\t".$chr."\t".$location."\t".$gen."\t".$dom."\t". +$new_pval."\t".$new_fst."\n"; }
        it goes wrong on the very last line of the file

        Between the 2 foreach blocks, add these 2 lines of code.

        $pval{$prev_join} = $prev_pval/$i; $fst{$prev_join} = $prev_fst/$i;

        There were alot of warnings about some of your variables not being initialized. To get rid of the warnings, I initialized them with values from the first line in the file. Posting some of that code here.

        open INFILE, "<$infile" or die $!; chomp(my @lines = <INFILE>); close INFILE or die $!; my ($chr, $prev_loc, $prev_pval, $prev_fst) = (split /\t/, $lines[0])[ +1,2,5,6]; my $prev_join = "$chr-$prev_loc"; my $i=1; my %pval; my %fst; foreach (@lines[1 .. $#lines]){ my ($name, $chr, $location, $gen, $dom, $pval, $fst) = split /\t/; my $join = "$chr-$location"; if ($location == $prev_loc){ $pval = $pval + $prev_pval; $fst = $fst + $prev_fst; $i++; } else { $pval{$prev_join} = $prev_pval / $i; $fst{$prev_join} = $prev_fst / $i; $i = 1; } $prev_loc = $location; $prev_pval = $pval; $prev_fst = $fst; $prev_join = $join; } $pval{$prev_join} = $prev_pval/$i; $fst{$prev_join} = $prev_fst/$i; foreach (@lines){ my ($name, $chr, $location, $gen, $dom, $pval, $fst) = split /\t+/ +; my $join = "$chr-$location"; my $new_pval = $pval{$join}; my $new_fst = $fst{$join}; print $name."\t".$chr."\t".$location."\t".$gen."\t".$dom."\t".$new +_pval."\t".$new_fst."\n"; }

        There are 2 different criteria about what you use to create a group to average. In your code, a group to average is determined by the code, if ($location == $prev_loc) .... But you are also joining the $chr and $location keys for the 2 different hashes here. I believe this leaves the door open for difficult to trace behaviors. It works without error with the sample data provided, but it could break down with an actual file.

        Would you want a line like if ($join eq $prev_join)...?

        Why join the chr location columns. Can the same location have different 'chr' columns? If not, there is no advantage to combining them like that. They could be more simply using the location for the 2 hashes here and the keys are simple to sort, the keys being 'location' alone - a floating point number.

        Here is a solution I came up with. The same general idea - but uses the module List::Util, (part of perl core since v5.7.3), to get the sum of the 2 columns.

        I compared the join of (chr, location), to determine the end of a run instead of just location by itself.

        #!/usr/bin/perl use strict; use warnings; use List::Util qw/ sum /; chomp(my $line = <DATA>); my $prev_chr_loc = join "-", (split /\t/, $line)[1,2]; my @lines = $line; while (my $line = <DATA>) { chomp $line; my $chr_loc = join "-", (split /\t/, $line)[1,2]; if ($chr_loc eq $prev_chr_loc) { push @lines, $line; } else { print "$_\n" for compute_avg( @lines ); @lines = $line; } $prev_chr_loc = $chr_loc; } print "$_\n" for compute_avg( @lines ); sub compute_avg { my @lines = @_; my $avg_col5 = sprintf "%.9f", sum(map {(split /\t/)[5]} @lines) / @lines; my $avg_col6 = sprintf "%.9f", sum(map {(split /\t/)[6]} @lines) / @lines; for (@lines) { my ($c5,$c6) = (split /\t/)[5,6]; s/$c5\t$c6$/$avg_col5\t$avg_col6/; } return @lines; }

        Update: Redid substitution for closer spec.

        well I've written something that almost does the job, except it goes wrong on the very last line of the file.

        Did you at any point sit down with pencil and paper, and write out steps to solve this problem as human computer with pen and paper?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://950963]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (8)
As of 2018-11-21 14:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My code is most likely broken because:
















    Results (239 votes). Check out past polls.

    Notices?