Reading the same file from all the subdirectories

Replies are listed 'Best First'.
Re: Reading the same file from all the subdirectories by moritz (Cardinal) on Sep 06, 2012 at 07:04 UTC
I don't see hwhat the code you have posted has to do with your question. I'd write something along the lines of `use autodie; for my $dir (qw/d1 d2 d3 d4 d5/) { open my $LIST, '<', "$dir/list.txt"; # read from $LIST and do something with it close $LIST; }` [download] Perl 6 - the future is here, just unevenly distributed	[reply] [d/l]
Re^2: Reading the same file from all the subdirectories by zing (Beadle) on Sep 06, 2012 at 16:49 UTC
Hi all. The problem is that I have 10 directories(a,b,c,d,e,f,g,h,i,j) in a folder. Each of these directories have 3 files - SUPERLIST_SUBSTRATE, SUPERLIST_PRODUCT and list.txt. The undermentioned code has to parse these three files and generates a matrix for the list.txt and another.txt, and a concatenated (logical "OR" as in "X" OR "Y") matrix of list.txt and another.txt (thus 3 matrices) What I want instead is pairwise selection as following, The code should select list.txt from first directory and then process all the 10 list.txt files one by one from each of the 10 directories. So I'd have 100 output files (10X10). Please help `a/list.txt a/list.txt a/list.txt b/list.txt a/list.txt c/list.txt a/list.txt d/list.txt a/list.txt e/list.txt a/list.txt f/list.txt a/list.txt g/list.txt a/list.txt h/list.txt a/list.txt i/list.txt a/list.txt j/list.txt` [download] ######## CODE.PL ############# #!/usr/bin/perl use strict; use warnings; use List::Util qw/ max /; open my $fh, "<", 'SUPERLIST_PRODUCT.txt' or die $!; my $spr_prod = do {local $/; <$fh>}; close $fh or die $!; my @spr_prod = $spr_prod =~ /\d+/g; open $fh, "<", 'SUPERLIST_SUBSTRATE.txt' or die $!; my $spr_substr = do {local $/; <$fh>}; close $fh or die $!; my @spr_substrate = $spr_substr =~ /\d+/g; my @matrix; my $path = '.'; # (current directory - '.') or path to data files my @file = qw/list.txt another.txt/; for my $file ( @file ) { my %data; my @substrate; open my $fh, "<", "$path/$file" or die "Unable to open $file for reading. $!"; while (<$fh>) { if (/^substrate/) { @substrate = /\d+/g; } elsif (/^product/) { while (/(\d+)/g) { for my $sub (@substrate) { $data{$sub}{$1} = 1 ; } } } else { die "Unknown format $file. $!"; } } close $fh or die "Unable to close $file. $!"; print "Processing file: $file\n"; process(\@spr_prod, \@spr_substrate, %data); push @matrix, \%data; } for my $i (0 .. $#matrix) { for my $j ($i+1 .. $#matrix) { print "Combining $file[$i] and $file[$j]\n"; my %data = combine($matrix[$i], $matrix[$j]); process(\@spr_prod, \@spr_substrate, %data); } } sub process { my ($spr_prod, $spr_subst, %data) = @_; my %seen; my @product = sort {$a <=> $b} grep ! $seen{$_}++, @$spr_prod, map keys %$_, values %data; # to get column width for print my $wid = 1 + max map length, @product; printf "%7s" . "%${wid}s" x @product . "\n", 'prod->', @product; undef %seen; my @substrate = sort {$a <=> $b} grep ! $seen{$_}++, @$spr_subst, keys %data; for my $substrate (@substrate) { printf "%7s", $substrate; printf "%${wid}s", $data{$substrate}{$_} \|\| '-' for @product; print "\n"; } printf "\n%5s\n%5s\n%s\n\n", '^', '\|', 'substrate'; } sub combine { my ($matrix1, $matrix2) = @_; my %new_hash = %$matrix1; for my $substrate (keys %$matrix2) { $new_hash{$substrate}{$_} = 1 for keys %{ $matrix2->{$substrat +e} }; } return %new_hash; } [download] ##########INPUT############### list.txt substrate: 1 2 product: 3 substrate: 6 9 product: 8 10 substrate: 3 product: 6 substrate: 9 product: 5 substrate: 5 product: 2 substrate: 3 product: 9 substrate: 8 product: 9 substrate: 8 product: 1 substrate: 7 product: 11 substrate: 19 product: 17 substrate: 14 product: 13 substrate: 14 product: 11 substrate: 18 product: 19 substrate: 7 14 product: 15 substrate: 7 16 product: 7 17 substrate: 5 product: 6 substrate: 18 15 product: 7 substrate: 7 8 product: 8 18 substrate: 6 product: 9 substrate: 11 product: 12 another.txt substrate[s]: 3649 product[s]: 3419 3648 substrate[s]: 3645 product[s]: 3647 substrate[s]: 3659 product[s]: 3647 substrate[s]: 3675 product[s]: 3674 substrate[s]: 3674 product[s]: 3490 3489 substrate[s]: 3489 product[s]: 3490 substrate[s]: 3490 product[s]: 3485 substrate[s]: 3485 product[s]: 3486 substrate[s]: 3486 product[s]: 3488 substrate[s]: 3488 product[s]: 3487 substrate[s]: 3487 product[s]: 3877 substrate[s]: 3877 product[s]: 3419 substrate[s]: 3182 product[s]: 1875 substrate[s]: 2809 product[s]: 3182 substrate[s]: 3186 product[s]: 2809 [download] ############OUTPUT################# Processing file: list.txt prod-> 1 2 3 5 6 7 8 9 10 11 12 13 15 17 18 19 1 - - 1 - - - - - - - - - - - - - 2 - - 1 - - - - - - - - - - - - - 3 - - - - 1 - - 1 - - - - - - - - 5 - 1 - - 1 - - - - - - - - - - - 6 - - - - - - 1 1 1 - - - - - - - 7 - - - - - 1 1 - - 1 - - 1 1 1 - 8 1 - - - - - 1 1 - - - - - - 1 - 9 - - - 1 - - 1 - 1 - - - - - - - 11 - - - - - - - - - - 1 - - - - - 14 - - - - - - - - - 1 - 1 1 - - - 15 - - - - - 1 - - - - - - - - - - 16 - - - - - 1 - - - - - - - 1 - - 18 - - - - - 1 - - - - - - - - - 1 19 - - - - - - - - - - - - - 1 - - ^ \| substrate Processing file: another.txt prod-> 1875 2809 3182 3419 3485 3486 3487 3488 3489 3490 3647 3648 36 +74 3877 2809 - - 1 - - - - - - - - - + - - 3182 1 - - - - - - - - - - - + - - 3186 - 1 - - - - - - - - - - + - - 3485 - - - - - 1 - - - - - - + - - 3486 - - - - - - - 1 - - - - + - - 3487 - - - - - - - - - - - - + - 1 3488 - - - - - - 1 - - - - - + - - 3489 - - - - - - - - - 1 - - + - - 3490 - - - - 1 - - - - - - - + - - 3645 - - - - - - - - - - 1 - + - - 3649 - - - 1 - - - - - - - 1 + - - 3659 - - - - - - - - - - 1 - + - - 3674 - - - - - - - - 1 1 - - + - - 3675 - - - - - - - - - - - - + 1 - 3877 - - - 1 - - - - - - - - + - - ^ \| substrate Combining list.txt and another.txt prod-> 1 2 3 5 6 7 8 9 10 11 12 13 +15 17 18 19 1875 2809 3182 3419 3485 3486 3487 3488 3489 3490 3 +647 3648 3674 3877 1 - - 1 - - - - - - - - - + - - - - - - - - - - - - - - + - - - - 2 - - 1 - - - - - - - - - + - - - - - - - - - - - - - - + - - - - 3 - - - - 1 - - 1 - - - - + - - - - - - - - - - - - - - + - - - - 5 - 1 - - 1 - - - - - - - + - - - - - - - - - - - - - - + - - - - 6 - - - - - - 1 1 1 - - - + - - - - - - - - - - - - - - + - - - - 7 - - - - - 1 1 - - 1 - - + 1 1 1 - - - - - - - - - - - + - - - - 8 1 - - - - - 1 1 - - - - + - - 1 - - - - - - - - - - - + - - - - 9 - - - 1 - - 1 - 1 - - - + - - - - - - - - - - - - - - + - - - - 11 - - - - - - - - - - 1 - + - - - - - - - - - - - - - - + - - - - 14 - - - - - - - - - 1 - 1 + 1 - - - - - - - - - - - - - + - - - - 15 - - - - - 1 - - - - - - + - - - - - - - - - - - - - - + - - - - 16 - - - - - 1 - - - - - - + - 1 - - - - - - - - - - - - + - - - - 18 - - - - - 1 - - - - - - + - - - 1 - - - - - - - - - - + - - - - 19 - - - - - - - - - - - - + - 1 - - - - - - - - - - - - + - - - - 2809 - - - - - - - - - - - - + - - - - - - 1 - - - - - - - + - - - - 3182 - - - - - - - - - - - - + - - - - 1 - - - - - - - - - + - - - - 3186 - - - - - - - - - - - - + - - - - - 1 - - - - - - - - + - - - - 3485 - - - - - - - - - - - - + - - - - - - - - - 1 - - - - + - - - - 3486 - - - - - - - - - - - - + - - - - - - - - - - - 1 - - + - - - - 3487 - - - - - - - - - - - - + - - - - - - - - - - - - - - + - - - 1 3488 - - - - - - - - - - - - + - - - - - - - - - - 1 - - - + - - - - 3489 - - - - - - - - - - - - + - - - - - - - - - - - - - 1 + - - - - 3490 - - - - - - - - - - - - + - - - - - - - - 1 - - - - - + - - - - 3645 - - - - - - - - - - - - + - - - - - - - - - - - - - - + 1 - - - 3649 - - - - - - - - - - - - + - - - - - - - 1 - - - - - - + - 1 - - 3659 - - - - - - - - - - - - + - - - - - - - - - - - - - - + 1 - - - 3674 - - - - - - - - - - - - + - - - - - - - - - - - - 1 1 + - - - - 3675 - - - - - - - - - - - - + - - - - - - - - - - - - - - + - - 1 - 3877 - - - - - - - - - - - - + - - - - - - - 1 - - - - - - + - - - - ^ \| substrate [download] </code>	[reply] [d/l] [select]
Re: Reading the same file from all the subdirectories by GrandFather (Saint) on Sep 06, 2012 at 05:28 UTC
Something like the following (untested) code perhaps? `opendir my ($scan), $path or die "Can't scan folder $path: $!\n"; while (my $entry = readdir $scan) { next if ! -d "$path/$entry" \|\| $entry !~ /^d[1-5]$/; next if ! -f "$path/$entry/list.txt"; ... }` [download] True laziness is hard work	[reply] [d/l]
Re: Reading the same file from all the subdirectories by pvaldes (Chaplain) on Sep 07, 2012 at 10:31 UTC
The code should select list.txt from first directory and then process all the 10 list.txt files one by one from each of the 10 directories. `chdir "my_parent_dir"; my @files = glob("*/list.txt"); for my $item(@files){ open my $currentfile, '<', $item; .. do something... close $currentfile; } # and go to the next item of the loop` [download]	[reply] [d/l]

I don't see hwhat the code you have posted has to do with your question. I'd write something along the lines of

use autodie;
for my $dir (qw/d1 d2 d3 d4 d5/) {
    open my $LIST, '<', "$dir/list.txt";
    # read from $LIST and do something with it
    close $LIST;
}
[download]

Perl 6 - the future is here, just unevenly distributed

[reply]
[d/l]

What I want instead is pairwise selection as following, The code should select list.txt from first directory and then process all the 10 list.txt files one by one from each of the 10 directories. So I'd have 100 output files (10X10). Please help

a/list.txt a/list.txt 
a/list.txt b/list.txt 
a/list.txt c/list.txt 
a/list.txt d/list.txt 
a/list.txt e/list.txt 
a/list.txt f/list.txt 
a/list.txt g/list.txt 
a/list.txt h/list.txt 
a/list.txt i/list.txt 
a/list.txt j/list.txt
[download]

######## CODE.PL #############
#!/usr/bin/perl
use strict;
use warnings;
use List::Util qw/ max /;

open my $fh, "<", 'SUPERLIST_PRODUCT.txt' or die $!;
my $spr_prod = do {local $/; <$fh>};
close $fh or die $!;
my @spr_prod = $spr_prod =~ /\d+/g;

open $fh, "<", 'SUPERLIST_SUBSTRATE.txt' or die $!;
my $spr_substr = do {local $/; <$fh>};
close $fh or die $!;
my @spr_substrate = $spr_substr =~ /\d+/g;

my @matrix;
my $path = '.'; # (current directory - '.') or path to data files
my @file = qw/list.txt another.txt/;

for my $file ( @file ) {
    my %data;
    my @substrate;
    
    open my $fh, "<", "$path/$file"
        or die "Unable to open $file for reading. $!";
    
    while (<$fh>) {
        if (/^substrate/) {
            @substrate = /\d+/g;
        }
        elsif (/^product/) {
            while (/(\d+)/g) {
                for my $sub (@substrate) {
                    $data{$sub}{$1} = 1 ;
                }
            }
        }
        else {
            die "Unknown format $file. $!";    
        }
    }
    close $fh or die "Unable to close $file. $!";
    
    print "Processing file: $file\n";
    process(\@spr_prod, \@spr_substrate, %data);
    
    push @matrix, \%data;
}

for my $i (0 .. $#matrix) {
    for my $j ($i+1 .. $#matrix) {
        print "Combining $file[$i] and $file[$j]\n";
        my %data = combine($matrix[$i], $matrix[$j]);
        process(\@spr_prod, \@spr_substrate, %data);
    }    
}

sub process {
    my ($spr_prod, $spr_subst, %data) = @_;
    my %seen;
    
    my @product = sort {$a <=> $b}
                  grep ! $seen{$_}++,
                  @$spr_prod, map keys %$_, values %data;
    
    # to get column width for print
    my $wid = 1 + max map length, @product;

    printf "%7s" . "%${wid}s" x @product . "\n", 'prod->', @product;
    
    undef %seen;
    my @substrate = sort {$a <=> $b}
                    grep ! $seen{$_}++,
                    @$spr_subst, keys %data;
                    
    for my $substrate (@substrate) {
        printf "%7s", $substrate;
        printf "%${wid}s", $data{$substrate}{$_} || '-' for @product;
        print "\n";
    }
    printf "\n%5s\n%5s\n%s\n\n", '^', '|', 'substrate';
}


sub combine {
    my ($matrix1, $matrix2) = @_;
    my %new_hash = %$matrix1;
    
    for my $substrate (keys %$matrix2) {
        $new_hash{$substrate}{$_} = 1 for keys %{ $matrix2->{$substrat
+e} };    
    }
    return %new_hash;
}
[download]

##########INPUT###############
list.txt
substrate: 1 2
product: 3
substrate: 6 9
product: 8 10
substrate: 3
product: 6
substrate: 9
product: 5
substrate: 5
product: 2
substrate: 3
product: 9
substrate: 8
product: 9
substrate: 8
product: 1
substrate: 7
product: 11
substrate: 19
product: 17
substrate: 14
product: 13
substrate: 14
product: 11
substrate: 18
product: 19
substrate: 7 14
product: 15
substrate: 7 16
product: 7 17
substrate: 5
product: 6
substrate: 18 15
product: 7
substrate: 7 8
product: 8 18
substrate: 6
product: 9
substrate: 11
product: 12

another.txt
substrate[s]: 3649
product[s]: 3419 3648
substrate[s]: 3645
product[s]: 3647
substrate[s]: 3659
product[s]: 3647
substrate[s]: 3675
product[s]: 3674
substrate[s]: 3674
product[s]: 3490 3489
substrate[s]: 3489
product[s]: 3490
substrate[s]: 3490
product[s]: 3485
substrate[s]: 3485
product[s]: 3486
substrate[s]: 3486
product[s]: 3488
substrate[s]: 3488
product[s]: 3487
substrate[s]: 3487
product[s]: 3877
substrate[s]: 3877
product[s]: 3419
substrate[s]: 3182
product[s]: 1875
substrate[s]: 2809
product[s]: 3182
substrate[s]: 3186
product[s]: 2809
[download]

############OUTPUT#################
Processing file: list.txt
 prod->  1  2  3  5  6  7  8  9 10 11 12 13 15 17 18 19
      1  -  -  1  -  -  -  -  -  -  -  -  -  -  -  -  -
      2  -  -  1  -  -  -  -  -  -  -  -  -  -  -  -  -
      3  -  -  -  -  1  -  -  1  -  -  -  -  -  -  -  -
      5  -  1  -  -  1  -  -  -  -  -  -  -  -  -  -  -
      6  -  -  -  -  -  -  1  1  1  -  -  -  -  -  -  -
      7  -  -  -  -  -  1  1  -  -  1  -  -  1  1  1  -
      8  1  -  -  -  -  -  1  1  -  -  -  -  -  -  1  -
      9  -  -  -  1  -  -  1  -  1  -  -  -  -  -  -  -
     11  -  -  -  -  -  -  -  -  -  -  1  -  -  -  -  -
     14  -  -  -  -  -  -  -  -  -  1  -  1  1  -  -  -
     15  -  -  -  -  -  1  -  -  -  -  -  -  -  -  -  -
     16  -  -  -  -  -  1  -  -  -  -  -  -  -  1  -  -
     18  -  -  -  -  -  1  -  -  -  -  -  -  -  -  -  1
     19  -  -  -  -  -  -  -  -  -  -  -  -  -  1  -  -

    ^
    |
substrate

Processing file: another.txt
 prod-> 1875 2809 3182 3419 3485 3486 3487 3488 3489 3490 3647 3648 36
+74 3877
   2809    -    -    1    -    -    -    -    -    -    -    -    -   
+ -    -
   3182    1    -    -    -    -    -    -    -    -    -    -    -   
+ -    -
   3186    -    1    -    -    -    -    -    -    -    -    -    -   
+ -    -
   3485    -    -    -    -    -    1    -    -    -    -    -    -   
+ -    -
   3486    -    -    -    -    -    -    -    1    -    -    -    -   
+ -    -
   3487    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    1
   3488    -    -    -    -    -    -    1    -    -    -    -    -   
+ -    -
   3489    -    -    -    -    -    -    -    -    -    1    -    -   
+ -    -
   3490    -    -    -    -    1    -    -    -    -    -    -    -   
+ -    -
   3645    -    -    -    -    -    -    -    -    -    -    1    -   
+ -    -
   3649    -    -    -    1    -    -    -    -    -    -    -    1   
+ -    -
   3659    -    -    -    -    -    -    -    -    -    -    1    -   
+ -    -
   3674    -    -    -    -    -    -    -    -    1    1    -    -   
+ -    -
   3675    -    -    -    -    -    -    -    -    -    -    -    -   
+ 1    -
   3877    -    -    -    1    -    -    -    -    -    -    -    -   
+ -    -

    ^
    |
substrate


Combining list.txt and another.txt
 prod->    1    2    3    5    6    7    8    9   10   11   12   13   
+15   17   18   19 1875 2809 3182 3419 3485 3486 3487 3488 3489 3490 3
+647 3648 3674 3877
      1    -    -    1    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
      2    -    -    1    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
      3    -    -    -    -    1    -    -    1    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
      5    -    1    -    -    1    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
      6    -    -    -    -    -    -    1    1    1    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
      7    -    -    -    -    -    1    1    -    -    1    -    -   
+ 1    1    1    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
      8    1    -    -    -    -    -    1    1    -    -    -    -   
+ -    -    1    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
      9    -    -    -    1    -    -    1    -    1    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
     11    -    -    -    -    -    -    -    -    -    -    1    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
     14    -    -    -    -    -    -    -    -    -    1    -    1   
+ 1    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
     15    -    -    -    -    -    1    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
     16    -    -    -    -    -    1    -    -    -    -    -    -   
+ -    1    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
     18    -    -    -    -    -    1    -    -    -    -    -    -   
+ -    -    -    1    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
     19    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    1    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
   2809    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    1    -    -    -    -    -    -    -  
+  -    -    -    -
   3182    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    1    -    -    -    -    -    -    -    -    -  
+  -    -    -    -
   3186    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    1    -    -    -    -    -    -    -    -  
+  -    -    -    -
   3485    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    1    -    -    -    -  
+  -    -    -    -
   3486    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    1    -    -  
+  -    -    -    -
   3487    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    -    1
   3488    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    1    -    -    -  
+  -    -    -    -
   3489    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    1  
+  -    -    -    -
   3490    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    1    -    -    -    -    -  
+  -    -    -    -
   3645    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  1    -    -    -
   3649    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    1    -    -    -    -    -    -  
+  -    1    -    -
   3659    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  1    -    -    -
   3674    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    1    1  
+  -    -    -    -
   3675    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    -    -    -    -    -    -    -  
+  -    -    1    -
   3877    -    -    -    -    -    -    -    -    -    -    -    -   
+ -    -    -    -    -    -    -    1    -    -    -    -    -    -  
+  -    -    -    -

    ^
    |
substrate
[download]

[reply]
[d/l]
[select]

Something like the following (untested) code perhaps?

opendir my ($scan), $path or die "Can't scan folder $path: $!\n";

while (my $entry = readdir $scan) {
    next if ! -d "$path/$entry" || $entry !~ /^d[1-5]$/;
    next if ! -f "$path/$entry/list.txt";
    
    ...
}
[download]

True laziness is hard work

[reply]
[d/l]

The code should select list.txt from first directory and then process all the 10 list.txt files one by one from each of the 10 directories.

chdir "my_parent_dir"; my @files = glob("*/list.txt"); for my $item(@files){ open my $currentfile, '<', $item; .. do something... close $currentfile; } # and go to the next item of the loop
[download]

[reply]
[d/l]