Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Loop through all directory and files and lines

by MissPerl (Sexton)
on Oct 18, 2018 at 13:25 UTC ( #1224232=perlquestion: print w/replies, xml ) Need Help??

MissPerl has asked for the wisdom of the Perl Monks concerning the following question:

Hi fellow monks! CODE EDITED

I want to go into first directory/i/j/*, then read all the files and all the lines inside every file. Eg, /somewhere/1/2/*

Currently the script gave me no error, but the output was surprisingly all from the (else condition).

I couldnt make sure, since there are a lot of files inside it, but there should at least have some cases match the (if ) condition.

Can anyone have a look at the script i wrote and tell me if anything i have overlooked ?

Each file, I want to find if any three of the strings exists, if it does, I'll store its filepath in store_location.

Also the (i, j), was from an array with even number contents, odd number will be i; while even is j.
# !/usr/bin/perl use strict; use warnings; use File::Glob 'bsd_glob'; my $store_location = '/path/to/file'; my $Dir1 = "/somewhere"; my $Dir2 = "/somewhereelse"; my @first_directory = ( bsd_glob("$Dir1/"), bsd_glob("$Dir2/") ); my @store_array = qw (1 2 3 4 5 6); foreach my $first (@first_directory){ while(my ($i,$j) = splice(@store_array,0,2)){ my $second_directory = "$first/$i/$j"; if (-e $second_directory and -d $second_directory){ my @third_directory = bsd_glob("$first/$i/$j/*"); foreach my $file (@third_directory){ open(FILE, "<" , $file) or die "Can't open file '$file +': $!"; while (<FILE>){ if (($_ =~ /TbhODK/) or ($_ =~ /octuov/) or ($_ =~ + /qas_uop/)) { open(my $filehand, '>>', $store_location) or d +ie "Fail to open file '$store_location' $!"; print $filehand "$first/$i/$j/$file \n"; close $filehand; } } } close FILE; } else{ open(my $filehand, '>>', $store_location) or die "Fail to +open file '$store_location' $!"; print $filehand "$first/$i/$j (fail to exist)\n"; close $filehand; } } }#first_directory
Kindly let me know if it's unclear,I will improve the sentence.thank you perlmonks !

Replies are listed 'Best First'.
Re: Loop through all directory and files and lines
by 1nickt (Canon) on Oct 18, 2018 at 13:32 UTC

    Hi, please post an SSCCE. The code you have shown does not compile. It certainly looks as if your solution does not need to be this complicated, but it's hard to know since the code is incomplete.

    Your problem description is also unclear. "I want to go into first directory/i/j/*" -- this path does not look like a directory.

    If you haven't yet done so, take a look at the module I recommended in your previous thread, Path::Iterator::Rule. Or another module by the same author, Path::Tiny, which provides the handy visit() method. The following should do what you want:

    use strict; use warnings; use feature 'say'; use Path::Tiny; use List::Util 'any'; my $regexp = qr/(?:TbhODK|octuov|qas_uop)/; my $found = path('/some/path')->visit( sub { my ($path, $results) = @_; return if $path->is_dir; $results->{$path}++ if any { /$regexp/ } $path->lines; }, { recurse => 1 }, ); say for keys %{ $found }; __END__

    Hope this helps!

    The way forward always starts with a minimal test.
      Hi 1nickt ! I have updated the code, could you try and run once?

      while I go through your code for now thank you !!

      I really wish to know what's wrong in the code. :'(

      I have review your code. seems like this two Path::Tiny; and List::Util 'any' are helpful module!

      I wasn't allowed to install module in the laptop. but I will be sure to check out these soon for own knowledge!
Re: Loop through all directory and files and lines
by zentara (Archbishop) on Oct 18, 2018 at 13:43 UTC
    Hi, here is a script I use often to search thru my files, it's not the best code, but it works. Notice the commented out lines which skip binary files. Usually, you don't want to search a binary file.
    #!/usr/bin/perl use warnings; use strict; use File::Find; $|++; # defaults are case-insensitive, no recurse, open and search files (no +t filename) if ($#ARGV < 0){die "Usage zgrep 'pattern' c(case sensitive optional) +r(recurse optional) n(search name only optional) examples: zgrep 'debug me' r will recursively search all files for 'debug me'\n"; } my ($recurse, $name, $case) =(0,0,0); if( grep{/\bn\b/} @ARGV ){@ARGV = grep { $_ ne 'n' } @ARGV; $name = 1 +}; if( grep{/\br\b/} @ARGV ){@ARGV = grep { $_ ne 'r' } @ARGV; $recurse = + 1 }; if( grep{/\bc\b/} @ARGV ){@ARGV = grep { $_ ne 'c' } @ARGV; $case = 1 +}; #print "$name $recurse $case @ARGV\n"; my $path = '.'; #only accept 1 search string, so quote phrases my $search = $ARGV[0]; my $regex; #defaults to case insensitive if ($case){$regex = qr/\Q$search\E/} else{$regex = qr/\Q$search\E/i} # use s modifier for multiline match find (sub { #skip directories which begin with 1 if (-d && $_ =~ /^1.*$/) { $File::Find::prune = 1; return; } if( ! $recurse ){ my $n = ($File::Find::name) =~ tr!/!!; #count slashes in file return $File::Find::prune = 1 if ($n > 1); } return if -d; # return unless (-f and -T ); # don't waste time on binaries if($name){ if ($_ =~ /$regex/){print "$File::Find::name\n"}; }else{ return unless (-f and -T ); # don't waste time on binaries open (FH,"< $_"); while(<FH>){ print "$File::Find::name: $. :$_\n " if /$regex/; } close FH; } }, $path); exit;

    I'm not really a human, but I play one on earth. ..... an animated JAPH
      thank you for the help !

      I'll have a look ! that's very nice.

      However, my priority now is to understand what is wrong with my code, which I still cant figure out till now.. :(
        what is wrong with my code

        My guess is you need to refresh @store_array within the @first_directory loop

        foreach my $first (@first_directory){ my @store_array = qw (1 2 3 4 5 6); # here while(my ($i,$j) = splice(@store_array,0,2)){ my $second_directory = "$first/$i/$j"; ..


        #!/usr/bin/perl use strict; use warnings; use File::Glob 'bsd_glob'; my $store_location = '/path/to/file/store.txt'; open my $fh_log, '>', $store_location or die "Fail to open file '$store_location' $!"; my $Dir1 = "/somewhere"; my $Dir2 = "/somewhereelse"; my @first_directory = ( $Dir1, $Dir2 ); foreach my $first (@first_directory){ my @store_array = qw (1 2 3 4 5 6 7 8); while (my ($i,$j) = splice(@store_array,0,2)){ my $second_directory = "$first/$i/$j"; if (-e $second_directory and -d $second_directory){ my @files = bsd_glob("$first/$i/$j/*"); foreach my $file (@files){ my $lines = 0; my $match = 0; open(FILE, "<" , $file) or die "Can't open file '$file': $!"; + while (<FILE>){ if (/TbhODK|octuov|qas_uop/) { ++$match; } ++$lines; } print "$file : $lines lines read. $match matches found\n"; close FILE; print $fh_log "File $file has $match matches\n"; } } else { print $fh_log "Directory $first/$i/$j (fail to exist)\n"; } } } close $fh_log;

        The problem is your code contains no subroutines , yet contains four loops -- it should contain three subroutines minimum

Re: Loop through all directory and files and lines
by haukex (Archbishop) on Oct 19, 2018 at 11:14 UTC

    This seems like a good place to add debugging statements using Data::Dumper (Basic debugging checklist). For example, if you show $second_directory, you'll see

    $VAR1 = '/somewhere//1/2'; $VAR1 = '/somewhere//3/4'; $VAR1 = '/somewhere//5/6';

    So first of all, you'll notice the double slashes, which is an issue that you can fix using e.g. File::Spec's catfile and catdir instead of building the strings yourself.

    Second, you might notice that '/somewhereelse/' is missing. This is because splice is a destructive operation. As poj showed, one possible solution would be to move the declaration of @store_array right before the while loop that uses it. poj's code also shows how the if with three regexes can be more efficiently written as if ( /TbhODK|octuov|qas_uop/ ), and opening the $store_location once, at the start of the program, instead of re-opening it in append mode for each line of output.

    In general, glob has several caveats, which might apply in your case, since you're using variables in the patterns. Switching to one of the other file-handling modules, such as Path::Class and/or File::Find::Rule, would help there.

    Other than that, your code looks relatively ok to me; whether it works for you or not will depend on whether the directory structure matches what your code is doing, which we can't tell.

Re: Loop through all directory and files and lines
by harangzsolt33 (Hermit) on Oct 19, 2018 at 05:34 UTC
    You have obviously worked a lot on this code, but I think you should rewrite this. Start from scratch. It seems like a big mess. I couldn't test it, because my perl interpreter crashes every time I try to run your script. :( Btw this is the first time I've ever seen perl crash.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1224232]
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (5)
As of 2022-12-03 19:55 GMT
Find Nodes?
    Voting Booth?

    No recent polls found