Extract information from several files in directory

Hi I am a beginner at Perl, trying to do some really simple things, but nothing seems simple.. I have a folder with a bunch of tab sep txt files. All the files have the same type of information, and I need to extract the information from one of the columns in each file and put in a new file. I have previously managed to do this with one file by readin gthe file into an array and iterating each row of the array, spliting it on tab and the taking the second column (where the data I want is) and printing this to a new file. But now I have several files. I started by using a while loop to open the directory and counting the number of files in the directory. But struggling to get to reading each file and extracting my data.. After lots of googling, there seems to be lots of different answers, but can't really understand them. I need something very simple so I can understand how it works.

#!/usr/bin/perl -w
use strict;


#create file to write to
open (LABNR, ">>Labnr_all.txt") or die "Could not open file";

#open the directory where files are located
$dirname = 'Filer';
opendir (DIR, $dirname) or die "Could not open $dirname\n";

#count files
$nrfiles = 0;

while ($filename = readdir(DIR)){
    $nrfiles ++ if $filename =~ /\.txt/;
    print "$filename\n" if -f $filename;
#this is where I want to look into each file and extract the info...
    }
    
print "The number of files in the folder: $nrfiles\n";
    
closedir(DIR)

[hippo]

I have previously managed to do this with one file

That's a good place to start. Might I suggest the following approach to help extend this to several files?

1. Take your working code and move it into a subroutine (see perlsub) which takes the filename to be read as the only argument. Confirm the script still works as before.
2. Now simply call the subroutine twice in the script with a different filename each time. Confirm that the script produces the output you expect for two input files.
3. Now write your loop to find all the files in the directory and just call the subroutine from within the loop. You should now have output from all the files.

Portioning up the code into smaller blocks (such as using subroutines) helps you to work on small parts of a larger problem in isolation. If you have specific problems with any particular part feel free to ask. Remember that an SSCCE is best. Good luck.

[tybalt89]

In before everyone else suggests using a module for this. Path::Tiny is my current favorite module for general file bashing.

#!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11124289
use warnings;
use Path::Tiny;

my $dirname = 'Filer';
my $outputfile = path('output.second.column');
$outputfile->spew(''); # empty file

for my $file ( path($dirname)->children( qr/\.txt\z/ ) )
  {
  $outputfile->append( map { (split /\t|\n/)[1] . "\n"} $file->lines )
+;
  }

# following for debug only :)
system 'more Filer/* output.second.column | cat';
# because 'more' shows filename and contents when piped

Outputs:

::::::::::::::
Filer/one.txt
::::::::::::::
one     two     three
four    five    six
::::::::::::::
Filer/three.txt
::::::::::::::
thirteen        fourteen        fifteen
sixteen seventeen       eighteen
::::::::::::::
Filer/two.txt
::::::::::::::
even    eight   nine
ten     eleven  twelve
::::::::::::::
output.second.column
::::::::::::::
two
five
fourteen
seventeen
eight
eleven

Is this the kind of thing you were looking for ?

P.S. See, it really is simple...

[AnomalousMonk]

This won't help you at the moment, but for future reference, keep in the back of your mind the modules Text::CSV, which is pure Perl, and the closely related Text::CSV_XS, which does the same things only faster, but requires local compilation that can be problematic.

These modules do lots of stuff and can be a great help when processing | reading and writing CSV files of any kind. However, their learning curve, while quite reasonable, is steep enough that you may wish to simplify your current situation by sticking close to what you know.

[Tux]

If you start simple, Text::CSV_XS' interface does not have a steep learning curve at all :)

If all your text files have a header, and you want "column" fooble extracted from all TAB-separated .txt files

use strict;
use warnings;
use Text::CSV_XS qw( csv );

my @result;
foreach my $f (sort glob "*.txt") { # Use File::Find for recursive act
+ions
    csv (in => $f, headers => "auto", sep => "\t", on_in => sub {
        push @result => $_{fooble};
        });
    }
open my $fh, ">", "results.txt" or die $!;
say $fh $_ for grep { length } @result;
close $fh;

[LanX]

your code can't compile, because you use strict but never my or even our

[wazat]

One thing to be aware of is that readdir() returns file names without the directory part. If using readdir() you will need to prepend the file names with the directory part of the file path in order to open it for reading.

You could just build the path yourself, which is not necessarily portable.

my $path = "Filer/$filename";

or you could use File::Spec->catfile() for portability

use File::Spec;
my $path = File::Spec->catfile('Filer', $filename");

This is an old fashioned approach.

If you use glob() it will return file paths. Then again for portability code>File::Spec->catfile()</code> could be used to build the argument to glob().

As others have pointed out, using Path::Tiny simple. It is a more modern approach. For example

my @paths = path("Filer")->children( qr/\.txt\z/ );
# now do something with each element of @paths