Re: Adding columns in a loop to an existing file using Perl

[aaron_baugher]

My first question would be how many input files you have. If it's not more than the number of files you can open at once, then I'd do this (pseudocode):

open all input files, making array of file descriptors
open output file for writing
while the first input file has remaining lines
  for each input file descriptor
    read one line
    get desired column
    write to output file
  write newline to output file
close all files

If your input files have different numbers of lines, you may need a bit more to handle that. But basically, open all files and process the first line of each input file, creating the first line of the output file. Then move on to the second lines, then the third, etc.

If there are more files than you can have open simultaneously, you'll have to do something else. Yes, you can repeatedly add columns to the end of your output file, preferably using something like Text::CSV to keep things correct, but that would be pretty inefficient, so don't do that if you don't have to. Honestly, I'd probably do this with shell tools, which are pretty handy for things like breaking lines on a delimiter (when you don't have to worry about things like quoted delimiters within fields) and handling multiple columns of text:

#!/bin/sh
# args: $1 - directory holding input files
#       $2 - column to save
#       $3 - output file
#  temporary files created in /tmp/columns

n=0

for i in $1/*; do
    n=`expr $n + 1`
    echo $i >/tmp/columns/$n
    cut -f $2 $i >>/tmp/columns/$n 
done
paste /tmp/columns/* >$3
rm /tmp/columns/*