Re: system, pipes, shell, quoting

[graff]

To expound a bit on dws's remarks: you would not want to run that system call if the value of $file1 was something like "/dev/null;rm -rf *"... The point would be to make sure that the variables being used are not so "weird" as to wreak havoc when they are passed "successfully" to the shell.

To be safe, the values of $program1, $program2, etc. would be "known", drawn from some limited set of alternatives, and the values of $file1, $file2 would be pre-conditioned to eliminate any characters that would cause problems in the shell (esp. semicolons, pipe symbols, ampersands and such) -- either remove them outright or replace them with safe punctuation characters, underscores, or whatever.

But if you need to handle some oddball file names, you might try the following alternative to the system call:

$file1 = s/(\W)/\\$1/; # and similarly for other filenames
open( SH, "| /bin/sh");
print SH "$program1 $file1 | $program2 | $program3 > $file2\n";
close SH;

I'm not saying this is guaranteed to work for you, but it's something to try (especially if you were intending to use the system call inside a loop: open the shell before going into the loop, then just print a command line to the shell on each iteration -- see a sample of this in a utility I posted a while ago).

[superpete]

$file1 = s/(\W)/\\$1/; # and similarly for other filenames

unfortunately, this doesn't seem to work. Here is an example :-(

foreach $d ( @dirs ) {
   -e $d or die;         # works
   -d $d or die;         # works
   $d =~ s/(\W)/\\$1/g;
   system("touch $d");   # touch can't find it, it prints an error!!!
}

"open" will most likely do the same thing. The problem is the basic act of quoting arbitrary unprintable characters to the shell (nevermind security for the time being). FYI, the shell is /bin/sh on FreeBSD 4.1, which I assume is pretty solid.

[graff]

"open" will most likely do the same thing

(ahem) You mean you won't even try it? Pull down the "shloop" utility I referred to earlier, and try this (I just did):

echo '?*;&'

(just to convince you that using single-quotes around a nasty string will help for some cases -- so long as the string doesn't include any single-quote characters) And then try this:

mkdir /tmp/junk
echo '?*;&' | shloop -e "touch '/tmp/junk/\i'"
ls -l /tmp/junk

If that doesn't work for you, please let me know. I don't have access to a FreeBSD system, and I'd be interested to learn how different it could be from linux & solaris (where this works).

BTW, if your list of potential file names does happen to involve cases that include one or more quote-like characters, then you're right -- the above will not work. In such cases, let me suggest that you should only be facing this issue with existing file names that are created by other processes and are to be used as input to a given pipeline -- don't ever create such file names yourself for output. To handle these, open the the nasty-named file for input in perl (get the file name via "readdir" so you don't need to present such a name as a command-line arg -- who knows, maybe a file name could include a "\n"!), then open your pipeline as a file handle, and pass file data to the pipeline that way. E.g.:

opendir( DIR, "some_dir" );
@files = grep /[^.]/, readdir( DIR ); # skips "." and ".."
# (assumes you never need to look for files named "...")
foreach $file ( @files ) {
  next unless ( -f $file );
  $outfile = "something_sane";
  open( PIPE, "| $prog1 | $prog2 -x -y | $prog3 > $outfile" );
  open( IN, "<", $file );
  while (<IN>) {
     print PIPE;
  }
  close PIPE;
  close IN;
# and/or unlink or rename that input file so it's less of a bother hen
+ceforth
}

I haven't tried that yet (but I think you should try it yourself before you say it won't work... ;^).

[superpete]

BTW, if your list of potential file names does happen to involve cases that include one or more quote-like characters, then you're right -- the above will not work. In such cases, let me suggest that you should only be facing this issue with existing file names that are created by other processes and are to be used as input to a given pipeline -- don't ever create such file names yourself for output. To handle these, open the the nasty-named file for input in perl (get the file name via "readdir" so you don't need to present such a name as a command-line arg -- who knows, maybe a file name could include a "\n"!), then open your pipeline as a file handle, and pass file data to the pipeline that way.

Good point... This is probably the strategy I will adopt. It still bothers me a little bit that you are so completely at the mercy of the shell in the general case of this situation.

[superpete]

This is offtopic so i'll reply to it first :-)

I don't have access to a FreeBSD system, and I'd be interested to learn how different it could be from linux & solaris

Because bourne shell is so important, the behavior should be identical (POSIX), but I can't swear to it. I've encountered /bin/sh scripts containing some bash-specific syntax (i.e., poorly written shell scripts). These worked on linux but not FreeBSD because FreeBSD's /bin/sh was more anal about POSIX correctness.

[superpete]

Just so you know, my test data really does contain EVERY conceivable character. For testing, the filenames are a random mix of the characters:

chr(rand(128))

except for "/" and "\0"

[hardburn]

I just thought of this, so correct me if I'm doing something naive.

If you have to get the values of $program1 and $program2 from an outside source, you can put allowed values in a hash like this:

my %allowed;
$allowed{'ls'} = '/bin/ls';
$allowed{'grep'} = '/bin/grep';
$allowed{'gzip'} = '/bin/gzip';
. . .

When it comes time to execute the program:

my $good_prog = $allowed{$program1};
system($good_prog) if($good_prog != undef);

This won't help you with arguments, of course. Also, you'll be limiting the actual programs that can be run (which is probably a good thing). If you want to allow everything in /bin and /usr/bin, try this (untested):

use File::Find;
my %allowed;
find(\&add, '/bin', '/usr/bin');

sub add 
{
     $allowed{$_} = $File::Find::name;
}