perl: Argument list too long

[sjhalani]

Hi all,

we have a shell script from which we invove the perl i.e

a.sh
----
#!/bin/sh

we set a lot of environment variables here and then call perl script finally as :

$PERL_BIN/perl a.pl "$@" The perl script has the following first line i.e

#!/usr/local/bin/perl
use lib (...)
...
Usual processing ...

But when we invoke it we get the error

a.sh bin/perl: Argument list too long

The arguments we are passing are way within the limits i.e not more than 400 characters then why is the perl complaining on the argument list ? This is running on Linux box.

Is there some limit on the number of environment variables we can set ?

[parv]

I think the error must be coming from the shell invoking perl, not perl itself. The error would not be due to setting environment variables, but what is passed to $PERL_BIN/perl.

To confirm ...

• insert set -x just before the perl invocation line (to see the actual invocation; may want to set +x just after the perl line to avoid clutter in case perl call succeeds);
• replace $PERL_BIN/perl with echo ("argument list too long" message would still be produced);
• reduce the number of things stuffed in $@ (error message would enventually go away when "enough" -- consult your shell|kernel documentation|source -- of the arguments have been stripped away).

Thanks (rest of the posters) for the clue that environment is also part of the space calculation, which I had outright dismissed in my reply.

[pc88mxer]

This is a kernel limitation. Shell arguments and environment variables are allocated from the same space and are limited to some number of pages - usually something like 32.

You'll probably have to come up with another way to pass your arguments, such as reading them through STDIN or passing them in a file.

For more discussion about this problem, see "Argument list too long": Beyond Arguments and Limitations

Here's a script to test what the limits for %ENV are:

Read more... (593 Bytes)
On my Ubuntu box it first fails at n = 125, which corresponds to approx. 128 K (= 32 x 4K pages.)

[arc_of_descent]

This is another good link.
GNU coreutils FAQ

[quester]

Thank you for that link! Oooh, sweet, unlimited environment size in Linux 2.6.23 (see the "recent news".)

[oko1]

As many other people have pointed out, your argument list is too long. However, rather than suggesting some special arg-processing scheme (e.g., some 'xargs'-like mechanism), I would suggest that you instead reconsider your approach to the way you handle your data: if you have so many arguments on the command line that the shell is choking, then you shouldn't be presenting that data as commandline arguments. Placing it in a file, for example, and processing that file would eliminate those problems. You could also, for example, create a named pipe and have your Perl script read from it, with each argument being processed as it comes in (assuming that this is a valid way of processing your data.) In short, it's not your code that's a problem: it's your approach to handling your data. Change your approach, and the problem will go away.

[Helter]

I've hit this before and I believe that the "argument list" size is made up of the argument list plus all of your environment. So if you have lots of stuff set (set and setenv calls) in environment variables then this can limit what you can have on the command line...

[jplindstrom]

Others have probably correctly identified your problem.

The solution is Getopt::Long + Getopt::ArgvFile, which makes it possible to specify a command line arguments file to be used. This is a good idea in general.

/J