Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

perl: Argument list too long

by sjhalani (Initiate)
on May 03, 2008 at 03:43 UTC ( [id://684300] : perlquestion . print w/replies, xml ) Need Help??

sjhalani has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

we have a shell script from which we invove the perl i.e ---- #!/bin/sh

we set a lot of environment variables here and then call perl script finally as :

$PERL_BIN/perl "$@" The perl script has the following first line i.e
#!/usr/local/bin/perl use lib (...) ... Usual processing ...

But when we invoke it we get the error bin/perl: Argument list too long

The arguments we are passing are way within the limits i.e not more than 400 characters then why is the perl complaining on the argument list ? This is running on Linux box.

Is there some limit on the number of environment variables we can set ?

Replies are listed 'Best First'.
Re: perl: Argument list too long
by parv (Parson) on May 03, 2008 at 03:46 UTC

    I think the error must be coming from the shell invoking perl, not perl itself. The error would not be due to setting environment variables, but what is passed to $PERL_BIN/perl.

    To confirm ...

    • insert set -x just before the perl invocation line (to see the actual invocation; may want to set +x just after the perl line to avoid clutter in case perl call succeeds);
    • replace $PERL_BIN/perl with echo ("argument list too long" message would still be produced);
    • reduce the number of things stuffed in $@ (error message would enventually go away when "enough" -- consult your shell|kernel documentation|source -- of the arguments have been stripped away).

    Thanks (rest of the posters) for the clue that environment is also part of the space calculation, which I had outright dismissed in my reply.

Re: perl: Argument list too long
by pc88mxer (Vicar) on May 03, 2008 at 04:54 UTC
    This is a kernel limitation. Shell arguments and environment variables are allocated from the same space and are limited to some number of pages - usually something like 32.

    You'll probably have to come up with another way to pass your arguments, such as reading them through STDIN or passing them in a file.

    For more discussion about this problem, see "Argument list too long": Beyond Arguments and Limitations

    Here's a script to test what the limits for %ENV are:

    On my Ubuntu box it first fails at n = 125, which corresponds to approx. 128 K (= 32 x 4K pages.)
Re: perl: Argument list too long
by arc_of_descent (Hermit) on May 03, 2008 at 07:49 UTC
      Thank you for that link! Oooh, sweet, unlimited environment size in Linux 2.6.23 (see the "recent news".)
Re: perl: Argument list too long
by oko1 (Deacon) on May 03, 2008 at 14:00 UTC

    As many other people have pointed out, your argument list is too long. However, rather than suggesting some special arg-processing scheme (e.g., some 'xargs'-like mechanism), I would suggest that you instead reconsider your approach to the way you handle your data: if you have so many arguments on the command line that the shell is choking, then you shouldn't be presenting that data as commandline arguments. Placing it in a file, for example, and processing that file would eliminate those problems. You could also, for example, create a named pipe and have your Perl script read from it, with each argument being processed as it comes in (assuming that this is a valid way of processing your data.) In short, it's not your code that's a problem: it's your approach to handling your data. Change your approach, and the problem will go away.

    Human history becomes more and more a race between education and catastrophe. -- HG Wells
Re: perl: Argument list too long
by Helter (Chaplain) on May 03, 2008 at 04:54 UTC
    I've hit this before and I believe that the "argument list" size is made up of the argument list plus all of your environment. So if you have lots of stuff set (set and setenv calls) in environment variables then this can limit what you can have on the command line...
Re: perl: Argument list too long
by jplindstrom (Monsignor) on May 03, 2008 at 13:36 UTC
    Others have probably correctly identified your problem.

    The solution is Getopt::Long + Getopt::ArgvFile, which makes it possible to specify a command line arguments file to be used. This is a good idea in general.