Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Ulimit being reached

by viffer (Beadle)
on Jan 13, 2015 at 04:33 UTC ( #1113028=perlquestion: print w/replies, xml ) Need Help??

viffer has asked for the wisdom of the Perl Monks concerning the following question:

Afternoon genuii

I'm writing a file on an aix system.

Despite the file size exceeding the ulimit, the script isn't terminating.

Aside of putting in explicit length checking, is there any method that makes a perl script either abort or return an error under such circumstances.

By doing a "$rc = print FILE", the $rc always returns 1 whether the write is successful or not.
Thanks - Kev

Replies are listed 'Best First'.
Re: Ulimit being reached
by Anonymous Monk on Jan 13, 2015 at 04:53 UTC
    print ... ends up eventually doing a system call, if the system call doesn't errno due to ulimit, perl can't make it errno ... you need to investigate aix ulimit faq :)

    I've read at ulimit that it applies to "shell" launched stuff, is your perl program launched from shell?

      The ulimit is set at system level - and has since been increased. I wanted to trap an error when perl tried to write to a file and exceeded the ulimit. The closing file handle worked ok.

      Since then we have come up with the following to trap the error - although it has thrown up a strange error of not liking the "|| die" appearing on a separate line.

      #!/usr/local/bin/perl use strict; use warnings FATAL => 'all'; open(OUT, '>', "splat") or die "Opening (write) 'splat' $!"; while (1) { my $line = time().; # THIS WORKS print (OUT $line) || die "writing 'splat' $!"; # But putting || die on another line does not. i.e: # print (OUT $line) # || die "writing 'splat' $!"; } close(OUT) or die "Closing 'splat' $!";
      The error message produced is:

      writing 'splat'

      A file cannot be larger than the value set by ulimit. at u3.pl line 12.

        You should always use low precedence or (in preference to ||) with the "or die" idiom. Also, you should normally avoid parens when calling print, so this is the recommended way to write it:

        print OUT $line or die "writing 'splat' $!";
        For a working example, see my reply below.

        Update: further notice that lexical file handles (e.g. $fh in my reply below) are preferred to the old global file handles (e.g. OUT in your example code).

        # THIS WORKS print (OUT $line) || die "writing 'splat' $!"; # But putting || die on another line does not. i.e: # print (OUT $line) # || die "writing 'splat' $!";
        That's strange. They both work for me.

        Moreover, I see no reason why they would be different, in fact Deparse tells me they produce identical code:

        % cat id1.pl print (OUT $line) || die "writing 'splat' $!"; % perl -MO=Deparse id1.pl die "writing 'splat' $!" unless print OUT $line; id1.pl syntax OK % cat id2.pl print (OUT $line) || die "writing 'splat' $!"; % perl -MO=Deparse id2.pl die "writing 'splat' $!" unless print OUT $line; id2.pl syntax OK
        Though you could dig further into the root cause of the difference you experienced, I suggest you just stick to the low precedence or and avoid parens with print as described earlier.

Re: Ulimit being reached
by pme (Monsignor) on Jan 13, 2015 at 08:35 UTC
    What is your ulimit -f setting? bash uses 1024bytes increments.
Re: Ulimit being reached
by FloydATC (Deacon) on Jan 13, 2015 at 12:22 UTC

    I think the error condition will normally not be detectable until you try to close() the file handle.

    The return value from close() should be 0 and $! should indicate the error. The actual message will depend on the operating system.

    -- FloydATC

    Time flies when you don't know what you're doing

Re: Ulimit being reached
by eyepopslikeamosquito (Bishop) on Jan 14, 2015 at 03:33 UTC

    Tested just now on aix61 and it worked for me. Saved the following program as f.pl:

    # From /etc/security/limits: # Sizes are in multiples of 512 byte blocks # fsize - soft file size in blocks # To test this program, I set block size via: # ulimit -f 900 use strict; use warnings; $| = 1; my $bsize = 512; my $ulimit = `ulimit -f`; chomp $ulimit; print "ulimit=$ulimit blocks (block size=$bsize)\n"; my $fname = 'f.tmp'; open my $fh, '>', $fname or die "error: open '$fname': $!"; my $niter = 1000; my $size = 0; my $block = 'a' x $bsize; for my $i (0..$niter) { $size += $bsize; print "$i: $size bytes written\n" if $i % 100 == 0; print $fh $block or die "$i: error in print: $!"; } close $fh or die "error: close: $!";
    After issuing the command:
    ulimit -f 900
    running the above program f.pl produced the following output:
    % perl f.pl ulimit=900 blocks (block size=512) 0: 512 bytes written 100: 51712 bytes written 200: 102912 bytes written 300: 154112 bytes written 400: 205312 bytes written 500: 256512 bytes written 600: 307712 bytes written 700: 358912 bytes written 800: 410112 bytes written 900: 461312 bytes written 904: error in print: A file cannot be larger than the value set by uli +mit. at f.pl line 22.

    Update: The reason for the program stopping at 904, rather than 900, is user-space buffering by stdio. To verify that it stops at 900 with user-space buffering switched off, I adjusted the above program by adding a call to autoflush right after the output file is opened. Running this adjusted program:

    use strict; use warnings; use IO::Handle; # <-- Added this line for autoflush $| = 1; my $bsize = 512; my $ulimit = `ulimit -f`; chomp $ulimit; print "ulimit=$ulimit blocks (block size=$bsize)\n"; my $fname = 'f.tmp'; open my $fh, '>', $fname or die "error: open '$fname': $!"; $fh->autoflush(); # <-- Added this line to make print autoflush my $niter = 1000; my $size = 0; my $block = 'a' x $bsize; for my $i (0..$niter) { print "$i: $size bytes written\n" if $i % 100 == 0; print $fh $block or die "$i: error in print: $!"; $size += $bsize; # <-- Moved this line to after successful print } close $fh or die "error: close: $!";
    produced:
    ulimit=900 blocks (block size=512) 0: 0 bytes written 100: 51200 bytes written 200: 102400 bytes written 300: 153600 bytes written 400: 204800 bytes written 500: 256000 bytes written 600: 307200 bytes written 700: 358400 bytes written 800: 409600 bytes written 900: 460800 bytes written 900: error in print: A file cannot be larger than the value set by uli +mit. at f.pl line 23.

    Note further that the above was run with the korn shell (ksh). The ulimit command is a shell builtin and the block size, on aix at least, seems to be 512 bytes for ksh and 1024 bytes for bash, as shown below:

    % type ulimit ulimit is a shell builtin. % ulimit -f 900 % ulimit -f 900 % ksh -c 'ulimit -f' 900 % bash -c 'ulimit -f' 450

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1113028]
Approved by Athanasius
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2022-05-23 12:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (82 votes). Check out past polls.

    Notices?