Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

vec overflow?

by Anonymous Monk
on Jun 25, 2013 at 00:42 UTC ( #1040499=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I seem to be encountering some kind of overflow bug with vec. To wit:

memtime perl -e '$h = ""; $x = (2**31)-2; vec($h,$x,8) = 0;' Exit [0] 0.94 user, 1.34 system, 2.30 elapsed -- Max VSize = 2115872KB, Max RSS + = 2042140KB

Runs as expected, but:

memtime perl -e '$h = ""; $x = (2**31)-1; vec($h,$x,8) = 0;' Out of memory! Exit [1] 0.00 user, 0.00 system, 0.10 elapsed -- Max VSize = 3720KB, Max RSS = +192KB

Seems to ask for too much memory, and

memtime perl -e '$h = ""; $x = (2**31); vec($h,$x,8) = 0;' Negative offset to vec in lvalue context at -e line 1. Exit [255] 0.00 user, 0.00 system, 0.10 elapsed -- Max VSize = 3720KB, Max RSS = +128KB

has another problem altogether.

Is this a problem with vec? My Perl build? My syntax? Any workarounds?

Comment on vec overflow?
Select or Download Code
Re: vec overflow?
by LanX (Canon) on Jun 25, 2013 at 00:48 UTC
    OFFSET is a signed integer and most Perl compilations restrict them to a maximum of 31 bits.

    If you need a larger OFFSET you can switch to a 64 bit platform.

    vec EXPR,OFFSET,BITS
                   Treats the string in EXPR as a bit vector made up of elements
                   of width BITS, and returns the value of the element specified
                   by OFFSET as an unsigned integer. ...
    

    Or just split the EXPR into smaller chunks and do basic arithmetics for calculating OFFSET.

    Cheers Rolf

    ( addicted to the Perl Programming Language)

    update

    ) doesn't help, according to BrowserUk

    update

    see older discussions and proposed workarounds

    update

    this workaround doesn't look efficient, since substr is an lvalue you can use constructs like

    vec ( substr( $str, OFFSET_HIGH, 2**31), OFFSET_LOW, BITS ) = NEW

    (well if substr allows OFFSETs that big ... can't test!)

      this workaround doesn't look efficient

      It is. It takes an lvalue reference (a scalar value) and passes that to vec. It is marginally less efficient than vec( substr( ... ), ... );, but not so much that you would be able to detect it.

      However, substr can only provide an lvalue for byte-sized units; where vec can provide an lvalue for 8 byte units.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        Ok, I had errors testing it out in the debugger, but turned out that just the automatic Data::Dump couldn't handle lvalues.

        Asa suggestion:

        I think using a foreach alias like shown in the docs for substr is a clearer alternative to lvalue-refs.

        Cheers Rolf

        ( addicted to the Perl Programming Language)

Re: vec overflow?
by BrowserUk (Pope) on Jun 25, 2013 at 00:58 UTC
    Is this a problem with vec? My Perl build? My syntax? Any workarounds?

    It's a problem with vec. Internally it is using a signed 32-bit values, even on 64-bit builds.

    One workaround I've used is to break the vector into subsections and use bitmasking to select the appropriate one:

    @vecs = ( chr(0)x2**29 ) x 8;; $n = 2**31; vec( $vecs[ $n >> 29 ], $n & 0x1fffffff, 1 ) = 1;;

    Not very satisfactory, but workable.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Thanks very much. Good to know.

        Another workaround that is useful if your algorithm benefits from having a single contiguous bitvector -- for example if you want to count the set bits quickly using my $popcount = unpack '%32b*', $bitvector; -- is to nest calls to vec. Eg:

        vec( vec( $bitvector, $n >> 5, 64 ), $n & 0x1f, 1 ) = 1;

        Theoretically, as neither offset breaches the 2**31-1 barrier, this can allow you to address bitvectors up to 16GB/137 billion bits, though I don't have enough memory to try it.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1040499]
Front-paged by BrowserUk
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (6)
As of 2014-10-01 01:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (386 votes), past polls