Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Unsigned 64-bit integer as Judy key

by Anonymous Monk
on Dec 06, 2022 at 16:42 UTC ( [id://11148614] : perlquestion . print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

$ perl -MJudy::L=Set -e 'Set($j, 1<<63, 123)' Coercing -9223372036854775808 to 0. Can't use negative values as keys. + at -e line 1.

Can anyone please test if they get same error? Argument can't be treated as signed int by design, can it? Or is my system/Perl/Judy broken?

Replies are listed 'Best First'.
Re: Unsigned 64-bit integer as Judy key
by swl (Parson) on Dec 07, 2022 at 06:04 UTC

    Do Judy arrays index from 1 instead of zero? Subtracting 1 from the key works without error.

    perl -MJudy::L=Set -E 'Set($j, (1<<63)-1, 123); say $j' 94763462791888

    And possibly not relevant, but I see these overflow warnings when compiling:

    ib/Judy.xs:54:37: note: in definition of macro ‘OOGA’ 54 | PerlIO_printf(PerlIO_stdout(),__VA_ARGS__);\ | ^~~~~~~~~~~ lib/Judy.xs: In function ‘XS_Judy__HS_Delete’: lib/Judy.xs:46:26: warning: overflow in conversion from ‘long unsigned + int’ to ‘int’ changes value from ‘16045690984833335023’ to ‘-5590387 +37’ [-Woverflow] 46 | # define DEADBEEF 0xdeadbeefdeadbeef | ^~~~~~~~~~~~~~~~~~ lib/Judy.xs:46:26: note: in definition of macro ‘DEADBEEF’ 46 | # define DEADBEEF 0xdeadbeefdeadbeef | ^~~~~~~~~~~~~~~~~~

    Perl version:

    perl -v | grep This This is perl 5, version 36, subversion 0 (v5.36.0) built for x86_64-li +nux
Re: Unsigned 64-bit integer as Judy key
by bliako (Monsignor) on Dec 07, 2022 at 08:27 UTC

    How was your perl compiled?: perl -V:ivsize

    It could well be that this has nothing to do with Judy but that for your Perl, 1<<63 overflows and becomes negative which Judy then receives it and complains.

    Also, from Judy source, Set() accepts a key as UWord_t which is T_UWORD which is:

    T_UWORD /* Accept: - IV that's -1. - IV that's negative, coerce to 0 and warn. - IV/UV that fits in (unsigned long int) - IV/UV that requires more bits than fit in (unsigned long int). Truncate it and throw a warning. - Cast everything else to UV and apply the above rules */ ...

    (edit: https://metacpan.org/release/JJORE/Judy-0.41/source/lib/Judy.xs and https://metacpan.org/release/JJORE/Judy-0.41/source/typemap)

    edit: IV: integer value, UV: unsigned valued. In my system with ivsize=8, perl -e 'print 1<<63' outputs 9223372036854775808, do you get the same?

    bw, bliako

      Thanks everyone for answers.

      https://metacpan.org/release/JJORE/Judy-0.41/source/typemap

      oh, but that explains it all. The argument is treated by default and by design as signed IV. Which is a bug in Perl Judy distribution, as very strange as it is. I understand it is "0-dot-something" version, but it's strange and sad this wasn't noticed in 10 years. Conditionals starting from line #88 should first check if $arg is an UV.

      For example:

      >perl -MDevel::Peek -E "Dump 1<<63" SV = IV(0x3e42b48) at 0x3e42b58 REFCNT = 1 FLAGS = (PADTMP,IOK,READONLY,PROTECT,pIOK,IsUV) UV = 9223372036854775808

      How can I check for IsUV flag in XS? Can't find it in perlguts. Hopefully, this easy fix in typemap file won't break anything else.

        so I changed that line #88 to

        if ( !SvIsUV($arg) && SvIOK($arg) && SvIV($arg) < 0 ) {

        now Judy installs OK with its tests, and a test I posted in OP passes OK. Actually, it's all about 11148465 meditation; I knew Judy technology is amazing and just wanted to check this yet again. With latest llil2d.pl I get

        llil2d start get_properties : 10 secs sort + output : 21 secs total : 31 secs

        and 2263M of htop's RES, with my Judy script:

        my_test start get_properties: 16 secs sort + output: 21 secs total: 37 secs

        and 378M of htop's RES. And like I said (in my plan outline there) "words" are not stored in RAM (suppose they are long strings) but read again while writing output, hence relatively long time in 2nd row with almost instantaneous sort by Sort::Packed. I'll post to that thread later.

Re: Unsigned 64-bit integer as Judy key
by Corion (Patriarch) on Dec 07, 2022 at 08:23 UTC

    I can confirm this for a Perl with 64bit integers - maybe the results are different for a 32bit Perl but I don't have one anymore.

    perl -V:ivsize ivsize='8';
Re: Unsigned 64-bit integer as Judy key
by 1nickt (Canon) on Dec 06, 2022 at 23:17 UTC

    Hello,

    I installed Judy::L and ran your one-liner. I got the same output you did.

    I know nothing of the module, but are you sure it's an error and not just a warning? You can print the value of $j, which suggests it is.

    $ perl -MJudy::L=Set -E 'Set($j, 1<<63, 123); say $j' Coercing -9223372036854775808 to 0. Can't use negative values as keys. + at -e line 1. 105553159750592

    Hope this helps!


    The way forward always starts with a minimal test.