Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

[XS] Weird behaviour on some Linux systems

by syphilis (Bishop)
on Oct 10, 2019 at 12:09 UTC ( #11107302=perlquestion: print w/replies, xml ) Need Help??

syphilis has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

The demo:
use strict; use warnings; use Inline C => Config => BUILD_NOISY => 1, ; use Inline C => <<'EOC'; void foo(SV * arg) { SV * keysv = sv_newmortal(); /* Make sure arg is an NV */ if(!SvNOK(arg)) croak("Not a valid arg passed to foo for this demo") +; printf( "NV: %.19" NVgf, SvNV(arg)); sv_setpvf(keysv, "%.19" NVgf, SvNV(arg)); printf("\nsv: %s\n\n", SvPV_nolen(keysv)); } void bar(SV * arg) { SV * keysv = sv_newmortal(); char buff[30]; /* Make sure arg is an NV */ if(!SvNOK(arg)) croak("Not a valid arg passed to bar for this demo" +); sprintf(buff, "%.19" NVgf, SvNV(arg)); printf("NV: %s", buff); sv_setpvf(keysv, "%s", buff); printf("\nsv: %s\n\n", SvPV_nolen(keysv)); } EOC my @v = (2 ** 55, 2 ** 56, 2 ** 57, 2 ** 58, 2 ** 63); foo($_) for @v; print "################\n\n"; bar($_) for @v;
On Windows (perl-5.30.0), output is as I expect:
NV: 36028797018963968 sv: 36028797018963968 NV: 72057594037927936 sv: 72057594037927936 NV: 144115188075855872 sv: 144115188075855872 NV: 288230376151711744 sv: 288230376151711744 NV: 9223372036854775808 sv: 9223372036854775808 ################ NV: 36028797018963968 sv: 36028797018963968 NV: 72057594037927936 sv: 72057594037927936 NV: 144115188075855872 sv: 144115188075855872 NV: 288230376151711744 sv: 288230376151711744 NV: 9223372036854775808 sv: 9223372036854775808
Note that foo() and bar() produce identical outputs.

However, on Ubuntu and Debian (perl-5.30.0), while bar() produces the same output as above, foo() does not:
NV: 36028797018963968 sv: 36028797018963968 NV: 72057594037927936 sv: 72057594037927936 NV: 144115188075855872 sv: 1.4411518807585587e+17 NV: 288230376151711744 sv: 2.8823037615171174e+17 NV: 9223372036854775808 sv: 9.2233720368547758e+18 ################ NV: 36028797018963968 sv: 36028797018963968 NV: 72057594037927936 sv: 72057594037927936 NV: 144115188075855872 sv: 144115188075855872 NV: 288230376151711744 sv: 288230376151711744 NV: 9223372036854775808 sv: 9223372036854775808
In foo() I assign "%.19" NVgf, SvNV(arg) directly to the SV "keysv" via sv_setpvf.

In bar() I assign "%.19" NVgf, SvNV(arg) directly to the string buffer "buff" via sprintf.
And then I assign "buff" to "keysv" via sv_setpvf.

I'm very surprised at foo's different behaviour on Linux - to the extent that it looks buggy, to me.
Any thoughts on that ?

Cheers,
Rob

PS
The perls I'm running are perls whose NV ($Config{nvtype}) is 'double'.
And the behaviour I'm wanting is that exhibited by bar(). (The bar function is actually my workaround for the problem that foo() presents.)
Debian Wheezy and Ubuntu-18.04 suffer from this problem, but freebsd-12.0 does not. (Those three systems, and windows 7, are the only systems I've tested.)

Replies are listed 'Best First'.
Re: [XS] Weird behaviour on some Linux systems
by Corion (Pope) on Oct 10, 2019 at 12:15 UTC

    My unfounded random guess would be that on the problematic system, Perl brings its own implementation of sprintf and the OS (or C-lib) implementation of printf differs from the Perl-supplied implementation.

    I assume that if you watch the output of ./Configure closely enough you can find what it thinks about the (un)availability of sprintf and how it replaces those.

      Thanks for the quick feedback, Corion ... not so sure that you're on the right track, but I certainly can't prove otherwise.

      Here's a clearer demo that contains only one instance of "sprintf", and no occurrences of "printf":
      use strict; use warnings; use Devel::Peek; use Inline C => Config => BUILD_NOISY => 1, ; use Inline C => <<'EOC'; SV * foo(SV * arg) { SV * keysv; keysv = NEWSV(0, 32); /* Make sure arg is an NV */ if(!SvNOK(arg)) croak("Not a valid arg passed to foo for this demo") +; sv_setpvf(keysv, "%.19" NVgf, SvNV(arg)); return keysv; } SV * bar(SV * arg) { SV * keysv; char buff[30]; keysv = NEWSV(0, 32); /* Make sure arg is an NV */ if(!SvNOK(arg)) croak("Not a valid arg passed to bar for this demo" +); sprintf(buff, "%.19" NVgf, SvNV(arg)); sv_setpvf(keysv, "%s", buff); return keysv; } EOC my $sv_foo = foo(2 ** 63); my $sv_bar = bar(2 ** 63); Dump $sv_foo; print "#####################\n"; Dump $sv_bar;
      On Windows 7 it produces (sanely, IMO):
      SV = PV(0x72be48) at 0x39bfb8 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x33b748 "9223372036854775808"\0 CUR = 19 LEN = 34 ##################### SV = PV(0x72be98) at 0x33c638 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x33b8d8 "9223372036854775808"\0 CUR = 19 LEN = 34
      On Ubuntu-18.04 I see the (insane) output of:
      SV = PV(0x56545ac36fd0) at 0x56545ac5b728 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x56545ac57170 "9.2233720368547758e+18"\0 CUR = 22 LEN = 34 ##################### SV = PV(0x56545ac37070) at 0x56545acb8178 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x56545b64be20 "9223372036854775808"\0 CUR = 19 LEN = 34
      Interestingly, on Ubuntu, it's the XSub containing the "sprintf" call that's producing the sane result.
      (At least, it's producing the output that I expect and want ;-)

      Cheers,
      Rob
Re: [XS] Weird behaviour on some Linux systems
by jcb (Vicar) on Oct 10, 2019 at 22:09 UTC

    Please check what the NVgf macro expands to on the different platforms and check what ANSI C and POSIX say that conversion is supposed to actually do.

      Please check what the NVgf macro expands to on the different platforms ....

      The puzzle is that, for certain cases (ie the aforementioned Debian and Ubuntu builds of perl) it apparently expands to different things, depending upon how it is expanded.
      I find that mostly:
      sprintf(buff, "%.19" NVgf, SvNV(arg)); and sv_setpvf(keysv, "%.19" NVgf, SvNV(arg));
      produce the same result.
      That is, the string in "buff" is exactly the same as the string in the PV slot of "keysv".
      However, on some systems, with perls whose NV is 'double', and for integer values that require more than 17 digits, the string in "buff" differs from the string in the PV slot of "keysv".

      Corion has suggested that this might be explained in terms of differences between perl's sprintf() and libc's sprintf().
      My only reason for doubting him is that in 15 years of fiddling around with (s)printf on these systems, I have encountered no such issue. (But as soon as I start trying to get NVgf to work as I expect with sv_setpvf, I hit issues.)

      ... and check what ANSI C and POSIX say that conversion is supposed to actually do

      Yes ... I've had no luck in finding any info on "NVgf" anywhere.
      However, I've just found that replacing:
      sv_setpvf(keysv, "%.19" NVgf, SvNV(arg)); with sv_setpvf(keysv, "%.19g", SvNV(arg));
      makes no difference at all (at least when NV is 'double').
      Given that I reckon I know what "%.19g" should expand to, this would suggest that I don't really need to wonder about what NVgf expands to - rather concentrate more on what sv_setpvf is doing (and on Corion's suggestion).
      Or just settle for the workaround that I've already got :-)

      Cheers,
      Rob

        NVgf is part of Perl, not ANSI C or POSIX. It is a preprocessor macro that, from context, must expand to a string constant that completes a printf conversion sequence. What does sv_setpvf(keysv, "NVgf: \"%s\"", NVgf) produce? (That might produce an error; I am guessing that sv_setpvf behaves like C sprintf but puts the result into a Perl SV.) From context, that must be a printf conversion character; what do ANSI C and POSIX say that character is supposed to do?

Re: [XS] Weird behaviour on some Linux systems
by Haarg (Curate) on Oct 12, 2019 at 11:06 UTC

    sv_setpvf, which is implemented by Perl_sv_vcatpvfn_flags does not use sprintf directly. Some internal parts will be be handled by sprintf, but a lot of that depends on how perl is configured.

    Assuming this you are not using a quadmath or longdouble build, %.19g is special cased to use SNPRINTF_G, which is a macro for Gconvert which is a macro determined at Configure time and can point to several different functions.

    On my Ubuntu system, it is using gcvt:

    $ perl -V:d_Gconvert
    d_Gconvert='gcvt((x),(n),(b))';
    

    On my macOS system, it is using sprintf:

    $ perl -V:d_Gconvert
    d_Gconvert='sprintf((b),"%.*g",(n),(x))';
    

    The perlclib documentation explicitly says that perl functions tend not to use the standard library. While it documents how to do things "the Perl way", it doesn't say that the functions will behave identically.

    I'm not entirely clear if this is a bug or not. The documentation for this is under sprintf in perlfunc. While this behavior would seem to be violating the spirit of specifying the width, it is technically within the maximum width being provided.

      UPDATE: After I sent off this post it occurred to me that the issues Haarg had just explained were very likely the same issues that Corion had hinted at earlier ;-)

      On my Ubuntu system, it is using sprintf: $ perl -V:d_Gconvert d_Gconvert='gcvt((x),(n),(b))';

      Yes - same for my Ubuntu system.
      That should mean that you should also find that the script I posted in my last post outputs:
      1..1 not ok 1 - they are the same # Failed test 'they are the same' # at w.pl line 40. # got: '9.2233720368547758e+18' # expected: '9223372036854775808' # Looks like you failed 1 test of 1.
      The unexpected '9.2233720368547758e+18' arises because:
      sv_setpvf(keysv, "%.19g", SvNV(arg)); is, in effect, converted to: sv_setpvf(keysv, "%.17g", SvNV(arg));
      Can we rely on this value of $Config{d_Gconvert} to reliably indicate that such behaviour will occur ? (If so, that would be most useful.)

      For my Windows 7 builds that avoid the problem, I'm finding that perl -V:d_Gconvert matches the output you're seeing on your macOS.

      For sane purposes on perl's whose nvsize is double, a width of 17 decimal digits is indeed sufficient.
      But when you're trying to (mis)use formatting of doubles in the way that List::Util::uniqnum() does, then you really do need a width of 19 decimal digits if ivsize is also 8.
      Otherwise, for example, uniqnum(1 << 60, 2 ** 60) won't detect that the 2 values are equal.

      About 6 hours ago I did submit a bug report about this to perlbug@perl.org but it hasn't yet been assigned a ticket number so I currently can't provide a link.
      It will be interesting to hear what p5p has to say.

      The perlclib documentation explicitly says that perl functions tend not to use the standard library. While it documents how to do things "the Perl way", it doesn't say that the functions will behave identically.

      I'm thinking that the perlclib docs could well make a point of stating that the functions might not behave identically - at least wrt sprintf vs sv_setpvf.

      Cheers,
      Rob
        But when you're trying to (mis)use formatting of doubles in the way that List::Util::uniqnum() does, then you really do need a width of 19 decimal digits if ivsize is also 8.

        I certainly sympathize. I've been trying to solve the same problem in List::Util::PP, and so far have settled on using roughly pack('FJ', $_, $_). There are still some issues around the unsigned int upper boundary though.

        It should be possible to avoid the special case to use Gconvert by tweaking the format you use. I can get the full width output on my Ubuntu system by using '%0.19g':

        $ perl -e'printf "%.19g\n", 2 ** 57'
        1.4411518807585587e+17
        $ perl -e'printf "%0.19g\n", 2 ** 57'
        144115188075855872
        

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11107302]
Approved by Athanasius
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (4)
As of 2020-12-02 09:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How often do you use taint mode?





    Results (37 votes). Check out past polls.

    Notices?