Data::Dumper turns floating points numbers into strings

[Hue-Bond]

Esteemed monks,

After playing with JSON::RPC::Client and obtaining the error "value is type str, expected real" (not sure if the error comes from client validation or from the server but that doesn't matter) I investigated and found that Data::Dumper was turning my floating point parameters into strings. I tried changing a couple of configuration variables listed in the module's documentation and was surprised to find that Useperl solves the problem:

use warnings;
use strict;
use Data::Dumper;
use JSON;

my $json = JSON->new;
for my $up (0, 1) {
    print "useperl: $up\n"; $Data::Dumper::Useperl = $up;
    my $data = { foo => 'bar', qux => 0.42 };

    printf "before: '%s'\n", $json->encode ($data);
    Dumper $data;
    #Data::Dumper->Dump ([$data], ['data']);
    printf "after:  '%s'\n", $json->encode ($data);
}
__END__
useperl: 0
before: '{"qux":0.42,"foo":"bar"}'
after:  '{"qux":"0.42","foo":"bar"}'
useperl: 1
before: '{"qux":0.42,"foo":"bar"}'
after:  '{"qux":0.42,"foo":"bar"}'

Both Dumper and Data::Dumper->Dump show this effect. When working with integers, we get more consistent results:

use JSON;
my $json = JSON->new;
for my $up (0, 1) {
    print "useperl: $up\n"; $Data::Dumper::Useperl = $up;
    my $data = { foo => 'bar', qux => 42 };

    printf "before: '%s'\n", $json->encode ($data);
    Dumper $data;
    #Data::Dumper->Dump ([$data], ['data']);
    printf "after:  '%s'\n", $json->encode ($data);
}
__END__
useperl: 0
before: '{"qux":42,"foo":"bar"}'
after:  '{"qux":42,"foo":"bar"}'
useperl: 1
before: '{"qux":42,"foo":"bar"}'
after:  '{"qux":42,"foo":"bar"}'

Is this difference in behaviour to be expected?

[ikegami]

When one stringifies a scalar, the string is attached to the scalar as an optimisation. It doesn't remove the existing number. This isn't a bug in D::D, but it doesn't mean that D::D couldn't be changed to avoid adding a string to the scalar. It already does in other circumstances.

(pPOK = has string)

>perl -MDevel::Peek -MData::Dumper -e"$_=0.42; Dumper($_); Dump($_);"
SV = NV(0x3335fc) at 0x328fe4
  REFCNT = 1
  FLAGS = (NOK,pNOK)
  NV = 0.42

>perl -MDevel::Peek -MData::Dumper -e"$_={ qux => 42 }; Dumper($_); Du
+mp($_->{qux});"
SV = IV(0x57b260) at 0x57b264
  REFCNT = 1
  FLAGS = (IOK,pIOK)
  IV = 42

>perl -MDevel::Peek -MData::Dumper -e"$_={ qux => 0.42 }; Dumper($_); 
+Dump($_->{qux});"
SV = PVNV(0x5490e4) at 0x25b264
  REFCNT = 1
  FLAGS = (NOK,POK,pNOK,pPOK)
  IV = 0
  NV = 0.42
  PV = 0x4c9f84 "0.42"\0
  CUR = 4
  LEN = 36

The problem is that JSON doesn't allow you to specify how to handle a variable that's both a float and a string.

[ig]

Note that this effect isn't unique to Data::Dumper. Any operation that stringifies a number or integer will change how JSON (the pure Perl implementation, I didn't look at the XS) represents it.

use strict;
use warnings;
use JSON;

my $json = JSON->new;
my $data = { foo => 'bar', qux => 0.42 };

printf "before: '%s'\n", $json->encode($data);
print "$data->{qux}\n";
printf "after: '%s'\n", $json->encode($data);

gives

before: '{"qux":0.42,"foo":"bar"}'
0.42
after: '{"qux":"0.42","foo":"bar"}'

This problem isn't unique to JSON either. I have seen similar issues in Win32::OLE, for example. Anything that inspects the scalar type and behaves differently depending on what it finds must resolve the ambiguity, shown by ikegami / Devel::Peek, that sometimes exists. When using such modules, you must be careful with your data.

One way to avoid the problem would be to make a copy of your data and stringify the copy.

[Hue-Bond]

Thank you both for the explanation about the stringification. Now the weird thing is that the Perl implementation of Data::Dumper doesn't add this string representation, while the XS one does...

[mje]

I had a similar problem with JSON::XS and DBD::Oracle. JSON::XS looks at the scalar and basically does if SvPOKp stringify else if SvNOKp treat as float else if SvIOKp treat as integer.

The problem is that DBD::Oracle does not allow you to bind a column as an integer. The object I wanted to JSONify was very large so all those extra " around integers were adding a lot to the size of the JSON encoded object. To get rid of the " I had to loop through my data and for every integer I had to add 0 to it - this took time. In the end we changed DBD::Oracle to add options to discard the pv.

I documented what we found and did at Support binding of integers in DBD::Oracle so they are returned as IVs