Re^3: Integers sometimes turn into Reals after substraction (error correction ?)

[pryrt]

IEEE double-precision float is 53 bit float. 0.00000000002910383046 = 2**-35. With the 18 bits for the integral portion of 255160 or 256160, that's the full 53bits. So, apparently, when you take the 53bit fractional representation of 255.16 multiplied by 1000dec, round-off sets the final bit to 0; when you take 256.16 * 1000, round-off sets the final bit to 1. Now (when I have time) I'm going to have to figure out the formula to figure out which powers of two will have this error and which won't. It all depends on the 53rd bit of the product of 2**N+16/25 times 128 (same as the bits of times 1000, since the factor of 8 is a binary shift). Oh, I'm nearly there... the 125 is 0b111_1101... and that will just interact with the 53rd bit... urgh but I've got to leave now... I guess it's an exercise for later. :-)

[pryrt]

Convert ieee floats into hex notation, for each of 2**n - 1 and 2**n:

cmd.exe> perl -e "use 5.010; for(0..31){$e=2**$_; $f=$e-1; $str = join
+ '', map { sprintf '%02x ', ord } split(//, pack('dddd', $f+4/25, $e+
+4/25, ($f+4/25)*1000, ($e+4/25)*1000)); $str =~ s/(.{24})(.{24})(.{24
+})(.{24})/\1\t\2\t\3\t\4/; say qq{$_\t$str};}"
0       7b 14 ae 47 e1 7a c4 3f         8f c2 f5 28 5c 8f f2 3f       
+  00 00 00 00 00 00 64 40         00 00 00 00 00 20 92 40
1       8f c2 f5 28 5c 8f f2 3f         48 e1 7a 14 ae 47 01 40       
+  00 00 00 00 00 20 92 40         00 00 00 00 00 e0 a0 40
2       48 e1 7a 14 ae 47 09 40         a4 70 3d 0a d7 a3 10 40       
+  00 00 00 00 00 b0 a8 40         00 00 00 00 00 40 b0 40
3       a4 70 3d 0a d7 a3 1c 40         52 b8 1e 85 eb 51 20 40       
+  00 00 00 00 00 f8 bb 40         00 00 00 00 00 e0 bf 40
4       52 b8 1e 85 eb 51 2e 40         29 5c 8f c2 f5 28 30 40       
+  00 00 00 00 00 9c cd 40         00 00 00 00 00 90 cf 40
5       29 5c 8f c2 f5 28 3f 40         14 ae 47 e1 7a 14 40 40       
+  00 00 00 00 00 6e de 40         ff ff ff ff ff 67 df 40
6       14 ae 47 e1 7a 94 4f 40         0a d7 a3 70 3d 0a 50 40       
+  00 00 00 00 00 d7 ee 40         00 00 00 00 00 54 ef 40
7       0a d7 a3 70 3d ca 5f 40         85 eb 51 b8 1e 05 60 40       
+  00 00 00 00 80 0b ff 40         00 00 00 00 00 4a ff 40
8       85 eb 51 b8 1e e5 6f 40         c3 f5 28 5c 8f 02 70 40       
+  00 00 00 00 c0 25 0f 41         01 00 00 00 00 45 0f 41
9       c3 f5 28 5c 8f f2 7f 40         e1 7a 14 ae 47 01 80 40       
+  00 00 00 00 e0 32 1f 41         ff ff ff ff 7f 42 1f 41
10      e1 7a 14 ae 47 f9 8f 40         71 3d 0a d7 a3 00 90 40       
+  00 00 00 00 70 39 2f 41         01 00 00 00 40 41 2f 41
11      71 3d 0a d7 a3 fc 9f 40         b8 1e 85 eb 51 00 a0 40       
+  00 00 00 00 b8 3c 3f 41         ff ff ff ff 9f 40 3f 41
12      b8 1e 85 eb 51 fe af 40         5c 8f c2 f5 28 00 b0 40       
+  00 00 00 00 5c 3e 4f 41         00 00 00 00 50 40 4f 41
13      5c 8f c2 f5 28 ff bf 40         ae 47 e1 7a 14 00 c0 40       
+  00 00 00 00 2e 3f 5f 41         00 00 00 00 28 40 5f 41
14      ae 47 e1 7a 94 ff cf 40         d7 a3 70 3d 0a 00 d0 40       
+  00 00 00 00 97 3f 6f 41         00 00 00 00 14 40 6f 41
15      d7 a3 70 3d ca ff df 40         ec 51 b8 1e 05 00 e0 40       
+  00 00 00 80 cb 3f 7f 41         01 00 00 00 0a 40 7f 41
16      ec 51 b8 1e e5 ff ef 40         f6 28 5c 8f 02 00 f0 40       
+  00 00 00 c0 e5 3f 8f 41         00 00 00 00 05 40 8f 41
17      f6 28 5c 8f f2 ff ff 40         7b 14 ae 47 01 00 00 41       
+  00 00 00 e0 f2 3f 9f 41         00 00 00 80 02 40 9f 41
18      7b 14 ae 47 f9 ff 0f 41         3d 0a d7 a3 00 00 10 41       
+  00 00 00 70 f9 3f af 41         ff ff ff 3f 01 40 af 41
19      3d 0a d7 a3 fc ff 1f 41         1f 85 eb 51 00 00 20 41       
+  00 00 00 b8 fc 3f bf 41         01 00 00 a0 00 40 bf 41
20      1f 85 eb 51 fe ff 2f 41         8f c2 f5 28 00 00 30 41       
+  00 00 00 5c fe 3f cf 41         ff ff ff 4f 00 40 cf 41
21      8f c2 f5 28 ff ff 3f 41         48 e1 7a 14 00 00 40 41       
+  00 00 00 2e ff 3f df 41         01 00 00 28 00 40 df 41
22      48 e1 7a 94 ff ff 4f 41         a4 70 3d 0a 00 00 50 41       
+  00 00 00 97 ff 3f ef 41         00 00 00 14 00 40 ef 41
23      a4 70 3d ca ff ff 5f 41         52 b8 1e 05 00 00 60 41       
+  00 00 80 cb ff 3f ff 41         00 00 00 0a 00 40 ff 41
24      52 b8 1e e5 ff ff 6f 41         29 5c 8f 02 00 00 70 41       
+  00 00 c0 e5 ff 3f 0f 42         00 00 00 05 00 40 0f 42
25      29 5c 8f f2 ff ff 7f 41         14 ae 47 01 00 00 80 41       
+  00 00 e0 f2 ff 3f 1f 42         ff ff 7f 02 00 40 1f 42
26      14 ae 47 f9 ff ff 8f 41         0a d7 a3 00 00 00 90 41       
+  00 00 70 f9 ff 3f 2f 42         00 00 40 01 00 40 2f 42
27      0a d7 a3 fc ff ff 9f 41         85 eb 51 00 00 00 a0 41       
+  00 00 b8 fc ff 3f 3f 42         00 00 a0 00 00 40 3f 42
28      85 eb 51 fe ff ff af 41         c3 f5 28 00 00 00 b0 41       
+  00 00 5c fe ff 3f 4f 42         01 00 50 00 00 40 4f 42
29      c3 f5 28 ff ff ff bf 41         e1 7a 14 00 00 00 c0 41       
+  00 00 2e ff ff 3f 5f 42         ff ff 27 00 00 40 5f 42
30      e1 7a 94 ff ff ff cf 41         71 3d 0a 00 00 00 d0 41       
+  00 00 97 ff ff 3f 6f 42         01 00 14 00 00 40 6f 42
31      71 3d ca ff ff ff df 41         b8 1e 05 00 00 00 e0 41       
+  00 80 cb ff ff 3f 7f 42         ff ff 09 00 00 40 7f 42

n       2**n - 1                        2**n                          
+  (2**n - 1)*1000                 (2**n) * 1000