Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Re: Bad math

by Marshall (Prior)
on Oct 31, 2011 at 15:04 UTC ( #934899=note: print w/ replies, xml ) Need Help??

in reply to Bad math

A floating point number will be internally represented by a mantissa and an exponent. This is of course in base 2. The fractional parts of the numbers you show cannot be represented exactly in base 2. Hence there are more binary digits than there are decimal digits and some precision is lost in the conversion process. This is normal. Your solution is the right way to do this.
Comment on Re: Bad math

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://934899]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (6)
As of 2016-04-30 16:17 GMT
Find Nodes?
    Voting Booth?
    :nehw tseb si esrever ni gnitirW

    Results (441 votes). Check out past polls.