Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

This is a common problem with floating point numbers. In general, you cannot rely on floating point numbers to be "rounded" to any precision unless the rounding is actually part of the program logic or underlying language/library. This happens with whole numbers used as floating points, too. For instance, 39 might become 38.999999999999, which can be a real problem if not handled correctly.

Small differences in rounding occur because floating point numbers are stored in a relatively small region of memory, and the mantissa and exponent are stored as powers of 2, rather than powers of 10, as you see on screen if you just print it out. As such, it's very improbable that a given number in base-10 (decimal) arithmetic will have a base-2 (binary) equivalent that gives the exact same number. The number will be close (yours broke on the seventeenth significant figure), but rarely exact. DBI and SQL Developer are likely clipping off spurious digit(s) from the result, while ASE is not, or it could be differences in machine/data architectures.

Do you actually need 17 significant figures, as your example indicates? Or do you just need the results to be the same across all tools? If it's the former, you will have to rethink your data design to include larger floating point numbers (larger means more precise with floating point). If it's the latter, I would just round the result off to whatever precision you need (and can safely expect to maintain).


In reply to Re: Rounding off the scale value after decimal point by rjt
in thread Rounding off the scale value after decimal point by venu_hs

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others romping around the Monastery: (4)
    As of 2014-10-22 05:32 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      For retirement, I am banking on:










      Results (112 votes), past polls