Re: follow up II - newbie questions, sum over range, rounding of special problematic values

On Sun, May 23, 2021 at 12:42:29PM -0600, Andreas J. Guelzow wrote:

On 2021-05-22 7:26 p.m., Steven D'Aprano wrote:

input        0.24999999999999997     

You are playing at the edge of the maximum mantissa length. Did you ever 
try using that number in LibreOffice or Excel?

I'm not discussing LibreOffice bugs on this gnumeric mailing list, we'd 
be here for years :-(

As far as I can tell, LibreOffice doesn't even allow you to create the 
number 0.24999999999999997 but immediately rounds it to 0.25.

Do you know what is the internal format for floating point numbers in 
gnumeric? I assume it is 64-bit IEEE-754 floats.

Do you know the algorithm used for round-down and round-up?


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]