Aw: Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

[newbie nullzwei <newbie-02 gmx de>]

hello @all, hello @Morten,

Morten pointed out:

> "There are two types of rounding in Gnumeric: the rounding functions
such as ROUND. These change the values. Then there is the rounding
implied by attaching a format to a cell. That does not change the
value, only its display:"

theese roundings partly produce different results, e.g.
0,24999999999999983 is displayed (16 decimals) as
0,2499999999999998  but "=round(0,24999999999999983;16)" produces
0,2499999999999999

(see red marked cells in attached sheet).

i'd like to learn how rounding for display is done, some of it might be useful for improvements i'm thinking about.

thus i'd like to know where it is in the code, can anybody help with a pointer?

- and i'd still like to know which routine doe's 'input parsing' and converting it into values -

tia, best regards,



b.
 

---
 

Gesendet: Donnerstag, 08. Juli 2021 um 00:16 Uhr
Von: "Morten Welinder" <mortenw gnome org>
An: "Steven D'Aprano" <steve pearwood info>
Cc: "Gnumeric Mailing List" <gnumeric-list gnome org>
Betreff: Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

Gnumeric's rounding here is indeed done with

round_to_int(x * 10^d) / 10^d

except that round_to_int deliberately misrounds 0.5-1ulp to 1. That's
not important here.
 

Attachment: differences_between_round_and_display.gnumeric
Description: application/gnumeric