Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

[Morten Welinder <mortenw gnome org>]

Gnumeric's rounding here is indeed done with

    round_to_int(x * 10^d) / 10^d

except that round_to_int deliberately misrounds 0.5-1ulp to 1.  That's
not important here.

This means that, in general, you should not expect last-bit accuracy
for this operation.  Both the multiplication and the division can
introduce rounding errors.  I am willing to entertain patches that fix
that if (a) someone is sufficiently interested to do the work, and (b)
can do it in a sane way that doesn't involve making strings out of
numbers.  This is not a beginner's task.

But note that the function implemented would be x ->
round_to_base2(round_to_base10(x,d),53).  In other words, you are
inherently double rounding.  Allow me to point out that while
well-defined, it is not a very sane thing to do, except in the cases
where the outer round is known to be a no-op.

There are two types of rounding in Gnumeric: the rounding functions
such as ROUND.  These change the values.  Then there is the rounding
implied by attaching a format to a cell.  That does not change the
value, only its display: even if you format A1 to show two decimals,
then another cell containing =A1*42 will still see the full precision
of A1.

Morten