Gnumeric's rounding here is indeed done with
round_to_int(x * 10^d) / 10^d
except that round_to_int deliberately misrounds 0.5-1ulp to 1. That's
not important here.
This means that, in general, you should not expect last-bit accuracy
for this operation. Both the multiplication and the division can
introduce rounding errors. I am willing to entertain patches that fix
that if (a) someone is sufficiently interested to do the work, and (b)
can do it in a sane way that doesn't involve making strings out of
numbers. This is not a beginner's task.
But note that the function implemented would be x ->
round_to_base2(round_to_base10(x,d),53). In other words, you are
inherently double rounding. Allow me to point out that while
well-defined, it is not a very sane thing to do, except in the cases
where the outer round is known to be a no-op.
There are two types of rounding in Gnumeric: the rounding functions
such as ROUND. These change the values. Then there is the rounding
implied by attaching a format to a cell. That does not change the
value, only its display: even if you format A1 to show two decimals,
then another cell containing =A1*42 will still see the full precision
of A1.
Morten
_______________________________________________
gnumeric-list mailing list
gnumeric-list gnome org
https://mail.gnome.org/mailman/listinfo/gnumeric-list