Re: Rounding??

["Andreas J. Guelzow" <aguelzow math concordia ab ca>]

Morten Welinder wrote:

>    Gnumeric seems to do all sorts of things I cant quite figure out.
>
>    The _important_ thing is to have _consistent_ behaviour.
>
>    At the moment
>
>    3.15, 3.25, 3.35, 3.45, 3.55, 3.65, 3.75, 3.85, 3.95, 4.05, 4.15
>    gives me
>    3.1,  3.2,  3.3,  3.5,  3.5,  3.6,  3.7,  3.8,  4.0,  4.0,  4.2
>
>    which just seems confusing.  Why are 3.4 and 3.9 missed??
>
>
> Think binary.
>
That's what I suspected (without doing the arithmetic), but the internal
representation should _not_ influence the expected rounding behaviour.
Without having looked at the source I suspect that the remainder is
simply being compared with 5,...

It should really cut at a value wich causes the expected behaviour for
decimals that are only a few digits longer. (There will probably always
be the problem that some values very close to the cutoff will round
incorrectly.)

Andreas

--
Prof. Dr. Andreas J. Guelzow
Chair of Science, Assoc. Prof. of Math.
Concordia University College of Alberta
http://www.math.concordia.ab.ca/aguelzow
Assoc. Prof. of Math.