follow up - newbie questions, sum over range, rounding of special problematic values

hi @all, i'm new here, pls. excuse if i am not well adapted to the manners,

(asked something some time ago, got answer from a 'private email', but none to follow up questions, thus try to write to the list again, sorry if wrong,

>> 1. gnumeric produces much better sums over ranges than other spreadsheets which i have seen (LO calc and MS excel), can somebody explain by which means this is
>> achieved ('kahan summation' or 'summand ordering' or 'using FPU registers and long doubles' or other?) and / or give me a code pointer?

> 1. We use an extension of Kahan summation, see

thank you, i'm not a coder, looks nice and short and - as i have seen - works very good!,
is it ok to suggest the people at LibreOffice to have a look at this, there is one who wants to implement Kahan (with moderate success so far), and another is a little paranoid and thinks LibreOffice would get into trouble just by looking at your code / that of gnumeric at all, see

>> 3. gnumeric produces quite accurate results (3.000000000000000444E-01 for '=0.1 + 0.2'), but has weaknesses when rounding that to e.g. 16 decimals digits, results in
>> 3.000000000000000999E-01 which is 'enhancing the fail'. ...

> 3. This is based on misunderstandings. Our calculations are based on
base 2.

there are other questionable rounding results, e.g. '=rounddown(0.249999999999999972244,16)' results in  0.250000000000000000000, while '=roundup(0.249999999999999972244,16)' produces 0.249999999999999888978 which - acc. my simple math knowledge - should be the other way around? (0.249999999999999972244 results from calculating 0.25 minus one ULP, i'd expect a 'roundDOWN' always to be less or equal to a 'roundUP', i suspect a systematic error, but might be wrong in my expectations as well, bear with me if the latter ... ),     



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]