# newbie questions, sum over range, rounding of special problematic values

• From: newbie-02 gmx de
• To: gnumeric-list gnome org
• Subject: newbie questions, sum over range, rounding of special problematic values
• Date: Thu, 6 May 2021 13:03:52 +0200

hi @all, i'm new here, pls. excuse if i am not well adapted to the manners

i'm studying how to get correct results for floating point calculations, and am curious about three points:

1. gnumeric produces much better sums over ranges than other spreadsheets which i have seen (LO calc and MS excel), can somebody explain by which means this is achieved ('kahan summation' or 'summand ordering' or 'using FPU registers and long doubles' or other?) and / or give me a code pointer?

2. gnumeric still fails for additions where the ordering / sequencing of the summands is dictated (e.g. '=1E16 + 0,1 -1E16 ...'), is there any idea to enhance that? (school math would allow 'associative reordering' if the result is correct? (less wrong as without reordering?)),

3. gnumeric produces quite accurate results (3.000000000000000444E-01 for '=0.1 + 0.2'), but has weaknesses when rounding that to e.g. 16 decimals digits, results in 3.000000000000000999E-01 which is 'enhancing the fail'. i suspect the value 3.000000000000000444E-01 is compared to the correct IEEE 754 representation of '0.3' (2.999999999999999889E-01), and then the difference of 5.551115123125782702E-17 tends to a round up instead of - correct - rounding down. that blocks 'corrective rounding' :-( :-( :-( as you have to decide either:
(a) - 'round with least impact' and 'risk increasing the problem' (as in above example), or
(b) - 'round to less digits' and risk 'rounding away valid content' (including the risk that an '(a)' problem might punch through by domino effect for values with 000000 or 999999 strings. onyone an idea how to do better? imho calculating the 17. decimals digit - as it's done for display - would do the trick, but might be (too much?) effort as it requires the full bin -> dec conversion ... ???

(all my tries with ver 1.12.17 win portable, if newer versions under linux are better just tell me),

besides being a little slow compared to calc and excel ... i like gnumeric very much!

reg.

b.

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]