follow up: Aw: Re: 'testing' in gnumeric - compatibility problem between 'double' and 'long double' versions

hello @all,

ok, one problem identified which i had seen before, a 'long double' version reads in nonsense for fractional values stored by a 'double' version,

e.g. '=prob({0,1,2,3},{0.10000000000000000555,0.4000000000000000222,0.2000000000000000111,0.2999999999999999889},2)' instead of '=prob({0,1,2,3},{0.1,0.4,0.2,0.3},2)' for cell 'B75' of 'statfuns.xls' ...

difficult point, affecting each use of long double versions, thus i'd like to solve.

which code parts do write to and read from files? and perform the conversions for that? think read will use some 'strtod' and write some 'to_string'?

(already looked up that it writes '0.10000000000000001' instead of '0.1', pretending 17 digit precision which 64-bit doubles don't have (i know! they have 'some significance' in 17th digit in some ranges, but 'precision' (you can count ~1, ~2, ~3, ~4, ~5, ~6, ~7, and calculate 'one by one', just on the fly don't find a better word for it) ... they don't have anywhere, there are always some values missing!))

thus i'd like to try to store less - but then correct - digits,

such will inject other problems, i know!, have to look which and how much impact, but pls. help and let me give it a try ;-)

best regards,


Gesendet: Mittwoch, 22. September 2021 um 17:01 Uhr
Von: "Morten Welinder" <mortenw gnome org>
An: "newbie nullzwei" <newbie-02 gmx de>
Betreff: Re: 'testing' in gnumeric | was: Aw: Re: Re: where can / must one activate 'LONG DOUBLES'? - works to some extend, feedback and additional questions
I don't see an attachment, but when you load statfuns.xls you will see
the previously calculated values that are stored in the file. Press
F9 to force recalculation, then look for problems.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]