# Re: [Gimp-developer] How precise is Gimp 32-bit floating point compared to 16-bit integer?

• From: Daniel Hornung <daniel hornung gmx de>
• To: gimp-developer-list gnome org
• Subject: Re: [Gimp-developer] How precise is Gimp 32-bit floating point compared to 16-bit integer?
• Date: Tue, 17 Dec 2013 02:00:38 +0100

```On Monday, 16. December 2013 11:59:20 Elle Stone wrote:
```
```To restate the question, in decimal notation, 1 divided by 65535 is
0.00001525878906250000. So 16-bit integer precision requires 16 decimal
places (lop off the four trailing zeros) in floating point to express
the floating point equivalent of 1 16-bit integer tonal step, yes? no?
```
```
Not quite:

0/(2^16-1) = 0
1/(2^16-1) ≈ 0.000015259022
2/(2^16-1) ≈ 0.000030518044
3/(2^16-1) ≈ 0.000045777066
...
(2^16-1)/(2^16-1) = 1

So it is quite sufficient to show the first 6 digits after the decimal point,
any error after that is smaller than 2%, and even that only in the worst case
of the first step.

Actually the big advantage of floating point numbers (IMHO) is that they do
not have a linear precision, but each exponentially in-/decreasing times-2-
interval (0, 1/2^8, 1/2^7, ..., 1/2, 1) is split into 2^23 linear steps.

Since our senses work basically logarithmically, this means that we perceive
the steps to be of about equal size -- except that we cannot distiguish such
small steps visually anymore anyways, be it float32 or int16, as pointed out

Cheers,
Daniel

--
Mein öffentlicher Schlüssel / My public key: 4096R/600ACB3B 2012-04-01
Fingerabdruck / Fingerprint:
9902 575B B9A0 C339 CFDF  250B 9267 CA6B 600A CB3B
Runterladen z.B. bei/ Get it e.g. from:
pgp.mit.edu, subkeys.pgp.net, pgp.uni-mainz.de, pool.sks-keyservers.net, ...```

Attachment: signature.asc
Description: This is a digitally signed message part.

• References: