# Re: [Gimp-developer] How precise is Gimp 32-bit floating point compared to 16-bit integer?

• From: Simon Budig <simon budig de>
• To: gimp-developer-list gnome org
• Subject: Re: [Gimp-developer] How precise is Gimp 32-bit floating point compared to 16-bit integer?
• Date: Mon, 16 Dec 2013 18:30:06 +0100

```Elle Stone (ellestone ninedegreesbelow com) wrote:
```
```My question is, for Gimp from git, is 32-bit floating point more
precise than 16-bit integer?
```
```
Yes, at least for the range from 0.0 to 1.0.

```
```If so, by how much, and does it depend
on the machine and/or things like the math instructions of the
processor (whatever that means)? And if not, how much less precise
is it?
```
```
AFAIK it does not depend on the processor, floating point numbers are
defined in IEEE 754, and to my knowledge that is what all processors
use.

For 32 bit floats there are 23 bits in the mantissa, so in the range
from 0.0 to 1.0 we easily have more precision than with 16 bit ints.

```
```To restate the question, in decimal notation, 1 divided by 65535 is
0.00001525878906250000. So 16-bit integer precision requires 16
decimal places (lop off the four trailing zeros)
```
```
you're barking up a wrong tree here. The length of the decimal expansion
is not necesssarily helpful, because most of them represent rounding errors.

(btw. - you divided by 65536)

1.0 / 65535 = 0.000015259021896696422

but

1.0 / 0.00001525913 = 65534.53... --> gets rounded to 65535

and

1.0 / 0.00001525891 = 65535.48... --> gets rounded to 65535

So with 11 decimal digits we easily have all the precision we need to
represent the fractions for a 16bit int.

```
```How many decimal places does Gimp 32-bit floating point actually provide?
```
```
It has 23 bits mantissa, 8 bit exponent and 1 bit sign.

For decimal notation it depends a lot on the range. Numbers with a
bigger magnitude (exponent > 0) the number of digits after the decimal
points become less.

BTW: You can view 16 bit ints "somewhat like a float with no sign bit,
no exponent bits and 16 bits mantissa". I.e. sign is always positive,
exponent is always 0. That makes it clear that a 32bit float completely
encompasses the 16 bit integer values

Bye,
Simon

--
simon budig de              http://simon.budig.de/
```

• Follow-Ups:
• References: