Re: [gtk-vnc-devel] PATCH: Yet more endianness fixes

[Peter Rosin <peda lysator liu se>]

Daniel P. Berrange skrev:
> On Thu, Mar 27, 2008 at 10:53:08PM +0100, Peter Rosin wrote:
> > Daniel P. Berrange skrev:
> > > The changes to gvnc.c then deal with a couple of other issues. First of
> > > all the RealVNC server does not follow the RFB spec for CPIXEL (or the
> > > RFB spec is badly written). Basically when looking to see if a CPIXEL is
> > > 3-bytes, we just examine the VNC pixel format depth field & compare to
> > > 24. This is not what RealVNC does when encoding the data. It look at each
> > > individual colour component to see if they all fit in the most or least
> > > significant 3 bytes. So I change the cpixel decoding to follow this logic.
> > Err, the following is pretty clear to me:
> >
> > "ZRLE makes use of a new type CPIXEL (compressed pixel). This is the 
> > same as a PIXEL for the agreed pixel format, except where 
> > true-colour-flag is non-zero, bits-per-pixel is 32, depth is 24 or less 
> > and all of the bits making up the red, green and blue intensities fit in 
> > either the least significant 3 bytes or the most significant 3 bytes. In 
> > this case a CPIXEL is only 3 bytes long, ..."
>
> The problem I hit is that I ran a server with:
>
>    "vncserver -depth 32 -geometry 1024x1024"
>
> And the pixel format sent by the server is:
>
>   Pixel format BPP: 32,  Depth: 32, Byte order: 4321, True color: 1
>                Mask  red: 255, green: 255, blue: 255
>                Shift red:  24, green:  16, blue:   8
>
> So, note that 'depth is 32' here, even though the shifts/masks clearly
> fit in 3 pixels. And empirically the server sends me ZRLE updates with
> a CPIXEL size of 3, not 4.
>
> So when the spec says 
>
>     "depth is 24 or less"
>
> This is clearly not corresponding to the 'depth' value sent in the pixel 
> format. So basically we need to ignore explicitly declared  pixel format
> depth completely and just look at the pixel masks/shifts to calculate
> the true depth, and thus whether CPIXEL is 3 or 4 bytes.

Ahhh, right, thanks for the clarification. It made me remember this 
comment in my (unrelated) server implementation:

	/* Don't trust depth from the client, they often
	 * ask for the wrong thing...
	 */

And then my code continues on with calculating the true depth from the 
max colour values.

Checking my (unrelated) client, and I do the same thing there. I.e. 
ignore the depth field from the server when true-colour-flag is 
non-zero, instead calculate it from the max colour values.

Cheers,
Peter