Re: [gtk-vnc-devel] PATCH: Yet more endianness fixes

[Peter Rosin <peda lysator liu se>]

Anthony Liguori skrev:
> Daniel P. Berrange wrote:
> > The problem I hit is that I ran a server with:
> >
> >    "vncserver -depth 32 -geometry 1024x1024"
> >
> > And the pixel format sent by the server is:
> >
> >   Pixel format BPP: 32,  Depth: 32, Byte order: 4321, True color: 1
> >                Mask  red: 255, green: 255, blue: 255
> >                Shift red:  24, green:  16, blue:   8
> >
> > So, note that 'depth is 32' here, even though the shifts/masks clearly
> > fit in 3 pixels. And empirically the server sends me ZRLE updates with
> > a CPIXEL size of 3, not 4.
> >
> > So when the spec says 
> >
> >     "depth is 24 or less"
> >
> > This is clearly not corresponding to the 'depth' value sent in the pixel 
> > format. So basically we need to ignore explicitly declared  pixel format
> > depth completely and just look at the pixel masks/shifts to calculate
> > the true depth, and thus whether CPIXEL is 3 or 4 bytes.
> >   
>
> Have you informed the realvnc folks of this?  Either the spec or realvnc 
> should be fixed.  I don't like the idea of violating the spec to fix a 
> bug with one server since we're then breaking another server that isn't 
> violating the spec.
>
> Let's try to get some official resolution of what the right thing to do 
> is before we change our CPIXEL handling.

Since CPIXEL is specified in terms of if the high or low byte is unused, 
depth by itself is clearly insufficient. IOW, you have to visit the 
masks/shifts anyway for the CPIXEL implementation. Seeing depth<=24 is
not valid as a 3 byte CPIXEL test, since it could be byte 2 that is 
unused (or other distributions of 8 bits, of course).

Cheers,
Peter