Re: [gtk-vnc-devel] PATCH: Yet more endianness fixes

["Daniel P. Berrange" <berrange redhat com>]

On Thu, Mar 27, 2008 at 10:53:08PM +0100, Peter Rosin wrote:
> Daniel P. Berrange skrev:
> >The changes to gvnc.c then deal with a couple of other issues. First of
> >all the RealVNC server does not follow the RFB spec for CPIXEL (or the
> >RFB spec is badly written). Basically when looking to see if a CPIXEL is
> >3-bytes, we just examine the VNC pixel format depth field & compare to
> >24. This is not what RealVNC does when encoding the data. It look at each
> >individual colour component to see if they all fit in the most or least
> >significant 3 bytes. So I change the cpixel decoding to follow this logic.
> 
> Err, the following is pretty clear to me:
> 
> "ZRLE makes use of a new type CPIXEL (compressed pixel). This is the 
> same as a PIXEL for the agreed pixel format, except where 
> true-colour-flag is non-zero, bits-per-pixel is 32, depth is 24 or less 
> and all of the bits making up the red, green and blue intensities fit in 
> either the least significant 3 bytes or the most significant 3 bytes. In 
> this case a CPIXEL is only 3 bytes long, ..."

The problem I hit is that I ran a server with:

   "vncserver -depth 32 -geometry 1024x1024"

And the pixel format sent by the server is:

  Pixel format BPP: 32,  Depth: 32, Byte order: 4321, True color: 1
               Mask  red: 255, green: 255, blue: 255
               Shift red:  24, green:  16, blue:   8

So, note that 'depth is 32' here, even though the shifts/masks clearly
fit in 3 pixels. And empirically the server sends me ZRLE updates with
a CPIXEL size of 3, not 4.

So when the spec says 

    "depth is 24 or less"

This is clearly not corresponding to the 'depth' value sent in the pixel 
format. So basically we need to ignore explicitly declared  pixel format
depth completely and just look at the pixel masks/shifts to calculate
the true depth, and thus whether CPIXEL is 3 or 4 bytes.

Regards,
Dan.
-- 
|: Red Hat, Engineering, Boston   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|