Re: [gtk-vnc-devel] Fix pixel shift for BGR format servers

["Daniel P. Berrange" <berrange redhat com>]

On Tue, Jul 10, 2007 at 04:31:31PM -0500, Anthony Liguori wrote:
> Daniel P. Berrange wrote:
> >The code which currently calculates the pixel shift for converting data
> >from remote to local encoding is assuming the server is in RGB format
> >pixels. If I run a server in BGR format, then the colours are rather
> >messed up. I've attached a patch which tries to correct it - although its
> >not perfect because now, the code assume either RGB, or BGR - is it 
> >possible
> >for a server to use GRB, or GBR ?!?!  If not then this may be sufficient.
> >
> >I'm also not entirely convinced this is correct wrt to local server
> >format, eg if your local desktop ran in BGR instead of RGB. I don't have
> >any system I can make run in BGR locally though, so can't test.
> >
> >BTW, for testing I was running a server with
> >
> >vncserver -geometry 1024x800 -depth 16 -pixelformat bgr565
> >
> >And another with
> >
> >vncserver -geometry 1024x800 -depth 16 -pixelformat rgb565
> >
> >With this patch, both now work.
> >  
> 
> VNC servers and clients are notoriously bad at respecting 
> SetPixelFormat().  While I don't think we properly support the 
> big_endian_flag, I do think we're doing right thing with the 
> mask/shifts.  Are you sure that RealVNC isn't screwing up here?  Your 
> patch looks a big magical to me.

No, the VNC server looks fine to me. What's going on is that for a RGB
server,the remote shifts are:

  R: 11
  G: 5
  B: 0

And the local shifts are

  R: 16
  G: 8
  B: 0

So, the resulting delta shift (the rp, gp & bp fields in vnc_framebuffer)
calculated are R: 19, G: 10, B: 3

In a BGR server, the remote shifts are:

  R: 0
  G: 5
  B: 11

And this ends up with us calculating a delta shift of R: 8, G: 21, B: 14

The problem, is that the delta shift should have been the same in both
cases - because the data we're shifting has already have the remote
shift removed. This patch basically makes sure that the delta shift
is the same in both cases.

Dan
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|