Client server handshake and missing color maps

[Marco Gidde <Marco Gidde br-automation com>]

Hello Daniel, hello list,

I noticed and analyzed a gtk-vnc crash, via vinagre on ubuntu 13.10 and
gvncviewer.py on an older SUSE release. The client tried to connect to
an older proprietary VNCserver running on a PLC, RFB version 3.3, maybe
some RealVNC derivative.

It turned out that the server uses 8bit color depth with a color map and
does not accept clients which request different color encodings or true
color. On the other hand the server does not send the color entries
unless the client explicitly sends the SetPixelFormat message.

There is some code/comment in src/vncdisplay.c:1182:

#if 0
        /* TigerVNC always sends back the encoding even if
           unchanged from what the server suggested. This
           does not appear to matter, so lets save the bytes */
        memcpy(&fmt, currentFormat, sizeof(fmt));
        break;
#else
        return TRUE;
#endif

If I activate that code, the client connects successfully, so there
might be a reason why TigerVNC always sends it :-)

It's not entirely clear whether the client is required to always send
SetPixelFormat - when using true color it is obviously superfluous. But
in this case its main purpose is to instruct the server to send the
color map. I couldn't find anything about this in the specification, but
I only skimmed through it.

For further information please do not hesitate to contact me.

Greetings, Marco

-- 
Marco Gidde
Bernecker + Rainer Industrie-Elektronik Ges.m.b.H.