Fractional scaling 2017 hackfest summary

[Jonas Ådahl <jadahl gmail com>]

Hi everyone!

Last week a GNOME Shell hackfest was held at the Canonical office in
Taipei, Taiwan. The topic was fractional scaling, and this E-mail aims
to summarize what was done, and future tasks. The hackfest was attended
by me, Marco Trevisan and Shih-Yuan Lee.

Prior to the hackfest, mutter had already been adapted to support very
rudimentary fractional scaling; mostly involving changing 'int' to
'float', hard coding the scales 1, 1.5 and 2, and various minor
adaptations.

During the hackfest, we made progress in many places, and I'll try to
split them up into topics, and summarize the current state and future
problems.

1) Overall approach

In GNOME 3.24, HiDPI scaling of window content is managed simply
resizing the window actor (for Wayland clients) or completely client
side (for X11 clients).

Currently on master, hidden behind a gsetting, this has changed to
decoupling the stage coordinate space from physical pixels. In other
words, depending on the scale a monitor is configured to have, the pixel
density of that area of the stage may be different from other areas.
This is done by transforming the viewport given a scale when drawing
each monitor, causing clutter to draw things bigger. We introduce two
types of coordinate spaces; logical pixel coordinate space - the
coordinate space of the stage, and physical pixel coordinate space - the
coordinate space of each framebuffer.

Windows, actors, cursor position etc are all in logical pixel coordinate
space. The framebuffer coordinate space depends on what framebuffer is
referenced. It can be the framebuffer flipped to the monitor; it can be
a framebuffer of a client surface buffer.

For X11 clients, this means that every client will be scaled according
to what monitor it is drawn on; but it also means that without further
changes, X11 style client side scaling is no longer possible. Thus,
until a solution to this is developed, gnome-settings-daemon will
configure X11 clients that are aware of the 'Gdk/WindowScalingFactor'
XSetting to never draw with anything other than scale 1.

In order to support fractional scaling, to make this work there are
various things to consider. For example, we wanted monitors in the stage
to be positioned on the integer grid of the logical coordinate space.
This means that we can't just use any arbitrary fractional scale for any
monitor mode, as when scaling using such a scale would result in a
non-integer-sized rectangle representing the monitor area on the stage.
To solve this, we define what scales we want to support, and find
scaling factors that result in integer monitor rectangles. For example,
we might want to support ~1.375 scaling on a 1920x1080 resolution, and
for that we find the scaling factor ~1.37931, resulting in a 1392x783
rectangle in logical pixel coordinate space.

2) Configuration API

For 3.26, Mutter will expose a different configuration API than before.
This work has been going on for quite a while, and is now fairly ready
for actual usage. In short, it moves all complex low level monitor
hardware management to mutter, exposing a higher level API. This new API
allows the API user to place monitors in logical monitors (making them
clones) and logical monitors in a global screen space.

During the hackfest we updated the API to pass a list of supported
scales for every supported mode of a monitor. Mutter was updated to
expose the calculated modes (see part 1), gnome-settings-daemon was
updated to deal with the API change, and gnome-control-center was
updated to be able to deal with supported scales being different for
each mode of a monitor.

As a proof of concept, and for allowing to test many different scales,
in the current branch, we list 8 scales per int, ranging from 0.5 to
4.0. This will of course be tweaked in some way or another.

3) Mutter related work

Mutter was updated here and there to support fractional scaling better
as well as improvements to the new way of dealing with different monitor
scales. For example, taking screenshots were fixed, sub-logical-pixel
cursor positioning was added, the new configuration system (D-Bus API
implementation, configuration file reading/writing) was updated to deal
with fractional scaling, and the accompanying precision loss issues.

4) GNOME Shell UI related work

Some time was spent looking into how to make GNOME Shell look good. The
issue is mainly that clutter thinks it's drawing in scale 1, but in fact
everything will be scaled according to the monitor scale when being
painted. The result is that low resolution images are scaled up. A proof
of concept of how to deal with this had been implemented some weeks
before, by exposing a new concept called "resource scale" to clutter
actors. The resource scale refers to how much any resource should be
scaled with to look good. For example, StIcon is changed to fetch larger
icons given the resource scale, and there were promising (yet to be
finalized) results of how to deal with all of the StThemeNode related
drawing without having to do any Javascript changes.

5) Calculation of a default scale

We spent some time discussing (internally and also with System 76 via
IRC) and testing out adding automatic calculation of default scales
including fractional ones. We had various hardware to test on, and have
a proof of concept algorithm. This is also a topic that of course will
be tweaked, given input received from tests. Canonical Taiwan provided
us with various hardware to test on, so we also started a list of
hardware and "nice" default scales, to get data on what magical values
to use some algorithm.

The test result table can be found here
<https://wiki.gnome.org/Projects/GnomeShell/ScaleFactorTests>. Feel free
to add more data into the table.

We also discussed whether it would be beneficial to allow vendors to
override any automatically calculated default scale using something
similar to how libinput deals with hardware differences via hwdb. We
wouldn't be able to use hwdb directly I'm assuming, as these things
would use data from EDID for identifying hardware.

6) Client side support

Previously, an idea has been that for scales like 1.5, tell clients to
scale by 3, then downscale by 2, resulting in a target framebuffer scale
of 1.5, but the benefits of this has not seemed to be worth it, thus we
ended up for monitor scales between 1 and 2, tell clients to scale using
scale 2 and then downscale accordingly. In practice we'll most often
never scale above 2, so we did not prioritize fixing client side bugs
related to drawing using scale > 2. There were various issues, such as
problems with drawing shadow, warnings and possibly crashes, that needs
to be dealt with before we can allow > 2 scales with good conscience.

7) Wayland protocol improvements

We touched on this, and concluded that the most important use case where
we cannot let clients draw larger, having the compositor later downscale
the client buffer, is for performance critical applications such as
games. For this, a future Wayland fullscreen protocol needs to take this
into account, by for example specifying the expected framebuffer
resolution, instead of specifying an expected window size in logical
pixel coordinate space.

Whether to add some way to let toolkits who are able to deal with
fractional scaling client side avoid any compositor side scaling is an
open question, and it's not clear whether the benefits are worth the
needed complications.

8) XWayland support

X11 clients, as mentioned previously, are now always considered scale 1
and are scaled according to the monitor scale. There have been various
discussions on this previously in some places, but we didn't spend much
time on this. One possible approach discussed is to add a new X11
specification allowing clients to do things similar to how Wayland
surface buffer scales are handled, allowing the Wayland compositor to
scale the window, while at the same time know how to transform child
windows.

For now XWayland completely ignores any wl_output::scale set, and
directly take the current mode and geometry of a wl_output and use this
to describe the X11 screen and monitor dimensions. For X11 clients to
be able to maximize, fullscreen etc, they need to have the correct idea
of the geometry of the monitors and screen in the logical pixel
coordinate space, so mutter will lie about the current mode and say it
is size the scaled logical monitor layout.

To support things such as X11 clients (games) setting a temporary
resolution, we probably need a custom XWayland protocol for
communicating the monitor and screen dimensions, so we can pass "real"
data about monitors.

Further reading on HiDPI XWayland support:
https://bugs.freedesktop.org/show_bug.cgi?id=93315



This more or less summarizes the high level things that were discussed
and worked on during the hackfest. There is a more detailed log
available here: <https://hackmd.io/KwQwxiAsCmCMwFoCcJiwTAHAEwSESiAbAOwDMYJcAZtWWdkA>
On that page, you can also find various git branches with actual things
to inspect and test, as well as a more detailed list of issues and
things to do.


Finally, many thanks to the Canonical Taipei office and staff for
hosting us last week!


Jonas