Re: Does GTK+ do automated/nightly performance regression testing?

[Emmanuele Bassi <ebassi gmail com>]

On 5 February 2018 at 09:35, Clemens Eisserer <linuxhippy gmail com> wrote:
> Hi there,
>
> I wonder - is there any automatic / nightly performance regression
> testing done against GTK+ development snapshots, like it is done by
> other performance-sensitive open-source projects (e.g. mozilla
> firefox)?

No.

Ideally, we'd be using Continuous for this — but the VM generation
broke last year, and while I'm still working to replace it with a new
build system, it's not finished. Performance regression testing is
also not entirely reliable when running on a VM.

Mozilla has a fairly big build and test farm, and a huge test suite,
something GNOME has no access to. This allows Mozilla to test changes
on real hardware, most of the time, and thus have a better picture of
what kind of performance trade-offs should be considered.

> The reason I ask is, from a subjective point of view, several times
> during GTK+'s development I had the impression of performance
> regressions introduced by new feature work:
> * During the GTK2-8 days, when GTK was migrated to use cairo instead
> of GDK's abstraction for drawing widgets

There was no "GDK abstraction": you either rendered using Xlib, or you
rendered using a reimplementation of Xlib primitives because you were
running GTK on Windows.

The introduction of Cairo added a new rendering abstraction API, but
the porting from GDK primitives to Cairo took definitely more time
than just the 2.8 release.

> * When GTK-3 was released: Eclipse's SWT is almost twice as slow when
> running on top of GTK3 compared to GTK2:
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=493714

Eclipse does really weird things to maintain compatibility with
plugins written for older releases and different platforms; those
plugins are also often out of tree and non-free, so they cannot be
easily ported.

Such is the life when writing an abstraction over a toolkit.

> Now GTK4 is almost ready to be released, and watching the
> presentations about the new drawing model, all the CSS properties etc.
> I am worried again of performance regressions for real world code.

"Real world code", as of this moment, does not exist.

Porting custom widgets to GTK 4.0 will require some work, depending on
what the custom widget does; for custom rendering, using Cairo should
not come at any additional cost, whereas rendering using the
GtkSnapshot API and the CSS drawing model (boxes, borders, shadows,
text) should come with a significant performance improvement compared
to the equivalent Cairo-based rendering in GTK 3.x.

> And if there is nothing automated, would there be interest in such a
> project -  Would it be useful and used by the developers doing feature
> work?

Of course, if you want to work on performance testing you're
absolutely free to do so; we can discuss methodology, but comparison
with GTK 2 and GTK 3 is kind of moot: the rendering pipeline is wildly
different, and the cost is allocated in different places.

I'm much more interested in incremental performance regression testing
over the same code base; if we plug it into the CI pipeline, we'd at
least be able to spot eventual regressions during the development
phase.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]