Re: GTK + Clutter next step(s)

[Robert Bragg <robert sixbynine org>]

Hi Benjamin,

I've added a few thoughts inline...

On Mon, Oct 10, 2011 at 9:19 AM, Benjamin Otte <otte gnome org> wrote:
> After Emmanuele's mail[1] I've been wondering how to get there. In
> particular, I've been wondering how to get there incrementally, so
> that when we finally release a GTK 4.0, we can support a transition
> that is as smooth for applications as the transition to GTK 3.0.
>
> I think that the best first step in this task would be to integrate
> Cogl into GDK. Here's the problems I'd like to solve with this
> - Improve integration between Clutter and GTK (read: ideally make
> clutter-gtk unnecessary)
> - Provide the requirements to implement frame-based painting in GDK/GTK.
> - Allow GTK applications to use GL without hacks (read: no more
> GTKGLExt, less WebGL pain in webkit-gtk)
> - Have an actual point of data for the Cogl 2.0 API design. [2]

At the moment, "cogl 2.0" is a relatively loose set of ideas that
probably only Neil Roberts and myself have a rough handle on. That
page hasn't been updated in a long time so although it gives a sense
of the direction its going it is not up to date and some of those
ideas have changed a bit.

Definitely I'd really like for more people to be familiar with the
cogl 2.0 direction and I hope that as more people get interested in
cogl then through asking questions and having discussions more people
will gain insight. I think my preference at this point would be for
cogl 2.0 to evolve on the cogl mailinglist[1] more so than a wiki page
mainly because I can see the page just getting out of date again
but also I think a mailing list may help us archive more detailed
reasoning along side design ideas - I think with wiki designs its
harder to record a rationale.

Currently the experimental 2.0 reference manual[2] is probably the
best reflection of where the 2.0 api is headed because we have to
update it in line with code changes.

To give a quick high level snapshot; the main things we've wanted
before we reach cogl 2.0 are:
- Removing the default context in favour of having all functions
  either be explicitly passed a pointer to a context or be implicitly
  related to a context via an object that itself links to a context.
- We want to get rid of any remaining use of CoglHandle.
- We want to further emphasize Cogl's interface oriented programming
  model.
- We need to cleanup the CoglTexture api by making all the low-level
  (2D,3D, Rectangle) backends public + expose a high level MetaTexture
  interface for sliced/atlas/sub textures.
- We want a shader snippets API for hooking small snippets of glsl
  code into well defined points of a CoglPipeline.
- We need to be able to track shader uniforms as part of CoglPipeline
  not as part of CoglProgram.

> - Make it possible for Clutter to get rid of its winsys layer and just
> use a GdkWindow.
> - Prove that people shouting "OMG, GL will make GTK slow" are stupid.
> If we don't manage to solve all of this problems - or if we figure out
> we can't solve all of them - I don't mind. I just want to know where
> we want to go.
>
> So how do I want to go about this? Here's a rough list:
> - Make GTK build (optionally?) depend on Cogl
>  - Should we try with Cogl 1.0 or go straight to Cogl 2.0?

I think the only option here really is cogl-2.0 - cogl-1.0 isn't
usable standalone, it just refers to the limited bits of Cogl that can
be used to write Clutter actors.

> - Make sure all native GdkWindows are valid framebuffers for Cogl
>  - Does that work today? What do we need to be careful about?

One thing worth noting here is that we don't yet have proper OSX
support in Cogl. Currently the OSX winsys support is still in Clutter
and we use a dummy/nop winsys in Cogl. The plan is to add proper
support to Cogl it just hasn't happened yet.

>  - How do other backends (in particular Broadway) do this?
> - Introduce a gdk-cogl.pc for new APIs
>  - What stability do we support here?
>  - I'd say "no stability, use at own risk", but is that feasible?

Yes, please keep the interfaces unstable. Cogl 2.0 still has quite
some way to go yet so we don't want to be prematurely constraining the
API. No doubt having gdk build on Cogl is going to provide valuable
input to evolve the api to cover gaps we may currently have.

Personally I really value our current freedom to evolve the API
incrementally according to what our usage requirements are. We are
only just starting to use Cogl outside of Clutter - with cairo-cogl
and a backend for Evas being our first experiments which have already
been able to provide valuable feedback for the API.

Clutter internally explicitly builds on the unstable Cogl API and I
think its best to avoid strong interface barriers between libraries if
they aren't strictly needed since they can often simply stifle
development. If we are going to move a lot of Clutter's winsys code
into gdk I think it would be a shame if we loose the freedoms we
currently get from having a slushy Cogl interface.

>  - What about major api changes to cogl?
I think the way we are managing 2.0 changes has been working pretty
well so far, so hopefully this wont be a problem. We are guarding 2.0
API changes with COGL_ENABLE_EXPERIMENTAL_2_0_API guards and for
incompatible APIs we use some cpp defines to give the 2.0 symbols a
suffix so they don't clash. It's perhaps not ideal that we have to
play these tricks but for the sake of not breaking the 1.x abi and
allowing us to work incrementally towards 2.0 this approach has been
working ok so far.

If you were more concerned about churn and breakages with the 2.x API
that you might have to deal with then I think here we just need to
make sure we communicate well. We would aim to routinely test gdk
ourselves to help avoid breakages, and help to keep gdk up to date
with api changes. Gdk developers would need to follow the mailinglist
to keep an eye on how Cogl is evolving too and make sure the design
meets your requirements.

>  - How likely are distros to ship this?
> - Add API to allow users of GdkWIndow to use Cogl
>  - What do we (read: clutter-gtk) need here?
>  - Can we have something like make_current() to get native GL drawing?

Exposing the GL api directly gets tricky. We can't simply allow
applications to use the same GL context that Cogl creates because the
application will clobber Cogl's own state. The plan currently is that
we will add API to allow an application to explicitly create their own
GL context using the Cogl API and bind the context to a
CoglFramebuffer. The aim here is to add enough API to allow apps to
create their own GL context which completely sandboxes them away from
Cogl's own state tracking but still allows them to use the
CoglFramebuffer API so apps don't have to use any window system
specific API.

Another option being considered is a tightly integrated cogl-gles2
sub-library that would actually provide a full GLES2 API but through
close integration with Cogl, developers would be able to efficiently
mix Cogl and GL drawing. This approach would also reduce the
complexity of dealing with different GL versions and extensions since
we would control all of that directly.

> - Add API to allow creating GdkWIndows for Cogl offscreens
>  - What do we (read: clutter-gtk) need here? Is GdkOffscreenWindow enough?
> - Port selected applications from clutter-gtk/GtkGLExt
>  - Which ones?
>  - Can we get the app authors to do this?
> - Get rid of clutter-gtk
>  - Is anything missing at this point?
> - Switch GTK to frame-based redrawing
>  - Does that require API "changes" to GDK?

I think this needs careful consideration. There are definitely lessons
to be learned from clutter's master clock, and there are also a number
of things that could be done better. I had started to implement a
paint-clock concept in Cogl recently that aimed to tackle some of the
issues with the master clock. I didn't finish it but it might be worth
looking at with relation to this. (I was originally doing it because I
thought it would be required for OSX but in the end we found it wasn't
the right approach.)

> - Add functionality to GtkWidget for GL-based widgets
>  - Can we use this to do more invasive changes?
>  - Can we even change GTK's event model with this?
>  - Can we get a simple API to do native GL drawing here?
>  - If we can, should we?

I'd be very cautious about exposing GL. Some of the reasons we ended
up developing Cogl is that GL is a very difficult API to deal with in
orthogonal components (such as separate widgets) because you have to
share a monolithic state machine and you have no way of knowing what
state you might need to save and restore to work with other
components.

Other things to consider are the number of incompatible "GL" varients
people might want to use: fixed-function GL 1.x, programmable GL
(arbfp/glsl) GL 3.2 core api with deprecated apis removed, GL 4.x,
GLES 1.1, GLES 2.0 and then for good measure throw in all kinds of
extensions which developers will invariably need to depend on.

Personally I think it would quickly become unmanageable to allow
widget authors to use use OpenGL directly except for a few special
cases.

As I mentioned earlier we have been considering offering a way to mix
raw GL usage with Cogl (by allocating a separate context) but I think
I would rather recommend that only be used for existing GL based games
or e.g. for something like WebGL. If you want to use the GPU for
actually rendering widgets its not going to work well at all if you
end up with a separate GL context for each widget so I would
definitely recommend using Cogl instead of GL for any widget where you
expect more than just one in an application.

> - Provide a way to switch Cairo drawing to GL
>  - We first need a cairo-gl/cairo-cogl to do this I guess?
I think cairo-gl would be quite difficult to integrate in a way that
you can share the GL context if you want to use the Cogl API for
drawing to widgets too.

The problem with cairo-cogl though is that currently I'm only
targeting mobile devices with a high dpi display and so I am
specifically not aiming to match the antialiasing quality of the image
backend (which cairo-gl does). I would need to do some extra work to
support the same level of antialiasing but that should be doable.

>  - I want this to improve cairo-gl, are there other uses?
>  - Any requirements from the Pango side?
>
> Note that all of these are incremental steps and they are possible to
> do at any point in time, ie none of those steps must happen in the
> same cycle as a previous step. Also note that we can at any point roll
> back a step, rethink it, and redo it, as we don't export any new API
> (apart from the experimental one that we don't give lots of guarantees
> on). I think this is a rather important thing, because there is
> probably nobody with enough of a clue on how the end result should
> look. Certainly not me.
>
> Comments?
>
> Benjamin
>
>
> 1: http://mail.gnome.org/archives/gtk-devel-list/2011-August/msg00058.html
> 2: http://wiki.clutter-project.org/wiki/CoglDesign
> _______________________________________________
> clutter-devel-list mailing list
> clutter-devel-list clutter-project org
> http://lists.clutter-project.org/listinfo/clutter-devel-list
>

ok, thanks for your ideas Benjamin, they seem pretty reasonable to me
and I hope I can help on the Cogl front at least.

kind regards,
- Robert

[1] https://groups.google.com/forum/?hl=en#!forum/cogl3d
[2] http://docs.clutter-project.org/docs/cogl-2.0-experimental/unstable/