Re: pixbuf<->cairo_surface_t conversion

[Benjamin Otte <otte gnome org>]

Havoc Pennington <hp <at> pobox.com> writes:

> 
> Hi,
> 
> I thought of another approach to this problem. Don't expose the pixbuf
> format at all; keep it as-is. However, add:
> 
>  gdk_pixbuf_get_cairo_surface()
>  gdk_pixbuf_new_from_cairo_surface(cairo_surface_t *surface);
> 
> Now, keep the cairo surface internally, strong-ref'd by the pixbuf.
> The pixbuf would have pixels == NULL. If someone does
> gdk_pixbuf_get_pixels(), then lazily create the old pixbuf format and
> keep that copy of the pixels around. Otherwise, never create the old
> format.
> 
gdk-pixbuf is a topic I currently don't spend a lot of time thinking about,
because I can't break API there. But my desire to improve it after GTK 3.0 is
growing. And this approach is pretty much what I'm favoring currently.. I'll
throw in some random thoughts that haven't come up in this thread that are
worthwhile to think about when designing this.

- We convert pixbufs every single time we paint them
This is important for performance considerations: We convert the pixbuf to an
image surface every single time we paint it. So whatever we end up doing, it
won't get any worse. Also, no one has complained yet.

- Not a lot of pixbufs have their pixels accessed
Almost all pixbufs created by applications are not accessed directly. They spend
their life as icons in toolbars and the like. Most developers wouldn't notice
that pixbufs are suddenly represented as cairo surfaces internally. (I think you
cannot modify the pixels of icon surfaces anyway, because they're loaded from
the read-only mmap icon cache).

- Cairo has strict requirements on directly accessing surfaces
You must call cairo_surface_flush() before modifying pixels of a surface and
call cairo_surface_mark_dirty[_area]() after you're done. When you modify a
pixbuf directly, here are no such requiremenets.

- I'm not sure where to store contents
Naïvely, it doesn't make a lot of sense to store images that are only used on
the X server in client memory and upload them every time we need them. But then,
closing a connection to the X server does invalidate the pixels of an Xlib
surface, but not of an image surface. And storing it twice is needless dulicate
memory usage. So no idea where to best store pixel data.

- Cairo surfaces have no size, and can change it
In general, Cairo surfaces have no size. That is why there is no call to
cairo_surface_get_size(). (This might change in the future, but hasn't yet.)
Quite a few surface types allow changing size (X11 Window surfaces and pdf
surfaces are the most well-known of those); a pixbuf doesn't allow that. 

- Cairo surfaces are a lot more flexible than GdkPixbufs
Besides changing size, there's other ways that surfaces can be modified. You can
modify the origin of surfaces with cairo_surface_set_device_offset() for
example. recording surfaces sizes are in doubles, not ints. Oh, and
cairo_show_page() can do funny thngs to surfaces. Things like this make handling
a Cairo surface a lot more complicated and a simple API like GdkPixbuf way nicer.

- We lose precision if we use Cairo formats
Premultiplied formats have less precision then unpremultiplied ones; the
conversion from unpremultiplied to premultiplied is not lossless. (0x40FFFFFF
and 0x40FEFEFE are both stored as 0x40404040 unpremultiplied.) So loading, then
saving a PNG might result in a different image being stored. Not sure that is a
good idea.

- I want to expose more formats in Cairo (and pixman)
I still think that the choice to only support 4 image formats was a bad idea. In
particular because it's formats that almost no one else supports. (I feel Mitch
breathing down my neck again already...) Not supporting someone's (libjpeg,
libpng, ffmpeg, HTML canvas, you name it) format usually results in someone
writing an ad-hoc, informally-specified, bug-ridden, slow conversion routine.
And I'd rather have these routines working properly in pixman and allow
fast-pathing them in the backends - GL can certainly speed things up
considerably here. If we added more formats to pixman first, we'd also get
around the precision loss mentioned above.


So with all that said, I think I see GdkPixbuf as a light version of
cairo_surface_t: No device offsets, well known size, mostly read-only usage and
no crazy surface types. And really fast, too. Painting thousands of pixbufs per
frame should not take any noticable CPU.

Benjamin