Re: next steps for touch support in GTK+

[Carlos Garnacho <carlosg gnome org>]

Hey :),

On jue, 2012-08-02 at 11:42 +0200, Simon Schampijer wrote:
> Hi,
> 
> there have been several discussions around GTK+ touch support during 
> GUADEC. To get a good touch experience we talked about the following 
> points. I ordered them by priority that we set in Sugar development for 
> them. If you come across something that we missed, please comment.
> 
> === 1. Text selection ===
> A lot of work has been going into text selection and manipulation those 
> days from the design and implementation point of view. The current 
> design ideas can be seen at [1].
> 
> Carlos created universally usable handles (implemented using GtkWindows) 

Just a minor annotation, GDK_WINDOW_TEMP GdkWindows

> to adjust the selection and wrote the code to use them in GtkEntry and 
> the GtkTreeView in this branch [2]. The idea is to make those available 
> as well for external widgets, e.g. in Abiword or WebKitGTK.

I'll update that branch in the following days with opposite-sided
handles and other minor improvements.

> 
> We discussed as well how the Edit Palette should look like and agreed 
> loosely on using GtkWindows instead of an Overlay for this to handle 
> cases like resizing windows for example.
> 
> Something to check is if we want to allow to adjust the selection in 
> moving the right handle over the left one. Both iOS and Android do block 
> that and snap to the minimum selection when releasing the finger which 
> seems a good interaction model and limits complexity and edge cases.

Given the loose interaction between the GtkTextHandle helper object and
the text widgets, that'd be per-widget implementation, hard to avoid
though.

> 
> Another thing to make sure is that we do not use fixed positions in 
> regards to Wayland.
> 
> 
> === 2. OSK - Slide canvas vertically to keep insertion point visible ===
> Carlos has been working on this already. He will send a mail with more 
> info about it. To get more background about the issue itself a 
> whitepaper is available at [3].

I'll better advance the rationale here. The target of this work was to
extend GtkIMContext to let onscreen keyboards announce the extents the
keyboard is going to take, so for example (mostly?) GtkScrolledWindows
can move the focus to a visible location if necessary.

Likely the root question here is whether this should be entirely a
compositor task or a toolkit-level task, if this is handled by the
compositor (eg. moving the window altogether) there's probably some
important context lost if the window scrolls upwards (toolbars, etc). If
this is fully handled by the toolkit there are other corner cases as
well, such as scrolled windows falling entirely within the keyboard
extents. So probably a mixed approach would be best, although it's a bit
unclear when applies what and how to best coordinate those.

I've pushed wip/im-osk-position with the initial work about this, it's
notably missing a GdkDisplay/Screen argument in the signaling, and this
concept altogether is likely not quite friendly with wayland...

> 
> 
> === 3. SpinButton ===
> We had several discussions around the SpinButton. One thing that came up 
> is why we do not want to reuse the existing one. One option is to put 
> the '<' on the left side (this could be done by a property for example). 
> When the SpinButton was redesigned they put the < and > close together 
> to not interfere with other widgets at the left side. Having something 
> like "[button] < | 3| >" might not be as clear. That at last was the 
> reasoning for putting the stepper buttons next to each other. In touch 
> the interaction gets a bit fiddly having the buttons next to each other 
> that is why we propose that configuration option.
> 
> We would keep the entry widget, on touch you can use the OSK to set a 
> value, or you can set it to be non-editable. Furthermore you can accept 
> a vertical swipe to alter the value, implementing this would be a 
> perfect use case for EventControllers.

It also could be a good use for captured events, so touch events only
focus the entry on tap, and change-value-as-you-swipe works after a
threshold.

> 
> Another option is introducing a complete new widget targeted at touch 
> usage (similar to the one in iOS Garageband) [4] which Carlos 
> implemented already [5]. The issue is here the height, which at least in 
> a Sugar toolbar could be not as ideal and introducing a new widget.

Another open question there is about the widget implementing
GtkOrientable, so "buttons" (and the swipe gesture) can be aligned on
both axes. The big plus of this widget though is the better behavior on
touch devices, so it could maybe stand as less useful if GtkSpinButton
gains a similar behavior.

Cheers,
  Carlos

> 
> 
> === 4. Which GTK+ widgets break with touch ===
> The SpinButton item from above is one example of those. Another one was 
> the Combobox which has been fixed by Carlos and landed in master [6]. I 
> will have a look if I come across more. Feel free to add if you are 
> aware of more blockers.
> 
> 
> === 5. Widget Gestures (e.g. swipe, pinch) ===
> The desired implementation is to use EventControllers for that. The work 
> has currently stagnated on that end. In Sugar we could implement this at 
> our toolkit level but an upstream solution would be desired at least 
> longer term.
> 
> 
> === 6. OSK widget context provider (e.g. search vs open vs go...) ===
> Matthias said there was a patch floating around for that. I looked in 
> the bugs with patches attached in bugzilla but could not find it. If 
> someone knows where it is would be great.
> 
> Regards,
>     Simon
> 
> [1] https://live.gnome.org/GnomeOS/Design/Whiteboards/Selections
> [2] http://git.gnome.org/browse/gtk+/log/?h=touch-text-selection
> [3] 
> https://wiki.maliit.org/images/a/a7/Technical-overview-widget-relocation.pdf 
> (from https://wiki.maliit.org/Documentation#Technical_documentation)
> [4] http://wiki.sugarlabs.org/go/File:Ios_spinbutton.JPG
> [5] https://bugzilla.gnome.org/show_bug.cgi?id=678108
> [6] https://bugzilla.gnome.org/show_bug.cgi?id=678113
> _______________________________________________
> gtk-devel-list mailing list
> gtk-devel-list gnome org
> https://mail.gnome.org/mailman/listinfo/gtk-devel-list