[mutter] clutter/pick-stack: Use exclusive bottom/right box borders when picking

[Marge Bot <marge-bot src gnome org>]

commit a7c4e8cefa8eb0b9f85471975d49be4c4d0d20de
Author: Sebastian Keller <skeller gnome org>
Date:   Tue Apr 27 18:10:51 2021 +0200

    clutter/pick-stack: Use exclusive bottom/right box borders when picking
    
    The graphene functions used by clutter for picking assume that boxes are
    inclusive in both there start and end coordinates, so picking at y
    coordinate 32 for an actor with the height 32 placed at y coordinate 0
    would still be considered a hit. This however is wrong as 32 is the
    first position that is not in the actor anymore.
    
    Usually this would not be much of a problem, because motion events are
    rarely ever at exactly these borders and even if they are there will be
    another motion event soon after. But since actors in gnome-shell usually
    are aligned with the pixel grid and on X11 enter/leave events are
    generated by the X server at integer coordinates, this case is much
    more likely for those.
    
    This can cause issues with Firefox which when using client side
    decorations, still requests MWM_DECOR_BORDER via _MOTIF_WM_HINTS to have
    mutter draw a border + shadow. This means that the Firefox window even
    when using CSD is still reparented. For such windows we receive among
    others XI_RawMotion and XI_Enter events, but no XI_Motion events. And
    the raw motion events are discarded after an enter event, because that
    sets has_pointer_focus to TRUE in MetaSeatX11. So when moving the cursor
    from the panel to a maximized Firefox window the last event clutter
    receives is the enter event at exactly integer coordinates. Since the
    panel is 32px tall and the generated enter event is at y position 32,
    the picking code will pick a panel actor and the focus will remain on it
    as long as the cursor does not leave the Firefox window.
    
    Fix this by excluding the bottom and right border of a box when picking.
    
    Fixes https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/4041
    
    Part-of: <https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1842>

 clutter/clutter/clutter-pick-stack.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
---
diff --git a/clutter/clutter/clutter-pick-stack.c b/clutter/clutter/clutter-pick-stack.c
index 2a18fa3d38..3e56ad704c 100644
--- a/clutter/clutter/clutter-pick-stack.c
+++ b/clutter/clutter/clutter-pick-stack.c
@@ -126,10 +126,28 @@ ray_intersects_input_region (Record                   *rec,
   if (G_LIKELY (is_axis_aligned_2d_rectangle (rec->vertices)))
     {
       graphene_box_t box;
+      graphene_box_t right_border;
+      graphene_box_t bottom_border;
+
+      /* Graphene considers both the start and end coordinates of boxes to be
+       * inclusive, while the vertices of a clutter actor are exclusive. So we
+       * need to manually exclude hits on these borders
+       */
 
       graphene_box_init_from_points (&box, 4, rec->vertices);
-      return graphene_box_contains_point (&box, point) ||
-             graphene_ray_intersects_box (ray, &box);
+      graphene_box_init_from_points (&right_border, 2, rec->vertices + 1);
+      graphene_box_init_from_points (&bottom_border, 2, rec->vertices + 2);
+
+      /* Fast path for actors without 3D transforms */
+      if (graphene_box_contains_point (&box, point))
+        {
+          return !graphene_box_contains_point (&right_border, point) &&
+                 !graphene_box_contains_point (&bottom_border, point);
+        }
+
+      return graphene_ray_intersects_box (ray, &box) &&
+             !graphene_ray_intersects_box (ray, &right_border) &&
+             !graphene_ray_intersects_box (ray, &bottom_border);
     }
   else
     {