Re: Fwd: ostree static-delta

[Gatis Paeglis <Gatis Paeglis qt io>]

>> Yeah, we do need to fix this. It's https://bugzilla.gnome.org/show_bug.cgi?id=764009

> would be helpful for now to add another pass, when not matching basename is
> found, that will consider something more fuzzy (e.g. compare the basenames up to the
f> irst '.')?

> I made a quick experiment with the patch below.

This sounds like a reasonable option when the tree is committed as a whole, and therefore can not be optimized at component-level. I will test the patch.

> It's pretty interesting that the wall clock time is so similar though - I'm

> guessing I/O performance (perhaps the syncfs()) is the blocker for

> your device?

I was expecting the static-delta approach to take a significantly less time. I am booting from a USB stick. I tried the following sequence of commands with "no-deltas", "deltas-fsync", "deltas-no-fsync":

ostree --dry-run --require-static-deltas pull ..

ostree pull --require-static-deltas .. (and with --disable-fsync)

ostree admin deploy ..

Time for all 3 cases was about the same ~6min.

> Nevertheless, the testing results on Intel NUC show that an update size is reduced by 2/3 when updating OpenEmbedded image (fido branch -> jethro branch).

It seems like these results were a fluke, as I can not reproduce 2/3 size improvement anymore. Today I am getting:

196612 KiB // no deltas

146121 KiB // with deltas

I will see how Giuseppe's patch improves the numbers and post them here later.

Gatis.

---

From: ostree-list <ostree-list-bounces gnome org> on behalf of Giuseppe Scrivano <gscrivan redhat com>
Sent: Wednesday, July 20, 2016 1:41:23 PM
To: Colin Walters
Cc: ostree-list gnome org
Subject: Re: Fwd: ostree static-delta

 

Colin Walters <walters verbum org> writes:

>  If I read the code correctly then OSTree won't generate a static delta for libQt5Core.so.5.6.1 -> libQt5Core.so.5.7.0 update. What is a definition of a "basename" here?
>  With Shell "basename libQt5Core.so.5.6.1" -> "libQt5Core.so.5.6.1"
>
> Yeah, we do need to fix this. It's https://bugzilla.gnome.org/show_bug.cgi?id=764009

would be helpful for now to add another pass, when not matching basename is
found, that will consider something more fuzzy (e.g. compare the basenames up to the
first '.')?

I made a quick experiment with the patch below.

Regards,
Giuseppe

diff --git a/src/libostree/ostree-repo-static-delta-compilation-analysis.c b/src/libostree/ostree-repo-static-delta-compilation-analysis.c
index ae5b8de..27b1380 100644
--- a/src/libostree/ostree-repo-static-delta-compilation-analysis.c
+++ b/src/libostree/ostree-repo-static-delta-compilation-analysis.c
@@ -189,7 +189,8 @@ build_content_sizenames_filtered (OstreeRepo              *repo,
 
 static gboolean
 string_array_nonempty_intersection (GPtrArray    *a,
-                                    GPtrArray    *b)
+                                    GPtrArray    *b,
+                                    gboolean      fuzzy)
 {
   guint i;
   for (i = 0; i < a->len; i++)
@@ -199,8 +200,17 @@ string_array_nonempty_intersection (GPtrArray    *a,
       for (j = 0; j < b->len; j++)
         {
           const char *b_str = b->pdata[j];
-          if (strcmp (a_str, b_str) == 0)
-            return TRUE;
+          const char *dot = strchr (a_str, '.');
+          if (dot && fuzzy)
+            {
+              if (strncmp (a_str, b_str, dot - a_str) == 0)
+                return TRUE;
+            }
+          else
+            {
+              if (strcmp (a_str, b_str) == 0)
+                return TRUE;
+            }
         }
     }
   return FALSE;
@@ -258,6 +268,8 @@ _ostree_delta_compute_similar_objects (OstreeRepo                 *repo,
   upper = from_sizes->len;
   for (i = 0; i < to_sizes->len; i++)
     {
+      int fuzzy;
+      gboolean found = FALSE;
       OstreeDeltaContentSizeNames *to_sizenames = to_sizes->pdata[i];
       const guint64 min_threshold = to_sizenames->size *
         (1.0-similarity_percent_threshold/100.0);
@@ -268,31 +280,41 @@ _ostree_delta_compute_similar_objects (OstreeRepo                 *repo,
       if (to_sizenames->size == 0)
         continue;
 
-      for (j = lower; j < upper; j++)
+      for (fuzzy = 0; fuzzy < 2 && !found; fuzzy++)
         {
-          OstreeDeltaContentSizeNames *from_sizenames = from_sizes->pdata[j];
-
-          /* Don't build candidates for the empty object */
-          if (from_sizenames->size == 0)
-            continue;
-
-          if (from_sizenames->size < min_threshold)
+          for (j = lower; j < upper; j++)
             {
-              lower++;
-              continue;
+              OstreeDeltaContentSizeNames *from_sizenames = from_sizes->pdata[j];
+
+              /* Don't build candidates for the empty object */
+              if (from_sizenames->size == 0)
+                {
+                  continue;
+                }
+
+              if (from_sizenames->size < min_threshold)
+                {
+                  lower++;
+                  continue;
+                }
+
+              if (from_sizenames->size > max_threshold)
+                break;
+
+              if (!string_array_nonempty_intersection (from_sizenames->basenames,
+                                                       to_sizenames->basenames,
+                                                       fuzzy == 1))
+                {
+                  continue;
+                }
+
+              /* Only one candidate right now */
+              g_hash_table_insert (ret_modified_regfile_content,
+                                   g_strdup (to_sizenames->checksum),
+                                   g_strdup (from_sizenames->checksum));
+              found = TRUE;
+              break;
             }
-
-          if (from_sizenames->size > max_threshold)
-            break;
-
-          if (!string_array_nonempty_intersection (from_sizenames->basenames, to_sizenames->basenames))
-            continue;
-           
-          /* Only one candidate right now */
-          g_hash_table_insert (ret_modified_regfile_content,
-                               g_strdup (to_sizenames->checksum),
-                               g_strdup (from_sizenames->checksum));
-          break;
         }
     }
 
_______________________________________________
ostree-list mailing list
ostree-list gnome org
https://mail.gnome.org/mailman/listinfo/ostree-list