Re: ostree - how to enable libcurl's continue/resume feature

[Davis Roman <davis roman84 gmail com>]

Hi Dan,

Thank you very much for the very insightful response.

Another approach that we're taking is trying to figure out why 'ostree pull' fails in the first place due to our very sluggish data connection. 

In https://github.com/ostreedev/ostree/blob/master/src/libostree/ostree-fetcher-curl.c, curl_easy_setopt is being called so potentially we could pass the following two options with more relaxed values to make curl behave.

https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html 

(It contains the time in number seconds that the transfer speed should be below the CURLOPT_LOW_SPEED_LIMIT for the library to consider it too slow and abort.)

https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html 

(It contains the average transfer speed in bytes per second that the transfer should be below during CURLOPT_LOW_SPEED_TIME seconds for libcurl to consider it to be too slow and abort.)

Do you think this is a reasonable path to consider? If not, does anything else come to mind?

Thank you,

Davis

On Tue, Dec 22, 2020 at 6:36 PM Dan Nicholson <dbn endlessos org> wrote:

On Tue, Dec 22, 2020 at 1:59 PM Davis Roman via ostree-list
<ostree-list gnome org> wrote:
>
> Hello,
>
> When using curl, there is an option to Continue/Resume a previous file transfer at the given offset using the -C flag.
>
> Our understanding is that ostree uses libcurl to download its metadata so we're trying to figure out if it's possible to instruct ostree to enable libcurl's "continue/resume" feature.

ostree doesn't support that feature and it would take some work to get
it right. I don't believe that it saves partially downloaded objects
that could be continued. Assuming that it did save partial objects
(this concept would need some thought to ensure that it could tell the
difference between a partial download and a corrupted download), then
presumably you could try to resume a failed transaction with partial
objects. Then the fetcher would need to recognize them and switch to
HTTP range requests. I think it would work OK for most types of
objects where the checksum is known ahead of time and there's a
reliable way to tell if the staged object is corrupt or partial.

I believe https://github.com/ostreedev/ostree/issues/639 describes
what you're after, but as you can see no one's worked on it.

--
Dan