Re: [BuildStream] list_relative_paths & import ordering

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Replying on the go...

One thing to note is that list_relative_paths() does not itself intend to follow symlinks or descend into 
symlinked directories, so we should be treating them just like any other file.

If os.walk() order truly behaves differently depending on what symlinks resolve to, then this is most 
certainly a problem (unless we have addressed the output of this function in some other way with a sort or 
such already, i'm not looking at the code right now...).

Cheers,
   -Tristan

[note: my first reply seems to have missed the list for some reason, if we are munging reply-to headers on 
this list, we should fix that]


> On Jul 27, 2018, at 6:04 PM, Jim MacArthur via Buildstream-list <buildstream-list gnome org> wrote:
>
> Is the function list_relative_paths meant to reliably produce the same order of output for a given 
> directory? I've been trying to reverse engineer the ordering it produces so I can make the CAS-backed 
> virtual directory implementation do the same time, and run into this problem:
>
> Existing list_relative_paths uses os.walk() to enumerate entries and lists symlinks which target 
> directories before listing files. The problem is that os.walk's classification of symlinks into either 
> dirnames and filenames is dependent on whether a symlink resolves and what to, so if there is a symlink 
> inside our directory which points outside it (say to '/usr/local') then the order in which it shows up in 
> the output of list_relative_paths depends on whether '/usr/local' is a file or directory in the host 
> environment.
>
> The returned order from list_relative_paths can affect the results of some operations. For example, 
> consider the following two orders:
>
> ----
>
> /usr/sbin
>
> /sbin (a symbolic link to /usr/sbin)
>
> /sbin/hello (a file)
>
> ----
>
> and:
>
> ----
>
> /usr/sbin
>
> /sbin/hello (a file)
>
> /sbin (a symbolic link to /usr/sbin)
>
> ----
>
> In the second example, /sbin is created as a normal directory so you can't create the symlink /sbin without 
> destroying or moving 'hello' (see also https://gitlab.com/BuildStream/buildstream/issues/476, which I 
> raised to try and pin down what we do in this situation)
>
> I don't know if the kind of misordering I've described in list_relative_paths can actually cause a problem 
> of the type given in the second example yet, but it seems likely.
>
> Should we try and properly specify the order list_relative_paths should return and make it independent of 
> host environment?
>
> -- 
> --
> Codethink privacy policy: https://www.codethink.co.uk/privacy.html
>
> _______________________________________________
> Buildstream-list mailing list
> Buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list