Re: [BuildStream] list_relative_paths & import ordering

[Jim MacArthur <jim macarthur codethink co uk>]

Here's an example of os.walk: https://paste.gnome.org/papwukrtn

I'll make up a proper test case which tests the output of 
list_relative_paths.


On 27/07/18 10:27, Tristan Van Berkom wrote:
> Replying on the go...
>
> One thing to note is that list_relative_paths() does not itself intend to follow symlinks or descend into 
> symlinked directories, so we should be treating them just like any other file.
>
> If os.walk() order truly behaves differently depending on what symlinks resolve to, then this is most 
> certainly a problem (unless we have addressed the output of this function in some other way with a sort or 
> such already, i'm not looking at the code right now...).
>
> Cheers,
>     -Tristan
>
> [note: my first reply seems to have missed the list for some reason, if we are munging reply-to headers on 
> this list, we should fix that]
>
>
> > On Jul 27, 2018, at 6:04 PM, Jim MacArthur via Buildstream-list <buildstream-list gnome org> wrote:
> >
> > Is the function list_relative_paths meant to reliably produce the same order of output for a given directory? 
> > I've been trying to reverse engineer the ordering it produces so I can make the CAS-backed virtual directory 
> > implementation do the same time, and run into this problem:
> >
> > Existing list_relative_paths uses os.walk() to enumerate entries and lists symlinks which target directories 
> > before listing files. The problem is that os.walk's classification of symlinks into either dirnames and 
> > filenames is dependent on whether a symlink resolves and what to, so if there is a symlink inside our 
> > directory which points outside it (say to '/usr/local') then the order in which it shows up in the output of 
> > list_relative_paths depends on whether '/usr/local' is a file or directory in the host environment.
> >
> > The returned order from list_relative_paths can affect the results of some operations. For example, consider 
> > the following two orders:
> >
> > ----
> >
> > /usr/sbin
> >
> > /sbin (a symbolic link to /usr/sbin)
> >
> > /sbin/hello (a file)
> >
> > ----
> >
> > and:
> >
> > ----
> >
> > /usr/sbin
> >
> > /sbin/hello (a file)
> >
> > /sbin (a symbolic link to /usr/sbin)
> >
> > ----
> >
> > In the second example, /sbin is created as a normal directory so you can't create the symlink /sbin without 
> > destroying or moving 'hello' (see also https://gitlab.com/BuildStream/buildstream/issues/476, which I raised 
> > to try and pin down what we do in this situation)
> >
> > I don't know if the kind of misordering I've described in list_relative_paths can actually cause a problem of 
> > the type given in the second example yet, but it seems likely.
> >
> > Should we try and properly specify the order list_relative_paths should return and make it independent of 
> > host environment?
> >
> > --
> > --
> > Codethink privacy policy: https://www.codethink.co.uk/privacy.html
> >
> > _______________________________________________
> > Buildstream-list mailing list
> > Buildstream-list gnome org
> > https://mail.gnome.org/mailman/listinfo/buildstream-list

--
--
Codethink privacy policy: https://www.codethink.co.uk/privacy.html