[BuildStream] list_relative_paths & import ordering

[Jim MacArthur <jim macarthur codethink co uk>]

Is the function list_relative_paths meant to reliably produce the same 
order of output for a given directory? I've been trying to reverse 
engineer the ordering it produces so I can make the CAS-backed virtual 
directory implementation do the same time, and run into this problem:

Existing list_relative_paths uses os.walk() to enumerate entries and 
lists symlinks which target directories before listing files. The 
problem is that os.walk's classification of symlinks into either 
dirnames and filenames is dependent on whether a symlink resolves and 
what to, so if there is a symlink inside our directory which points 
outside it (say to '/usr/local') then the order in which it shows up in 
the output of list_relative_paths depends on whether '/usr/local' is a 
file or directory in the host environment.

The returned order from list_relative_paths can affect the results of 
some operations. For example, consider the following two orders:

----

/usr/sbin

/sbin (a symbolic link to /usr/sbin)

/sbin/hello (a file)

----

and:

----

/usr/sbin

/sbin/hello (a file)

/sbin (a symbolic link to /usr/sbin)

----

In the second example, /sbin is created as a normal directory so you 
can't create the symlink /sbin without destroying or moving 'hello' (see 
also https://gitlab.com/BuildStream/buildstream/issues/476, which I 
raised to try and pin down what we do in this situation)

I don't know if the kind of misordering I've described in 
list_relative_paths can actually cause a problem of the type given in 
the second example yet, but it seems likely.

Should we try and properly specify the order list_relative_paths should 
return and make it independent of host environment?

--
--
Codethink privacy policy: https://www.codethink.co.uk/privacy.html