Re: [BuildStream] Proposal: bst artifact subcommand group

[Paul Sherwood <paul sherwood codethink co uk>]

On 2018-07-24 11:48, Tristan Van Berkom via Buildstream-list wrote:
<snip>
> It was recently raised that we cannot determine the build date of an
> existing artifact either[1], as it's not a part of artifact metadata,
> this discussion evolved to proposing that we have a subgroup of CLI
> commands for dealing with specific artifacts.

Perhaps a dumb question, but why isn't creation date of the artifact 
file/archive good enough?

> As such I'd like to propose the following, probably uncontroversial
> enhancements:
>
>   o Add the build date to the artifact metadata, as a timezone neutral
>     unix timestamp (seconds since epoch in UTC).

Don't forget to bump the 'cache-key algorithm' (does bst have a 
different name for this?) to avoid future bst implementations making 
incorrect assumptions about what's in a give artifact

>   o Standardize on the name of an artifact, this is not really exposed
>     in the UX except the user may see "artifact names" at times.

I agree this has to be done.

>     An artifact name is essentially:
>
>        ${project-name}/${element-name}/${cache-key}

I believe that $(project-name) is not relevant. If you encode that, (it 
will at least seem like) artifacts can't be shared between projects.

Is there a reason for choosing "/" ?

For ybd I chose "." so that artifacts can be stored in a single 
directory. Maybe you are expressly trying to establish that the 
artifacts should all sit under one subdir for the name, but that also 
seems to be a wrong turn to me.

Given that I don't understand the ostree or CAS implementations I may be 
conflating filesystem names with artifact names, though.

>     Exposing this to the user essentially means that we have a handy
>     syntax for the user to express an artifact on the command line.

From a user POV, the name.key standard (with creation time on the 
artifact itself) has proved useful in practice

>   o Add the `bst artifact` subgroup with the following commands:
>
>     o bst artifact list <artifact name glob pattern>

As you know I'm rather against adding lots of commands and options since 
it increases the documentation and learning surface. Also, I'm not 
convinced that people (maybe you included) are using bst in enough 
scenarios to be clear about what functionality will really be useful.

In the ybd scenario, the kbas functionality (and the shape of artifact 
surface) evolved based on user feedback (including my own lessons as a 
user of previous tooling)

>       By default listing only the artifacts which match the glob
>       pattern.
>
>       An additional `--long/-l` argument can get us more human
>       readable information, similar to what was outlined here[2]
>
>       List the artifacts in order of build date, displaying the build
>       date in local time along with the size of the files portion of
>       the artifact, and the active workspace if it was created locally
>       with a workspace.
>
>     o bst artifact log <artifact name or pattern>
>
>       Display the artifact build log in the $PAGER, similar to how we
>       implement the "log" choice on the prompt of a failing build.
>
>       If a pattern is specified then the log of each will be sent to
>       the system pager in series (matching to the behavior of
>       specifying wildcards to programs like "less").
>
>     o bst artifact delete <artifact name or glob pattern>
>
>       Delete artifacts from the local cache which match the pattern
>
>     o bst artifact list-content <artifact name or glob pattern>
>
>       This can be useful for scriptability purposes, where one wants to
>       generate a manifest of what an artifact contains, or simply for
>       a curious user to see what files an artifact contains.
>
>       Similarly, this should have `--long/-l` options to show detailed
>       information about the files in the artifact, such as
>       user/group/everyone permission bits, ownership bits, file size,
>       etc (I think `tar -t` offers this with a `-vv` switch or such).
>
>     o bst artifact diff <artifact name> <artifact name>
>
>       Show differing added/removed and differing files in two artifacts
>
> Beyond this:
>
>   o It will be interesting to allow artifact operations to specify
>     elements instead of artifact names, for the cases where you just
>     want to use the artifact who's cache key corresponds to the project
>     state.
>
>   o It would also be interesting to move `checkout`, `push` and `pull`
>     commands under the new `artifact` group (deprecating the existing
>     commands).
>
>     This will open up the door to performing checkouts etc at the
>     artifact level instead of only supporting the artifacts who's
>     cache keys correspond to the project state.
>
> If nobody opposes the proposal, I will go ahead and roadmap this by
> creating a flagship issue on gitlab with a task list, each task
> pointing to individual separate issues for the individually proposed
> commands.

I'm not "opposing", just generally concerned that you may be specifying 
at least in part "nice to have" functionality that may not actually be 
needed by users.

Much of the above could be achieved with other tools, if the artifacts 
are presented as tarballs for example.

br
Paul