Re: [BuildStream] Strength our testing effort: overnight builds.

[Laurence Urhegyi <laurence urhegyi codethink co uk>]

Hi,

On 2018-11-26 10:08, Agustín Benito Bethencourt via BuildStream-list 
wrote:
<snip>
> additional effort. I am talking about the Overnight tests[1].
>
> In order to do so, the first thing we need is a list of actions (to
> test) we can add to be performed in these builds, executed on daily
> basis. In the future we will structure them based on different
> criteria so we will decide how and when to execute them. For now, we
> are happy with a simple and flat list. The idea is to prioritize them
> to then evaluate who and how will implement them. Ideally we will add
> some of the items of the list to the current master build before
> branching 1.4.

Questions have been raised recently about the testing of remote 
execution with BuildStream [0], which was something that wasn't fully 
possibly until CommandBatching [1] landed. Elsewhere, others have 
discussed end to end testing, potentially using a behavioural test suite 
framework [2]. Not sure, but this could be a good candidate for 
overnight builds [3]. I think some manual testing steps should be 
written up, and they should be useful.

We also have the benchmarks [4], which are separate to the overnight 
tests, but possibly share some similarities and I think whoever is 
focusing on benchmarks should probably work with Valentin, at least to 
share knowledge and maybe even share aspects of the infrastructure, to 
ensure we're not duplicating effort. The benchmarks as they are at the 
moment are not part of BuildStream's CI, they run on a daily cron job, 
but do run against each new commit to master. Most people would actually 
like to get these into the CI, as far as I can tell. My point is let's 
look at the picture here as a whole.

Thanks,
Laurence

[0] 
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00093.html
[1] https://gitlab.com/BuildStream/buildstream/issues/675
[2] 
https://lists.buildgrid.build/pipermail/buildgrid/2018-December/000098.html
[3] https://gitlab.com/BuildStream/buildstream/pipeline_schedules
[4] https://gitlab.com/BuildStream/benchmarks