Re: [BuildStream] How to catch regressions in buildstream's interactive interface

[Mathieu Bridon <bochecha daitauha fr>]

On Tue, 2018-10-16 at 10:19 +0100, Tristan Van Berkom via BuildStream-
list wrote:
> On Mon, 2018-10-15 at 17:59 +0100, Jonathan Maw via BuildStream-list
> wrote:
> > Hi,
> >
> > I am working on an issue that involves changing how buildstream
> > works 
> > when you use buildstream in interactive mode, fail a build, and
> > launch 
> > the shell. While it's mostly the same codepath as `bst shell --
> > build`, I 
> > am concerned that my ability to test its workings are extremely
> > manual.
> >
> > The solutions I can think of are:
> > * A checklist of behaviour that can only be manually checked? Bad, 
> > because it'd lead to an ever-expanding list of things to check as
> > time 
> > goes on.
> > * Some kind of test harness that monitors stdout/stderr and can
> > write 
> > commands to stdin? Very hard to write something general-purpose,
> > and 
> > since it's not using machine-readable output, prone to behaving in 
> > unexpected ways.
> >
> > Is this a goal worth pursuing? Does anyone know any other
> > solutions?
>
> Definitely the latter yes. We need a test harness which allows
> interactive communication with BuildStream.
>
> This is a significant hole in our current CI and is definitely worth
> pursuing :)

Can https://pexpect.readthedocs.io/en/stable/ help here?


-- 
Mathieu