Re: [BuildStream] Dropping support for `setup.py test`

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Chandan,

> On Dec 6, 2019, at 12:39 AM, Chandan Singh <chandan chandansingh net> wrote:
>
> Hi all,
>
> In this thread, I am proposing to remove support for running tests via
> `setup.py test`. More details following, but please respond if you think this
> will negatively affect some workflows.
>
> Currently, BuildStream recommends using `tox` as the primary test runner, but we
> also support `setup.py test` to some extent. In practice, it is unverified and
> does not offer a good developer experience for following reasons:
>
> * `setup.py test` does not handle build requirements correctly, so it will _not_
>  install Cython automatically before running tests.
>
> * External plugins are currently not listed as hard dependency, so running any
>  tests that involves external plugins will also fail if the user doesn't happen
>  to have the right version installed.
>
> * Last but not the least, it is officially deprecated [1]. Setuptools folks
>  recommend using `tox` in similar cases, which we already do.
>
> Personally, I am not sure what problem we are solving with supporting
> `setup.py test`.

The `setup.py test` invocation is important precisely *because* it does not install any specific set of 
dependencies automatically or run in any virtual environment.

This is the correct testing path we need to have for the integrator workflow (not for the developer 
workflow), as this allows the integrator to properly test how BuildStream will be have once it is integrated 
with your system and installed on your machine.

If `setup.py test` invocations are really going to be removed from setuptools, then certainly there must be 
an alternative to replace it (tox without venv ?) because it really does continue to serve a real purpose, 
not only for BuildStream.

Cheers,
    -Tristan



> Quoting from our documentation:
>
> > While using tox is practical for developers running tests in more predictable
> > execution environments, it is still possible to execute the test suite against
> > a specific installation environment using pytest directly:
> >
> > ./setup.py test
>
> In this case, I am not sure what `setup.py test` offers on top of just running
> `pytest`. If the user has to install Cython, BuildStream and external plugins
> manually anyway, why not let them deal with test dependencies as well? Also,
> `tox --sitepackages` should also be equivalent in these scenarios.
>
> `setup.py test` is going to be removed in a future setuptools version anyway. Do
> people see value in supporting it in the meantime?
>
> Thanks,
> Chandan
>
> [1]: https://setuptools.readthedocs.io/en/latest/setuptools.html#test-build-package-and-run-a-unittest-suite
> _______________________________________________
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list