Re: How to properly import a plugin?

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Tue, 2017-08-15 at 10:11 +0100, Sam Thursfield wrote:
> Hi Chandan,
>
> On 14/08/17 20:17, Chandan Singh wrote:
> > Hello,
> >
> > I was trying to extend one of the Element Plugins but ran into an
> > issue
> > quite early on. I'm hoping that someone on this list might have
> > some
> > answers.
> >
> > I was trying to naively extend a plugin (DpkgElement specifically)
> > like so:
> >
> > ```
> > from buildstream.plugins.elements.dpkg_build import DpkgElement
> >
> > # Element implementation for the 'my_dpkg_build' kind.
> > class MyDpkgElement(DpkgElement):
> >      pass
>
> Thanks for trying out BuildStream!
>
> By design, elements from the plugins/ module can't be extended in
> Python 
> code.

Not entirely by design, but for lack of going further.

I.e. if you allow plugins to import from other plugins, then you also
need an elaborate dependency system for plugins so that we can load
them in the right order. Currently pluginsbase gives us the feature we
want (ability to load plugins in isolated namespaces, so that you can
have two different git.py plugins loaded in the same interpretor) and
it will by itself load all the plugins in a directory.

So because of this, we added some convenience classes for plugins to
derive from (so we would not duplicate too much code in plugins).

I was not aware that `from . import sibling_plugin` works, I expect
that if Sam tried it, it perhaps works *within buildstream* ?

Or, does it work to import a buildstream plugin from a project specific
custom plugin ?

Or, does it work only to import sibling plugins from a project specific
custom plugin directory ?

If there is a solution that "just works", that would be fine; otherwise
it costs a bit more design and implementation to get us a plugin system
that understands dependencies (and then plugin revisioning will
probably weasel it's way into the spaghetti too).

> The idea is that the plugin should be flexible enough that you can 
> achieve whatever you want just from the .bst files you create, i.e.
> by 
> writing YAML only. If there's something that the DpkgElement can't
> do, 
> we should look at improving the DpkgElement plugin itself.

True, if there is a problem with dpkg elements, we would like to fix
those in buildstream rather than have subclassed python fragmented and
floating around distributed projects.

With that said, I dont want to send a message discouraging custom
plugins, that's part of the fun :)

> We do have a few elements types which are subclassed (such as the
> BuildElement base class) but these are implemented in the core 
> 'buildstream' module, not as plugins.

So yeah, we provide some base classes in the core library, these are
Element, BuildElement and ScriptElement, these are intended to be
subclassed.

Both BuildElement and ScriptElement are convenience for implementing
patterns we find common. Everything the BuildElement and ScriptElement
do can also be done by subclassing Element yourself and using
BuildStream public APIs.

Cheers,
    -Tristan

> classes (Element, BuildElement, While thee import statement worked
> successfully when I was
> > trying it
> > out in a python shell, it doesn't seem to work when it is run as
> > part of
> > the `bst build` command. Upon doing that, I get an error message
> > like so
> > (truncated):
> >
> > ```
> > ...
> >
> >    File "/usr/local/lib/python3.5/dist-packages/pluginbase.py",
> > line 410, in
> > plugin_import
> >      fromlist, level)
> >    File "/src/plugins/elements/bb_dpkg_build.py", line 1, in
> > <module>
> >      from buildstream.plugins.elements.dpkg_build import
> > DpkgElement
> >    File "/usr/local/lib/python3.5/dist-packages/pluginbase.py",
> > line 410, in
> > plugin_import
> >      fromlist, level)
> > ImportError: No module named
> > 'pluginbase._internalspace._sp606628c88e67eb4efbaa8e0085714209.elem
> > ents'
> > ```
>
> It looks like we need to make limitation this clearer in the code & 
> documentation, this error message is nonsense :-) I've opened 
> https://gitlab.com/BuildStream/buildstream/issues/64 to see if we
> can 
> give a better error.
>
> Sam