IDE completion features and BuildStream [Was: Re: GNOME Modulesets migrating to BuildStream - clang completion in text editor]

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi,

I'd like to start a thread on our mailing list to address this concern
which was brought up on d-d-l:

    https://mail.gnome.org/archives/desktop-devel-list/2017-November/msg00070.html

In summary, some people like to use IDE features which give them symbol
browsers and symbol completion features based on some introspection and
context of what the user is editing.

For this purpose, the user needs to have a sysroot of sorts, a
filesystem tree where header files, girs and basically the developer
metadata installed by the modules which the project depends on.

Using BuildStream, there is no single filesystem tree which makes
sense, one build might have a slightly different tree; depending on
what exact version (usually git commit sha) of each dependency is
active for a given build, however; whipping up a build directory is
something that BuildStream obviously does, it's a pretty core feature.

So, to help users who want to use these kind of IDE features I'd like
to brainstorm a bit about what the feature should look like.

It should:

  a.) Be easy to use for a regular user, the user should be able to
      obtain a sysroot they can feed to their IDE of choice.

  b.) Be interoperable with IDEs like GNOME Builder, such that it is
      possible for an IDE to automatically introspect where an
      appropriate sysroot can be found for a given element.

After a week of this brewing in the background, I feel like the most
elegant solution might be to extend workspaces a bit to work with this.

Creating checkouts on demand, even if we add a `--hardlink` option to
the checkout command for optimization, when it must be done explicitly,
periodically, this is unwieldy. This means that every time the latest
versions of things change; the user is required to update their
checkout themselves in order to have the correct development headers to
develop against.

> From a workspace point of view however, there is an opportunity to
update a fixed sysroot every time we do tracking, fetching or building
operations (every time that the weak cache keys of the dependencies
have changed and buildstream is run, or something along those lines).

These workspace sysroots would be hardlinked staging grounds, and the
user MUST not ever open those files for writing, lest they go and
corrupt their local artifact caches.

As they would belong to the workspace, opening and closing the
workspace would automatically create or delete the said sysroot.

The command `bst workspace list` which outputs machine readable YAML
would also have to output the given corresponding sysroot for an active
workspace.

There are some pretty serious edge cases to consider further here
though, which also make me hesitant about this plan, which includes
some additional local state management which is undesirable as well:

  o what happens when `bst build --track` is run, and the newly tracked
    source references are never saved anymore ?

    Is updating the local state at `bst build --track` time enough ?

    Would we have to shadow the local state of actual checked out
    versions in the local buildstream state directory in order
    to keep track of what needs updating ?

  o When the user updates their project via git (like pulling the
    latest of the GNOME modulesets over git), buildstream is not
    informed of this, so maybe this alone is a good argument to
    have a less controlled approach, and demand that something
    external create checkouts on demand ?

  o Workspaces are Source based, an element can have more than
    one source; is the structure correct to have a sysroot tied
    to a given source of an element ?


What do people think about this approach ? Does anyone have other ideas
of how to elegantly satisfy this use case ?

Cheers,
    -Tristan



-------- Forwarded Message --------
From: Tristan Van Berkom <tristan vanberkom codethink co uk>
To: Sébastien Wilmet <swilmet gnome org>
Cc: desktop-devel-list <desktop-devel-list gnome org>
Subject: Re: GNOME Modulesets migrating to BuildStream - clang completion in text editor
Date: Fri, 24 Nov 2017 17:54:08 +0900

Hi!

Quick handphone reply...

> > On Nov 24, 2017, at 5:13 PM, Sébastien Wilmet <swilmet gnome org> wrote:
>
> > On Fri, Nov 24, 2017 at 03:16:09PM +0900, Tristan Van Berkom wrote:
> > Had not considered this use case yet, thanks !
> >
> > I'm an emacs user but have never used any kind of symbol completion
> > feature myself.
> >
> > For this, one could use `bst checkout` to get a full checkout of the
> > build outputs which your element (module) of choice depends on, and
> > then periodically delete / re-checkout your sysroot checkout when it
> > starts to get out of date (or make parallel checkouts for separate
> > projects).
> >
> > However currently this is a slow operation which uses a lot of disk
> > space. Since it is an export of the built artifacts, we dont risk using
> > hardlinks for the checkout, because that leaves the user in a position
> > where they can accidentally corrupt their local artifact cache.
> >
> > I have been considering adding some explicit switch like `--unsafe` to
> > the checkout command to allow users to do this "at their own risk", but
> > haven't really found a use case to justify this, maybe you just
> > provided one !
>
> As Christian explained, the text editor plugin also needs the list of
> CFLAGS, especially the -I's.
>
> With Meson the easiest is to read the compile_commands.json file created
> at the root of the build directory. See:
> https://clang.llvm.org/docs/JSONCompilationDatabase.html
>
> CMake can also create a compile_commands.json file.
>
> With the Autotools/make, the Vim plugin that I use provides a script to
> extract the CFLAGS while `make` is running:
> $ make CC='~/.vim/bin/cc_args.py gcc'
> this creates a .clang_complete file containing the list of CFLAGS.
> See:
> https://github.com/Rip-Rip/clang_complete/

In this case, most probably `bst shell --build` gets us closest, it will create a sandbox on the fly, 
containing the sources you intend to build, and fail if the dependencies are not yet built, but persist only 
for the lifetime of the interactive shell it spawns.

Same build environment that BuildStream will create for a build.

The functionality is all there but it sounds like we should carefully design the right interface for casual 
users and Builder users alike.

Cheers,
    -Tristan

> > Philip's reply is also a possibility but I would prefer recommending
> > something via BuildStream, mostly because you want to use exactly the
> > header files that you need for your target environment, but also
> > because there is no guarantee that a flatpak SDK will even have headers
> > for the dependencies you might want to use.
>
> Yes the Flatpak SDK doesn't contain all dependencies.
>
> > For the moment, I've added this issue[0] to make sure we dont lose
> > sight of this, in any case it should be very easy to implement.
> >
> > [0]: https://gitlab.com/BuildStream/buildstream/issues/162
>
> OK, it's a good news if it's easy to implement.
>
> --
> Sébastien

_______________________________________________
desktop-devel-list mailing list
desktop-devel-list gnome org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list