Re: GNOME Build situation and BuildStream

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2017-04-27 at 14:41 +0200, Sébastien Wilmet wrote:
> Hi Tristan,

[...]
> With jhbuild, when we enter into a jhbuild shell we are still in the
> same directory, usually inside the git repository. With builddir ==
> srcdir we have all the files that we can directly open with our
> preferred text editor. When we open a new terminal tab, we are in the
> same directory where we are able to 1) do git commands 2) building
> (with
> recursive make) 3) launching executables 4) editing files, etc.
>
> With BuildStream, will it be similar?

Hi Sébastien,

Good question :)

There are some things which will inevitably be different. I think the
most disruptive thing is that you will not have the experience of
having a single, persistent filesystem tree where the things you've
built "are".

This is because BuildStream does not have a serial build model but
rather will parallelize builds where possible; every build result is
stored in a separate "artifact", and sandboxed environments are created
on demand.

So, first of all to talk about VMs, launching a full VM is the
preferred way to:

  o Test how some software interacts in a full GNOME environment,
    usually the bleeding edge of development.

  o Work on modules like GNOME Shell, GDM, GNOME Session etc which
    is very difficult to isolate and work on in your host environment.


That said, today BuildStream has a `bst shell` option to stage a given
module's dependencies in a sandbox and run shell on demand.

There are two semantics for this, first of all let's assume that you
have a checkout of the GNOME build metadata (or "modulesets"), your
current working directory is at the root of that checkout, and the
module you want to hack on is called "foo".


  bst shell --scope build foo.bst
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Stage all of the build time dependencies for the module `foo.bst`,
  and also stage the sources for building `foo.bst`, and drop you
  into a shell in the build directory.

  Useful for debugging failed builds (however, when a build fails
  you will be presented an option to shell into the actual failed
  build sandbox anyway).


  bst shell --scope run foo.bst
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Stage all of the runtime dependencies for the module `foo.bst`,
  including the built `foo.bst` module itself, and drop you
  into a shell in that sandboxed environment.

  Useful for debugging applications to a certain degree, can
  run gdb and strace and similar things in here.

  This shell differs from the actual build sandbox because it allows
  some pass through of the host environment variables. This makes
  it possible to launch graphical applications like say, gedit.

  This will _only_ work well on systems which have a somewhat
  conforming environment, i.e. your host should be running dbus
  and you should have DBUS_SESSION_BUS_ADDRESS set in your environment,
  similarly you want to have DISPLAY in your environment.

  So essentially, launching graphical applications inside
  `bst shell --scope run foo.bst` should work only for the cases that
  it would have worked when using jhbuild, so no loss there really.


Now that part is already working, and dont worry about speed; even if
hundreds of "artifacts" need to be staged into a directory, this is
lightning fast and uses hardlinks to get it done.


But what you will be more interested in is your edit/compile/debug
cycles, for that we have a hard blocker before BuildStream can be
really efficient for the type of workflow you want; we're calling this
"Workspaces"[0].

With workspaces, you will be able to use a directory of your choosing
to build a specific module from (and you can have more than one "active
workspace" at a time, so you might open a workspace to hack on glib,
and another one to hack on GTK+, and have your local trees for both be
effective for any builds).

This is not done yet but here's an approximate mock shell of what I
think the UX would look like:

  # First get a local copy of the modulesets
  host$ git clone <gnome modulesets>
  host$ cd gnome-modulesets

  # Now lets create some workspaces
  host$ bst init-workspace glib.bst ~/glib
  host$ bst init-workspace gtk+.bst ~/gtk+

  # Open your favorite text editor, and edit
  # files directly in ~/glib and/or ~/gtk+
  #
  # Now build something, maybe we want to just test with gtk3-demo
  host$ bst build gtk+.bst

  # Lets test it
  host$ bst shell --scope run gtk+.bst

  # We're in the sandbox now
  sandbox$ gtk3-demo

  # Hmmm, why did it crash ?
  sandbox$ gdb gtk3-demo
  
  # Ah, I see what I did there...
  sandbox$ exit

  # Edit some files in ~/glib and/or ~/gtk+ and try again
  #
  host$ bst build gtk+.bst
  host$ bst shell --scope run gtk+.bst
  sandbox$ gtk3-demo
  sandbox$ exit

  # Ok that worked !
  host$ cd ~/gtk+
  host$ git commit -a -m "Its a patch !"

  # Do appropriate thing, maybe you push, maybe you
  # do `git format-patch` and post some patch
  #
  # At this point you may want to continuously leave
  # the workspace open, or you may be done with that
  # work and want to track upstream again.
  host$ cd -
  host$ bst close-workspace glib.bst
  host$ bst close-workspace gtk+.bst


So, this is a little bit fiddly compared to working entirely within one
build sandbox, only because you really need to exit and enter a sandbox
environment when you want to try something out, otherwise it's snappy
(and maybe a convenience command to say "build + shell" in one go could
reduce a bit of typing).

On the bright side, you never ever trust your host environment for
anything, except for a display server and session bus in the case that
you use `bst shell` to run things.

Cheers,
    -Tristan


[0]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/Workspaces