Re: Benchmarking BuildStream commands from user point of view

[Angelos Evripiotis <angelos evripiotis gmail com>]

Here's an early look at how I think the benchmarking can work, I'm looking for
feeback to steer this work in progress.

I've put together a Python script here:
https://gitlab.com/BuildStream/buildstream/merge_requests/136

It outputs delimiter seperated values for easy handling on the command-line.
It's not CSV just now, as the number of fields is variable.

Here you can see that compared to starting Python, BuildStream has a lot of
work to do, running time in seconds:

    python,0.03
    python_import_buildstream,0.57
    help,0.69 ('bst help')

Personally I find this start-up pause to be quite noticable when working with
'bst'.

Here you can see that the cost of composing grows steadily with the number of
files involved. In this case we are only using 'import' and 'compose' elements
to handle text files, there is no compiling etc. The ascii graph seems able to
reveal a steady trend from the 10 data points:

                      local files: build compose.bst (secs)

  35 +-+----+------+------+------+------+------+------+------+------+----+-+
     +      +      +      +      +      +      +      +      +   ****      +
  30 +-+                                                  *******        +-+
     |                                               *****                 |
     |                                           ****                      |
  25 +-+                                    *****                        +-+
     |                                 *****                               |
  20 +-+                           ****                                  +-+
     |                        *****                                        |
     |                   *****                                             |
  15 +-+             ****                                                +-+
     |          *****                                                      |
  10 +-+    ****                                                         +-+
     |    **                                                               |
     |  **                                                                 |
   5 +**                                                                 +-+
     *      +      +      +      +      +      +      +      +      +      +
   0 +-+----+------+------+------+------+------+------+------+------+----+-+
     0     1000   2000   3000   4000   5000   6000   7000   8000   9000  10000
                                   file count

Planned next steps:

    o Add more benchmarks!
        o Vary the number of .bsts.
        o Involve workspaces.
        o Involve the git source plugin.
        o Involve 'bst shell'.

    o Make it possible to aim for different running times for the suite, e.g.
      'quick test', 'an hour', 'exhaustive'. Adjusting the sampling rate and
      range to the running time.

    o Use pandas and Jupyter notebooks to look for and present findings. Ascii
      only goes so far :)

Later:

    o Involve project configuration, see how rules change things.
    o Involve all elements / plugins.
    o Select which tests to run.
    o Involve 'bst push' and 'bst pull'.
    o Cover everything, e.g. splits

Cheers,
Angelos

P.S. here's the gnuplot invocation for the ascii graph:

$ grep 'import_local_files:build compose element' times.txt | cut -d,
-f2,3 | gnuplot -e 'set terminal dumb; set title "local files: build
compose.bst (secs)"; set datafile separator ","; set xlabel "file
count"; plot "/dev/stdin" with lines notitle'