Hello all, IMPORTANT NOTE: To view the (text) tables in this email, you'll need at least 83 chars for word wrapping. I've recently been working on generating data and preparing results to benchmark our Debian-like project. I've gathered data for all of the *merge commits* since 01/01/2019 which show the time taken to show, show with the YAML cache, build and show once built, Debian's base files (base- files/base-files.bst from the Debian-like project [0]). This produces a pipeline of ~6100 elements. Attached is: 1. A zip file with 4 results files (containing all results since 1st of Jan), and a Jupyter notebook which reads these in these files and produces the graphs. 2. A Jupyter notebook (weekly_results.ipynb), which contains embedded data for the last 7 days worth of merge commits, this will be used to track our improvements week-by-week. IMPORTANT: To view the notebooks, you'll need to install Jupyter notebook, matplotlib and the python package mplcursors. I'd recommend the following: * apt install jupyter-notebook python-matplotlib - or equivalent * pip3 install --user mplcursors To view the notebooks, I'd recommend downloading the notebook and extracting the zip into an empty directory and executing `jupyter-notebook`, open the link and then 'Cell' -> 'Run All' (note that you may have to rerun a cell to reproduce the interactive graph, do this with shift+Enter). You should then see the graphs, these are pretty self explanatory and they have some cool interactive features, e.g. you can zoom into certain regions with the tools provided in the toolbar below the graph. Also, by clicking on a data point, you should be able to see the !MR and the branch of that (merge) commit. You should see 4 graphs: 1. Time taken to show base-files/base-files.bst (without YAML cache, with YAML cache and with YAML cache and populated local cache). 2. Max memory to show base-files/base-files.bst(without YAML cache, with YAML cache and with YAML cache and populated local cache). 3. Time taken to build base-files/base-files.bst. 4. Max memory when building base-files/base-files.bst. All results were obtained on the same hardware (a Codethink dev machine configured to be a gitlab runner - this machine is not being used for anything else), and the results files are being appended to by a CI job [1]. Here are some specs of the Dev machine: * Linux (Debian stable) * x86_64 * 16G RAM * 500Gb SSD Now, because we have been focusing on performance since the gathering, I've taken an average of the results, per week, which I will now send out as weekly updates so that we can track our improvements. Below, are the averages of the show and build times of base-files/base-files.bst for the last 4 weeks: | Dates | Show (s) | Show with cache (s) | Build (s) | Show once built (s) | |-------------|----------|---------------------|-----------|---------------------| | 21/01-27/01 | 46.69 | 27.40 | 378.58 | 33.65 | | 28/01-03/02 | 46.36 | 27.35 | 383.83 | 33.49 | | 04-02-10/02 | 45.72 | 26.53 | 268.70 | 32.68 | | 11/02-17/02 | 43.07 | 24.04 | 228.94 | 30.13 | Below, are the averages for the maximum memory usage for the show and build of base-files/base- files.bst for the last 4 weeks: | Dates | Show (MB) | Show with cache (MB) | Build (MB) | Show once built (MB) | |-------------|-----------|----------------------|------------|----------------------| | 21/01-27/01 | 1084 | 1108 | 1109 | 1110 | | 28/01-03/02 | 1083 | 1108 | 1109 | 1110 | | 04-02-10/02 | 556 | 581 | 582 | 582 | | 11/02-17/02 | 403 | 425 | 425 | 426 | Hopefully, over the coming weeks/months, we'll be seeing all of these numbers continuously drop! Note that in the future, I will not be attching the zip with *all* the data, I'll just leave a (gitlab) link which will let you download the artifacts. However, I will be attaching the weekly_results notebook. Please let me know if you have any thoughts/improvements on the benchmarks. I hope you find the data and results helpful! Thanks, James [0] https://gitlab.com/jennis/debian-stretch-bst [1] https://gitlab.com/jennis/benchmark_debian
Attachment:
all_results.zip
Description: Zip archive
Attachment:
weekly_results.ipynb
Description: application/ipynb