[gnome-software] ci: Rework the CI to support multiple Docker images
- From: Richard Hughes <rhughes src gnome org>
- To: commits-list gnome org
- Cc:
- Subject: [gnome-software] ci: Rework the CI to support multiple Docker images
- Date: Thu, 7 Nov 2019 11:04:23 +0000 (UTC)
commit f6f8f22c0e96a7dbee0f0bba79514cadfd813d4b
Author: Philip Withnall <withnall endlessm com>
Date: Wed Nov 6 10:41:06 2019 +0000
ci: Rework the CI to support multiple Docker images
This is a major reworking of the CI system. Rather than downloading all
the build dependencies on every CI run, we now build a couple of Docker
images with all the relevant build dependencies pre-installed. Rather
than building only on Fedora, we now build on Fedora *and* Debian
Stable.
To update one of the Docker images, update the `.Dockerfile`, then run
```
.gitlab-ci/run-docker.sh build --base=fedora --base-version=2
.gitlab-ci/run-docker.sh push --base=fedora --base-version=2
```
The `--base-version` should be incremented each time you produce a new
Docker image (i.e. once per MR which touches the `Dockerfile`s). This
means you will end up with multiple versions of each Docker image in the
registry, which is necessary as old MRs will only be run against the
version which was listed in `.gitlab-ci.yml` when they branched.
Once a new Docker image is pushed, it needs to be referenced from the
`image:` keys in `.gitlab-ci.yml`.
It includes a script, `meson-junit-report.py`, which takes the JSON
test report and converts it to a Cobertura format suitable for GitLab to
understand and summarise.
It’s worth noting the FIXME comment in `run-tests.sh` at the
moment: tests can only be run as root. That seems like an unnecessary
restriction which implies that the tests need further sandboxing and
more mock D-Bus services added to them.
Signed-off-by: Philip Withnall <withnall endlessm com>
Helps: #825
.gitlab-ci.yml | 78 ++++++++++++++---------
.gitlab-ci/README.md | 22 +++++++
.gitlab-ci/debian-stable.Dockerfile | 58 +++++++++++++++++
.gitlab-ci/fedora.Dockerfile | 63 +++++++++++++++++++
.gitlab-ci/meson-junit-report.py | 114 ++++++++++++++++++++++++++++++++++
.gitlab-ci/run-docker.sh | 120 ++++++++++++++++++++++++++++++++++++
.gitlab-ci/run-tests.sh | 35 +++++++++++
7 files changed, 462 insertions(+), 28 deletions(-)
---
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8a174843..690c6fc4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,34 +1,56 @@
-image: fedora:31
-
stages:
- build
-before_script:
- # Update and use base build deps
- - dnf -y update
- - dnf -y install 'dnf-command(builddep)' dbus-daemon
- - dnf -y builddep gnome-software
+cache:
+ paths:
+ - _ccache/
+
+variables:
+ MESON_TEST_TIMEOUT_MULTIPLIER: 2
+ G_MESSAGES_DEBUG: all
+ MESON_COMMON_OPTIONS: "--buildtype debug --prefix /usr"
+
+fedora-x86_64:
+ image: registry.gitlab.gnome.org/gnome/gnome-software/fedora:v1
+ stage: build
+ except:
+ - tags
+ script:
+ - meson ${MESON_COMMON_OPTIONS}
+ -Drpm_ostree=true
+ -Dmalcontent=false
+ _build
+ - ninja -C _build
+ - .gitlab-ci/run-tests.sh
+ artifacts:
+ reports:
+ junit: "_build/${CI_JOB_NAME}-report.xml"
+ name: "gnome-software-${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}"
+ when: always
+ paths:
+ - "_build/config.h"
+ - "_build/meson-logs"
+ - "_build/${CI_JOB_NAME}-report.xml"
-build-gnome-software:
+debian-stable-x86_64:
+ image: registry.gitlab.gnome.org/gnome/gnome-software/debian-stable:v1
stage: build
+ except:
+ - tags
script:
- - meson -Drpm_ostree=true -Dmalcontent=false _build .
- - ninja -v -C _build
- - mkdir -p /run/dbus
- - mkdir -p /var
- - ln -s /var/run /run
- - dbus-daemon --system --fork
- - /usr/lib/polkit-1/polkitd -n &
- - /usr/libexec/fwupd/fwupd -v &
- - meson test -v -C _build gs-self-test-lib
- - meson test -v -C _build gs-self-test-core
- - meson test -v -C _build gs-self-test-dpkg
- - meson test -v -C _build gs-self-test-dummy
- - meson test -v -C _build gs-self-test-epiphany
- - meson test -v -C _build gs-self-test-fwupd
- - meson test -v -C _build gs-self-test-modalias
- - meson test -v -C _build gs-self-test-repos
- - meson test -v -C _build gs-self-test-src
- - meson test -v -C _build gs-self-test-shell-extensions
- - meson test -v -C _build gs-self-test-flatpak
- - meson test -v -C _build gs-self-test-fedora-langpacks
+ - meson ${MESON_COMMON_OPTIONS}
+ -Drpm_ostree=false
+ -Dmalcontent=false
+ _build
+ - ninja -C _build
+ - .gitlab-ci/run-tests.sh
+ --no-suite fedora-langpacks
+ artifacts:
+ reports:
+ junit: "_build/${CI_JOB_NAME}-report.xml"
+ name: "gnome-software-${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}"
+ when: always
+ paths:
+ - "_build/config.h"
+ - "_build/meson-logs"
+ - "_build/${CI_JOB_NAME}-report.xml"
diff --git a/.gitlab-ci/README.md b/.gitlab-ci/README.md
new file mode 100644
index 00000000..5821c0d6
--- /dev/null
+++ b/.gitlab-ci/README.md
@@ -0,0 +1,22 @@
+# CI support stuff
+
+## Docker image
+
+GitLab CI jobs run in a Docker image, defined here. To update that image
+(perhaps to install some more packages):
+
+1. Edit `.gitlab-ci/Dockerfile` with the changes you want
+1. Run `.gitlab-ci/run-docker.sh build --base=debian-stable --base-version=1` to
+ build the new image (bump the version from the latest listed for that `base`
+ on https://gitlab.gnome.org/GNOME/gnome-software/container_registry)
+1. Run `.gitlab-ci/run-docker.sh push --base=debian-stable --base-version=1` to
+ upload the new image to the GNOME GitLab Docker registry
+ * If this is the first time you're doing this, you'll need to log into the
+ registry
+ * If you use 2-factor authentication on your GNOME GitLab account, you'll
+ need to [create a personal access token][pat] and use that rather than
+ your normal password
+1. Edit `.gitlab-ci.yml` (in the root of this repository) to use your new
+ image
+
+[pat]: https://gitlab.gnome.org/profile/personal_access_tokens
diff --git a/.gitlab-ci/debian-stable.Dockerfile b/.gitlab-ci/debian-stable.Dockerfile
new file mode 100644
index 00000000..7ea1fd93
--- /dev/null
+++ b/.gitlab-ci/debian-stable.Dockerfile
@@ -0,0 +1,58 @@
+FROM debian:buster
+
+RUN apt-get update -qq && apt-get install --no-install-recommends -qq -y \
+ appstream \
+ appstream-util \
+ clang \
+ clang-tools-7 \
+ dbus \
+ desktop-file-utils \
+ docbook-xsl \
+ gcc \
+ g++ \
+ gettext \
+ git \
+ gnome-pkg-tools \
+ gobject-introspection \
+ gsettings-desktop-schemas-dev \
+ gtk-doc-tools \
+ itstool \
+ lcov \
+ libappstream-glib-dev \
+ libflatpak-dev \
+ libfwupd-dev \
+ libgirepository1.0-dev \
+ libglib2.0-dev \
+ libgnome-desktop-3-dev \
+ libgoa-1.0-dev \
+ libgspell-1-dev \
+ libgtk-3-dev \
+ libgudev-1.0-dev \
+ libjson-glib-dev \
+ libpackagekit-glib2-dev \
+ libpolkit-gobject-1-dev \
+ libsoup2.4-dev \
+ libxmlb-dev \
+ libxml2-utils \
+ ninja-build \
+ packagekit \
+ pkg-config \
+ python3 \
+ python3-pip \
+ python3-setuptools \
+ python3-wheel \
+ shared-mime-info \
+ unzip \
+ valgrind \
+ wget \
+ xsltproc \
+ xz-utils \
+ && rm -rf /usr/share/doc/* /usr/share/man/*
+
+RUN pip3 install meson==0.47.0
+
+ARG HOST_USER_ID=5555
+ENV HOST_USER_ID ${HOST_USER_ID}
+RUN useradd -u $HOST_USER_ID -ms /bin/bash user
+
+ENV LANG=C.UTF-8 LANGUAGE=C.UTF-8 LC_ALL=C.UTF-8
diff --git a/.gitlab-ci/fedora.Dockerfile b/.gitlab-ci/fedora.Dockerfile
new file mode 100644
index 00000000..231604ab
--- /dev/null
+++ b/.gitlab-ci/fedora.Dockerfile
@@ -0,0 +1,63 @@
+FROM fedora:31
+
+RUN dnf -y install \
+ clang \
+ clang-analyzer \
+ dbus-daemon \
+ dbus-devel \
+ desktop-file-utils \
+ docbook-style-xsl \
+ flatpak-devel \
+ fwupd-devel \
+ gcc \
+ gettext \
+ git \
+ glib2-devel \
+ gnome-desktop3-devel \
+ gnome-online-accounts-devel \
+ gobject-introspection \
+ gobject-introspection-devel \
+ gsettings-desktop-schemas-devel \
+ gspell-devel \
+ gtk-doc \
+ gtk3-devel \
+ json-glib-devel \
+ itstool \
+ lcov \
+ libappstream-glib-devel \
+ libdnf-devel \
+ libgudev-devel \
+ liboauth-devel \
+ libsecret-devel \
+ libsoup-devel \
+ libxmlb-devel \
+ libxslt \
+ NetworkManager-libnm-devel \
+ ninja-build \
+ ostree-devel \
+ PackageKit \
+ PackageKit-glib-devel \
+ pcre-devel \
+ polkit-devel \
+ python3 \
+ python3-pip \
+ python3-wheel \
+ rpm-devel \
+ rpm-ostree-devel \
+ shared-mime-info \
+ snapd-glib-devel \
+ unzip \
+ valgrind \
+ valgrind-devel \
+ wget \
+ xz \
+ zlib-devel \
+ && dnf clean all
+
+RUN pip3 install meson==0.47.0
+
+ARG HOST_USER_ID=5555
+ENV HOST_USER_ID ${HOST_USER_ID}
+RUN useradd -u $HOST_USER_ID -ms /bin/bash user
+
+ENV LANG C.UTF-8
diff --git a/.gitlab-ci/meson-junit-report.py b/.gitlab-ci/meson-junit-report.py
new file mode 100755
index 00000000..9b772cbb
--- /dev/null
+++ b/.gitlab-ci/meson-junit-report.py
@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+
+# Turns a Meson testlog.json file into a JUnit XML report
+#
+# Copyright 2019 GNOME Foundation
+#
+# SPDX-License-Identifier: LGPL-2.1-or-later
+#
+# Original author: Emmanuele Bassi
+
+import argparse
+import datetime
+import json
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+aparser = argparse.ArgumentParser(description='Turns a Meson test log into a JUnit report')
+aparser.add_argument('--project-name', metavar='NAME',
+ help='The project name',
+ default='unknown')
+aparser.add_argument('--job-id', metavar='ID',
+ help='The job ID for the report',
+ default='Unknown')
+aparser.add_argument('--branch', metavar='NAME',
+ help='Branch of the project being tested',
+ default='master')
+aparser.add_argument('--output', metavar='FILE',
+ help='The output file, stdout by default',
+ type=argparse.FileType('w', encoding='UTF-8'),
+ default=sys.stdout)
+aparser.add_argument('infile', metavar='FILE',
+ help='The input testlog.json, stdin by default',
+ type=argparse.FileType('r', encoding='UTF-8'),
+ default=sys.stdin)
+
+args = aparser.parse_args()
+
+outfile = args.output
+
+testsuites = ET.Element('testsuites')
+testsuites.set('id', '{}/{}'.format(args.job_id, args.branch))
+testsuites.set('package', args.project_name)
+testsuites.set('timestamp', datetime.datetime.utcnow().isoformat())
+
+suites = {}
+for line in args.infile:
+ data = json.loads(line)
+ (full_suite, unit_name) = data['name'].split(' / ')
+ try:
+ (project_name, suite_name) = full_suite.split(':')
+ except ValueError:
+ (project_name, suite_name) = (args.project_name, full_suite)
+
+ duration = data['duration']
+ return_code = data['returncode']
+ log = data['stdout']
+ log_stderr = data.get('stderr', '')
+
+ unit = {
+ 'suite': suite_name,
+ 'name': unit_name,
+ 'duration': duration,
+ 'returncode': return_code,
+ 'stdout': log,
+ 'stderr': log_stderr,
+ }
+
+ units = suites.setdefault(suite_name, [])
+ units.append(unit)
+
+for name, units in suites.items():
+ print('Processing suite {} (units: {})'.format(name, len(units)))
+
+ def if_failed(unit):
+ if unit['returncode'] != 0:
+ return True
+ return False
+
+ def if_succeded(unit):
+ if unit['returncode'] == 0:
+ return True
+ return False
+
+ successes = list(filter(if_succeded, units))
+ failures = list(filter(if_failed, units))
+ print(' - {}: {} pass, {} fail'.format(name, len(successes), len(failures)))
+
+ testsuite = ET.SubElement(testsuites, 'testsuite')
+ testsuite.set('name', '{}/{}'.format(args.project_name, name))
+ testsuite.set('tests', str(len(units)))
+ testsuite.set('errors', str(len(failures)))
+ testsuite.set('failures', str(len(failures)))
+
+ for unit in successes:
+ testcase = ET.SubElement(testsuite, 'testcase')
+ testcase.set('classname', '{}/{}'.format(args.project_name, unit['suite']))
+ testcase.set('name', unit['name'])
+ testcase.set('time', str(unit['duration']))
+
+ for unit in failures:
+ testcase = ET.SubElement(testsuite, 'testcase')
+ testcase.set('classname', '{}/{}'.format(args.project_name, unit['suite']))
+ testcase.set('name', unit['name'])
+ testcase.set('time', str(unit['duration']))
+
+ failure = ET.SubElement(testcase, 'failure')
+ failure.set('classname', '{}/{}'.format(args.project_name, unit['suite']))
+ failure.set('name', unit['name'])
+ failure.set('type', 'error')
+ failure.text = unit['stdout'] + '\n' + unit['stderr']
+
+output = ET.tostring(testsuites, encoding='unicode')
+outfile.write(output)
diff --git a/.gitlab-ci/run-docker.sh b/.gitlab-ci/run-docker.sh
new file mode 100755
index 00000000..7992a900
--- /dev/null
+++ b/.gitlab-ci/run-docker.sh
@@ -0,0 +1,120 @@
+#!/bin/bash
+
+read_arg() {
+ # $1 = arg name
+ # $2 = arg value
+ # $3 = arg parameter
+ local rematch='^[^=]*=(.*)$'
+ if [[ $2 =~ $rematch ]]; then
+ read "$1" <<< "${BASH_REMATCH[1]}"
+ else
+ read "$1" <<< "$3"
+ # There is no way to shift our callers args, so
+ # return 1 to indicate they should do it instead.
+ return 1
+ fi
+}
+
+set -e
+
+build=0
+run=0
+push=0
+list=0
+print_help=0
+no_login=0
+
+while (($# > 0)); do
+ case "${1%%=*}" in
+ build) build=1;;
+ run) run=1;;
+ push) push=1;;
+ list) list=1;;
+ help) print_help=1;;
+ --base|-b) read_arg base "$@" || shift;;
+ --base-version) read_arg base_version "$@" || shift;;
+ --no-login) no_login=1;;
+ *) echo -e "\e[1;31mERROR\e[0m: Unknown option '$1'"; exit 1;;
+ esac
+ shift
+done
+
+if [ $print_help == 1 ]; then
+ echo "$0 - Build and run Docker images"
+ echo ""
+ echo "Usage: $0 <command> [options] [basename]"
+ echo ""
+ echo "Available commands"
+ echo ""
+ echo " build --base=<BASENAME> - Build Docker image <BASENAME>.Dockerfile"
+ echo " run --base=<BASENAME> - Run Docker image <BASENAME>"
+ echo " push --base=<BASENAME> - Push Docker image <BASENAME> to the registry"
+ echo " list - List available images"
+ echo " help - This help message"
+ echo ""
+ exit 0
+fi
+
+cd "$(dirname "$0")"
+
+if [ $list == 1 ]; then
+ echo "Available Docker images:"
+ for f in *.Dockerfile; do
+ filename=$( basename -- "$f" )
+ basename="${filename%.*}"
+
+ echo -e " \e[1;39m$basename\e[0m"
+ done
+ exit 0
+fi
+
+# All commands after this require --base to be set
+if [ -z $base ]; then
+ echo "Usage: $0 <command>"
+ exit 1
+fi
+
+if [ ! -f "$base.Dockerfile" ]; then
+ echo -e "\e[1;31mERROR\e[0m: Dockerfile for '$base' not found"
+ exit 1
+fi
+
+if [ -z $base_version ]; then
+ base_version="latest"
+else
+ base_version="v$base_version"
+fi
+
+TAG="registry.gitlab.gnome.org/gnome/gnome-software/${base}:${base_version}"
+
+if [ $build == 1 ]; then
+ echo -e "\e[1;32mBUILDING\e[0m: ${base} as ${TAG}"
+ sudo docker build \
+ --build-arg HOST_USER_ID="$UID" \
+ --tag "${TAG}" \
+ --file "${base}.Dockerfile" .
+ exit $?
+fi
+
+if [ $push == 1 ]; then
+ echo -e "\e[1;32mPUSHING\e[0m: ${base} as ${TAG}"
+
+ if [ $no_login == 0 ]; then
+ sudo docker login registry.gitlab.gnome.org
+ fi
+
+ sudo docker push $TAG
+ exit $?
+fi
+
+if [ $run == 1 ]; then
+ echo -e "\e[1;32mRUNNING\e[0m: ${base} as ${TAG}"
+ sudo docker run \
+ --rm \
+ --volume "$(pwd)/..:/home/user/app" \
+ --workdir "/home/user/app" \
+ --tty \
+ --interactive "${TAG}" \
+ bash
+ exit $?
+fi
diff --git a/.gitlab-ci/run-tests.sh b/.gitlab-ci/run-tests.sh
new file mode 100755
index 00000000..89226045
--- /dev/null
+++ b/.gitlab-ci/run-tests.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+set +e
+
+case "$1" in
+ --log-file)
+ log_file="$2"
+ shift
+ shift
+ ;;
+ *)
+ log_file="_build/meson-logs/testlog.json"
+esac
+
+# FIXME: The tests need to be run as root
+if ! [ $(id -u) = 0 ]; then
+ echo "Tests need to be run as root"
+ exit 1
+fi
+
+meson test \
+ -C _build \
+ --timeout-multiplier ${MESON_TEST_TIMEOUT_MULTIPLIER} \
+ --no-suite flaky \
+ "$@"
+
+exit_code=$?
+
+python3 .gitlab-ci/meson-junit-report.py \
+ --project-name=gnome-software \
+ --job-id "${CI_JOB_NAME}" \
+ --output "_build/${CI_JOB_NAME}-report.xml" \
+ "${log_file}"
+
+exit $exit_code
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]
