[pitivi/thiblahute/pre_commit_ci: 1/3] docs: How to maintain the sandbox and the hooks

[Alexandru Băluț <alexbalut src gnome org>]

commit 5a89408195966306fb4a599a710fd0aa4d21905f
Author: Alexandru Băluț <alexandru balut gmail com>
Date:   Mon Jan 11 23:33:51 2021 +0100

    docs: How to maintain the sandbox and the hooks

 docs/How_to_update_SDK.md            |  2 +-
 docs/Precommit_hooks.md              | 77 ++++++++++++++++++++++++++++++++++++
 docs/Updating_Python_dependencies.md | 69 ++++++++++++++++++++++++++++++++
 docs/release.md                      |  2 +-
 docs/sitemap.txt                     |  2 +
 5 files changed, 150 insertions(+), 2 deletions(-)
---
diff --git a/docs/How_to_update_SDK.md b/docs/How_to_update_SDK.md
index d6feb8232..3e9632648 100644
--- a/docs/How_to_update_SDK.md
+++ b/docs/How_to_update_SDK.md
@@ -2,7 +2,7 @@
 short-description: How to update the SDK used by the Pitivi development environment
 ...
 
-# How to update the SDK
+# Updating the SDK
 
 We use the same [Flatpak
 manifest](https://gitlab.gnome.org/GNOME/pitivi/blob/master/build/flatpak/org.pitivi.Pitivi.json)
diff --git a/docs/Precommit_hooks.md b/docs/Precommit_hooks.md
new file mode 100644
index 000000000..70f7ee081
--- /dev/null
+++ b/docs/Precommit_hooks.md
@@ -0,0 +1,77 @@
+---
+short-description: How to update the Git pre-commit hooks which sanitize the code
+...
+
+# pre-commit hooks
+
+When you enter the [development environment](HACKING.md), our `pre-commit.hook`
+[script](https://gitlab.gnome.org/GNOME/pitivi/-/blob/master/pre-commit.hook) is
+symlinked in your local git repository as `.git/hooks/pre-commit`, serving as
+the [Git pre-commit
+hook](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks).
+
+When you make a git commit, the following happens:
+
+- The `pre-commit.hook` script runs the (confusingly named) `pre-commit` tool
+**in the flatpak sandbox**.
+
+- The `pre-commit` tool runs the hooks configured in
+[.pre-commit-config.yaml](https://gitlab.gnome.org/GNOME/pitivi/-/blob/master/.pre-commit-config.yaml)
+
+Sooner or later you'll want to update the `pre-commit` framework and its hooks.
+
+## How to update the `pre-commit` tool
+
+The `pre-commit` tool is installed in the flatpak sandbox through
+[python3-pre-commit.json](https://gitlab.gnome.org/GNOME/pitivi/-/blob/master/build/flatpak/python3-pre-commit.json).
+The `python3-pre-commit.json` file is generated with `flatpak-pip-generator`.
+See the instructions for [updating the Python
+dependencies](Updating_Python_dependencies.md).
+
+## How to update the hooks
+
+The `.pre-commit-config.yaml` file contains the list of hooks we use, grouped by
+the git repo in which they can be found. The repos are downloaded and cached by
+the `pre-commit` tool on demand. These hooks are executed **each in their own
+own virtualenv**.
+
+A special case is the `local` repo which groups the hooks installed alongside
+the pre-commit framework, in the sandbox. We only need the `pylint` hook to be
+run **in the sandbox** instead of in its own virtualenv, such that it has access
+to the Python environment in the sandbox.
+
+To update `pylint`, follow the instructions for [updating the Python
+dependencies](Updating_Python_dependencies.md).
+
+To update the regular hooks, take the repos one by one, check what is the latest
+version and update the entry in the `.pre-commit-config.yaml` file. After
+updating each repo, validate the entire codebase and fix all the newfound
+errors.
+
+For example, suppose we just updated the
+`https://github.com/pre-commit/pre-commit-hooks.git` repo to revision `v3.4.0`:
+
+```
+  - repo: https://github.com/pre-commit/pre-commit-hooks.git
+    rev: v3.4.0
+    hooks:
+      - id: check-yaml
+```
+
+To run the `check-yaml` hook on the entire codebase:
+
+```
+(ptv-flatpak) $ ptvenv pre-commit run -a check-yaml
+```
+
+If you feel brave, you can use `pre-commit` itself to update the hooks:
+
+```
+(ptv-flatpak) $ ptvenv pre-commit autoupdate
+```
+
+At the end, do a final check by running all the hooks on the entire codebase:
+
+```
+(ptv-flatpak) $ ptvenv pre-commit run -a
+```
diff --git a/docs/Updating_Python_dependencies.md b/docs/Updating_Python_dependencies.md
new file mode 100644
index 000000000..3da7b2a4b
--- /dev/null
+++ b/docs/Updating_Python_dependencies.md
@@ -0,0 +1,69 @@
+---
+short-description: How to update the Python dependencies
+...
+
+# Python dependencies
+
+Pitivi has a single Python runtime dependency `matplotlib` but in the
+development sandbox we install quite a few Python tools.
+
+Each Python package has its own `python3-*.json` file in
+[build/flatpak](https://gitlab.gnome.org/GNOME/pitivi/-/tree/master/build/flatpak),
+generated with
+[flatpak-pip-generator](https://github.com/aleb/flatpak-builder-tools/tree/master/pip):
+
+```
+$ git clone git github com:aleb/flatpak-builder-tools.git
+```
+
+Note, we use a custom `flatpak-pip-generator` which downloads the compiled
+Python packages. The upstream version downloads only the source packages, for
+security reasons.
+
+The `hotdoc` and `matplotlib` packages are quite complex, needing additional
+build dependencies which have to be installed beforehand. Jan 2021 it's much
+easier to install the compiled packages instead of the source packages. When
+upstream makes it easier to use source packages we should switch.
+
+## Updating matplotlib
+
+```
+$ cd build/flatpak
+$ python3 flatpak-pip-generator --runtime org.gnome.Sdk/x86_64/3.38 matplotlib
+```
+
+## Updating the development tools
+
+```
+$ cd build/flatpak
+$ python3 flatpak-pip-generator --runtime org.gnome.Sdk/x86_64/3.38 ipdb
+$ python3 flatpak-pip-generator --runtime org.gnome.Sdk/x86_64/3.38 nose2
+$ python3 flatpak-pip-generator --runtime org.gnome.Sdk/x86_64/3.38 nose setuptools_git setuptools_pep8 
sphinx hotdoc
+```
+
+## Updating the pre-commit framework
+
+```
+$ cd build/flatpak
+$ python3 flatpak-pip-generator --runtime org.gnome.Sdk/x86_64/3.38 pre-commit
+$ python3 flatpak-pip-generator --runtime org.gnome.Sdk/x86_64/3.38 pylint
+```
+
+## Updating your local sandbox
+
+After you update the json files, try out the build by updating your local
+development environment:
+
+```
+$ . bin/pitivi-env
+(ptv-flatpak) $ ptvenv --update
+```
+
+If errors happen, you can inspect the SDK by launching `bash` in a sandbox, for
+example:
+
+```
+$ flatpak run --user --command=bash --devel org.gnome.Sdk/x86_64/3.38
+[📦 org.gnome.Sdk ~]$ python --version
+Python 3.8.6
+```
diff --git a/docs/release.md b/docs/release.md
index 4ce22ad3f..59fcc6db2 100644
--- a/docs/release.md
+++ b/docs/release.md
@@ -2,7 +2,7 @@
 short-description: How to make a new release
 ...
 
-# How to make a release
+# Making a release
 
 Ideally these instructions are in line with the [GNOME releasing
 process](https://live.gnome.org/MaintainersCorner/Releasing).
diff --git a/docs/sitemap.txt b/docs/sitemap.txt
index 8da93332b..650d6994d 100644
--- a/docs/sitemap.txt
+++ b/docs/sitemap.txt
@@ -26,6 +26,8 @@ index.md
        Administrative.md
                release.md
                How_to_update_SDK.md
+               Updating_Python_dependencies.md
+               Precommit_hooks.md
        Praise.md
        Project_history.md
        Current_events.md