[gimp-web] devel-docs: update the mirror procedure.

From: Jehan <jehanp src gnome org>
To: commits-list gnome org
Cc:
Subject: [gimp-web] devel-docs: update the mirror procedure.
Date: Tue, 30 Aug 2022 12:00:24 +0000 (UTC)
commit a3cf4459b364e75c410d87262aedacdee316d2e0
Author: Jehan <jehan girinstud io>
Date:   Tue Aug 30 13:59:48 2022 +0200

    devel-docs: update the mirror procedure.

 devel-docs/mirror-howto.md | 89 +++++++++++++++++++++++++++++++---------------
 1 file changed, 61 insertions(+), 28 deletions(-)
---
diff --git a/devel-docs/mirror-howto.md b/devel-docs/mirror-howto.md
index 3a722bc9..6ca3756a 100644
--- a/devel-docs/mirror-howto.md
+++ b/devel-docs/mirror-howto.md
@@ -2,17 +2,19 @@
 
 ## Architecture
 
-Mirrors are synced from the static download server `download.gimp.org`
-(a.k.a. `master.gnome.org`).
-Current infrastructure is using
-[Mirrorbits](https://github.com/etix/mirrorbits) which handles mirror
+Mirrors are synced from `download.gimp.org` (hosted on
+`master.gnome.org`). Current mirroring infrastructure is using
+[Mirrorbits](https://github.com/etix/mirrorbits) which handles
 geoIP-based redirection, health check (and removing down mirrors from
 rotation automatically), security checks and more for us.
 
-We have some redirection still available as [Apache rewrite
-rules](https://gitlab.gnome.org/Infrastructure/puppet/-/blob/0df77787596314f41de55b270e016cfd99ce2a53/modules/httpd/files/sites.d/download.gimp.org.conf#L65-82)
-so that old mirrored URIs (starting with `/mirror/`), from the time we
-had an Apache-based round-robin rotation, now just redirect to root.
+Note that to we still have some redirection from `/pub/` and
+`/mirror/pub/` to `/` in order for old URLs to stay alive (cf. the old
+basic round-robin mirroring (based on [Apache rewrite
+rules](https://gitlab.gnome.org/Infrastructure/puppet/-/blob/0df77787596314f41de55b270e016cfd99ce2a53/modules/httpd/files/sites.d/download.gimp.org.conf#L65-82)).
+Now the download root is just the old `/pub/`, yet it is important to
+keep the redirections forever because the web is full of legacy GIMP
+download URLs.
 
 The list of mirrors in rotation is found at: `/etc/httpd/download.gimp.org.map`
 
@@ -34,10 +36,19 @@ follow is:
 * Create a mirror request on the [Infrastructure 
tracker](https://gitlab.gnome.org/Infrastructure/Infrastructure/issues/new?issuable_template=new-mirror)
 * Write in title and introduction that you want to be a mirror for GIMP in particular
 * Add `/cc @Jehan` in the request description or in a comment
-* Fill the relevant fields
-* If you want a specific name and URL to be shown in the mirror sponsor list, please write it explicitly in 
the report (otherwise we will choose whatever organization name and URL seems the most relevant from what you 
wrote down)
-* Once GNOME is done verifying your organization, they will send you rsync credentials for syncing your 
mirror
-* There is nothing to do in particular to appear on the Sponsors page which will be updated regularly 
through scripts. Yet it may take a few days or even weeks at times. So don’t worry if your organization name 
does not appear immediately!
+* Fill the relevant fields, and add in particular: the final `https`
+  mirror URL, as well as either `rsync` or `ftp` public access to your
+  mirror.
+* If you want a specific name and URL to be shown in the mirror sponsor
+  list, please write it explicitly in the report (otherwise we will
+  choose whatever organization name and URL seems the most relevant from
+  what you wrote down)
+* Once we are done verifying your organization, we will send you rsync
+  credentials for syncing your mirror
+* There is nothing to do in particular to appear on the Sponsors page
+  which will be updated regularly through scripts. Yet it may take a few
+  days or even weeks at times. So don’t worry if your organization name
+  does not appear immediately!
 
 Note: a web URL will soon be available giving this procedure.
 
@@ -56,20 +67,22 @@ instance @Jehan can take care of part of the actions:
      verify they are the really linked (through `whois` or other means)
 2. Verify that the `https` URL has no major issue (our redirect happens
    in https-only so a working https URL is mandatory)
-3. If 1. and 2. are fine, generate an user and a password (e.g. with
+3. Test the given `rsync` server with `rsync -avz rsync://example.org`
+   or the FTP server with a FTP client.
+4. If 1. to 3. are fine, generate an user and a password (e.g. with
    `pwgen`) and add them to `/etc/rsyncd/secrets` in
-   `download.gimp.org`. This is done by directly editing the file.
+   `master.gnome.org`. This is done by directly editing the file.
    Do not forget to leave a comment as well for reference (possibly a
    Gitlab report link).
    [*requires to be in proper admin group*]
-4. Once this is done, edit
+5. Once this is done, edit
    
[/etc/rsyncd.conf](https://gitlab.gnome.org/Infrastructure/puppet/-/blob/master/modules/ftpadmin/files/rsyncd.conf)
    by creating a Merge Request to
    [Infrastructure/puppet](https://gitlab.gnome.org/Infrastructure/puppet)
    (not editing it directly on the server, unlike 3.).
-5. Then once the MR is merged, ask the mirror admin for their public GPG
+6. Then once the MR is merged, ask the mirror admin for their public GPG
    key.
-6. When they return their key, send the rsync credentials, encrypted
+7. When they return their key, send the rsync credentials, encrypted
    with this key, then signed by yours (which should be on a public key
    server for non-tampering verification) by email, and ask them to
    notify when the mirror is properly set-up and synced. Wait for their
@@ -103,40 +116,60 @@ instance @Jehan can take care of part of the actions:
    (provided by their organization) in https. What should be avoided is
    exchanging credentials in plain text over the internet.
 
-7. Once they notify you that the sync is complete, land the mirror on
-   the map file by editing `/etc/httpd/download.gimp.org.map` directly
-   on the server.
-   [*requires to be in proper admin group*]
-8. Close the report.
-9. on `gimp-web` repository, run:
+8. Once they notify you that the sync is complete, land the mirror this
+   way:
+
+   * Connect to https://console-openshift-console.apps.openshift4.gnome.org/
+   * Click your name on top-right corner, then "Copy login command"
+   * Copy the given command on terminal (assuming `oc` CLI tool is installed):
+     `oc login --token=sha256~some-token --server=https://api.openshift4.gnome.org:6443`
+   * Now `oc get pods --selector app=mirrorbits` will give you a list of
+     pods. Take any from the list and run: `oc rsh -t mirrorbits-<random-stuff>`
+   * Now you are communicating with mirrorbits (and have a new prompt).
+     The command `mb help` gives you a list of commands. For instance
+     `mb list` returns the mirrors and their status.
+   * To add a mirror, look at `mb add -help`. For instance with ftp
+     scanning:
+```
+mb add -http https://example.org/mirrors/gimp/ -ftp ftp://example.org/mirrors/gimp/ example.org
+mb enable cc.uoc.gr
+```
+     Or with rsync:
+```
+mb add -http https://example.org/mirrors/gimp/ -rsync rsync://example.org/gimp example.org
+mb enable lysator.liu.se
+```
+9. If all went well (i.e. that `mb list` should eventually tell you that
+   the mirror is "up"), close the report.
+10. on `gimp-web` repository, run:
 
    ```sh
    tools/downloads/update-mirrors.py --ssh-user <yourlogin>
    ```
    [*requires to have simple access to the server*]
-10. The script will update `tools/downloads/downloads.http.txt`
+11. The script will update `tools/downloads/downloads.http.txt`
    automatically and should tell you that
    `content/downloads/mirrors.json` has to be updated too. Do so by
    writing the public name of the mirror organization, link, location
    and other data. In "more", also add the report link for reference.
    If other data need to be updated, do so as well (for instance if
    other mirrors changed).
-11. Run again:
+12. Run again:
 
    ```sh
    tools/downloads/update-mirrors.py --ssh-user <yourlogin>
    ```
    This time, it should tell you everything is fine.
 
-12. Verify all mirrors (especially new ones) are well synced at least
+13. Verify all mirrors (especially new ones) are well synced at least
    for the last release:
 
    ```sh
    tools/downloads/gimp-check-mirrors.py
    ```
-13. Commit all the changes and push them to `gimp-web`'s `testing`
+14. Commit all the changes and push them to `gimp-web`'s `testing`
    branch.
-14. After a short time, make sure that testing's [sponsor 
page](https://testing.gimp.org/donating/sponsors.html)
+15. After a short time, make sure that testing's [sponsor 
page](https://testing.gimp.org/donating/sponsors.html)
    is properly updated.
 
 Of course, the public website will be updated when you merge `testing`
[Date Prev][Date Next] [Thread Prev][Thread Next] [Thread Index] [Date Index] [Author Index]