On-call sysadmin, new emergency procedures (was: HOW TO: a new way for managing the DNS zone files)

[Patrick Uiterwijk <puiterwijk gnome org>]

Hi Sysadmins,

I want to give some info on another recent improvement: we (so far me and Andrea) have setup an schedule for 
on-call sysadmins.
This means that any email sent to the emergency RT queue (emergency gnome org) or notice sent by Nagios gets 
sent to the sysadmin on duty, who is responsible for looking into the issue.

In case the on-call sysadmin does not respond in 30 minutes by acknowlodging the problem, it is automatically 
escelated to the other sysadmin.
In case anyone wants to offer himself as another on-call sysadmin, just let me know.

Patrick Uiterwijk


----- Original Message -----
> Hi Sysadmins,
>
> we recently introduced DNSSEC on gnome.org's tree (we'll be slowly moving
> all the other important domains like guadec.org to it) and we've just
> updated the guidelines to properly manage the DNS zone file.
>
> I made a wiki page for this which is available at [1], please follow all
> the instructions carefully and eventually ask if unsure about something.
>
> As a side note I did start introducing the SSHFP DNS field to properly
> check if a specific host SSH fingerprint is the one you should be
> connecting to and not the wrong one in case of a MITM attack.
>
> An example:
>
> ;; ANSWER SECTION:
> git.gnome.org. 900 IN SSHFP 1 1 7CCC918309F2724D444E7FBE3E19901AF6F56BA9
>
> The above is what it's stored on our DNS server, checking if my known_hosts
> file has the right value can be done this way:
>
> ssh -oVerifyHostKeyDNS=yes -v git.gnome.org (or {master, webapps2}.gnome.org
> )
>
> The result should be something like:
>
> debug1: Server host key: RSA 00:39:fd:1a:a4:2c:6b:28:b8:2e:95:31:c2:90:72:03
> debug1: matching host key fingerprint found in DNS
>
> There are also a few news about emergency gnome org and the Pagerduty setup
> we just finalized on Nagios / Request Tracker. Patrick will mail the list
> later today with more details about that given he personally set it up.
>
> Have an awesome day!
>
> [1]
> https://wiki.gnome.org/Sysadmin/DNSZoneUpdates<https://wiki.gnome.org/Sysadmin/DNSZoneUpdates#preview>
>
> Cheers,
>
> Andrea
>
> Debian Developer,
> Fedora / EPEL packager,
> GNOME Sysadmin,
> GNOME Foundation Membership & Elections Committee Chairman
>
> Homepage: http://www.gnome.org/~av
>
> _______________________________________________
> gnome-infrastructure mailing list
> gnome-infrastructure gnome org
> https://mail.gnome.org/mailman/listinfo/gnome-infrastructure