Re: GNOME Foundation Budgeting - Infrastructure needs

[Jeff Schroeder <jeffschroeder computer org>]

On Thu, Sep 2, 2010 at 3:42 PM, Paul Cutler <pcutler gnome org> wrote:
> Hi,
>
> I wanted to follow-up quickly on a discussion we were just having IRC.
> Germán recently sent an email to foundation-list asking teams for
> feedback if they need any funds budgeted for next fiscal year which
> starts in October.
>
> Thanks to Jeff's donation of Combobox earlier this year we have a shiny
> new server we need to utilize more.  With the issues with Label today, I
> would like to ask a couple questions:
>
> * What would we like to migrate to Combobox?

Well originally, I wanted to buy a new database server. Drawable
appeared and killed that idea. After talking to owen it was decided
that container would be a good box to retire. So... I bought combobox
with the idea of moving some vms to it and having it be the primary
storage for all of the gnome infrastructure in the PHX redhat
facility.

> * What server hardware (and configuration) would be optimal for the
> Foundation, considering most of our hardware is now out of support
> contracts?  (How many new servers hosting what services?)

1.) Ideally, another server with stats comparable to combobox for more
vms and builders.

For instance, fpeters's report generation scripts kill 1 of the boxes
(window or fixed can't remember which) regularly and eat all of the
memory. That box runs a few other services if I recall. I would be
great to give him a few decent vms to beat up all day for those tasks.
Having a few dedicated build boxes is always a good thing as well.
Additionallyu

2.) 1 lower end server that is still fast to run less resource hungry
but still important services.

To split up failure domains, it is wise to put nfs + ldap/dns on
separate servers so that if 1 server tanks, you don't lose everything.
Due to the chicken / egg problem, ldap, dns, and nfs should be on real
hardware and not on virtual machines. I originally bought combobox for
nfs. It would be nice to have another box for vms and misc projects
that need horsepower and then a lower end but still decent new box for
ldap/dns and things like perhaps mango and civicrm.

As you know, some projects such as blip (http://blip.paulcutler.org)
are great and might eventually have gnome hosted versions.
Additionally, snowy (for tomboy online), and the inevitable getting
things gnome web based version will be gnome hosted at some point. We
need the infrastructure to provide these services for our users. The
current setup is unsuitable for doing this well.

> * What would our backup plan be if we could only buy one server?

Well only buying 1 server is better than 0 but  1 really nice one and
1 lower end one would be perfect. The benefit outweights the downsides
in my professional opinion here.

The "backup plan" at a bare minimum would be to split ldap/dns to the
backup server from label and menubar respectively. This is my opinion
only and is up to the team ultimately. This would allow us to
potentially retire 2 out of support servers with a newer faster and
more energy efficient one. We could figure out the details should that
be the case.

> * What are the risks with the current setup?

Some of the drives in critical gnome servers are a bit flaky and the
hardware is out of support. If it breaks, we're screwed! Things like
git or any of the web services such as wiki could go down and be down
for long periods of time.

> That's off the top of my head.  Please discuss!  :)
>
> Paul

I thought (incorrectly) that in spending my own personal $$ to buy
combobox, the foundation would front the money to buy one of equal or
greater value. Perhaps I was a bit naive in thinking this without
clarification, but I assumed this was the case. Ping me _offline_ for
my reasoning on this if you've got questions why. We desperately need
at least 1 new server to retire some of the older hardware such as
label which is performing critical functions such as ldap master. If
that goes down, things like git checkings will stop. We are going to
be rolling out some changes fairly soon to make our ldap setup more
robust, but for now it is what it is.

TL;DNR

We need at very least 1 new server if we want to ensure gnome services
remain available at all times. Two would be ideal so we can separate
out critical services. When label recently went offline and spent a
looooong time coming up, we lost the following services:
    - jabber
    - live.gnome.org aka "the wiki"
    - piwik - which would cause all other gnome websites with piwik
embedded to load more slowly
    - ldap - which breaks most everything. It causes fun things like
preventing any git checkins
    - CiviCRM - which is critical for Stormy + Rosanna from my understanding
    - something I missed surely

Thats just a good recent example. Combobox is a Dell 1U server which
I'll have to recommend as they are better bang for the buck without
skimping quality. The ILO stuff isn't as schmexy as the HP or
Sun^WOracle servers, but the price point is right and the quality is
very good. They extended warranties are also well cheap.

[1] http://live.gnome.org/Sysadmin/Servers#combobox.gnome.org

-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com