Re: GNOME Bugzilla Upgrade: Test Upgrade On Friday?

[Owen Taylor <otaylor redhat com>]

On Tue, 2009-07-28 at 14:40 -0700, Max Kanat-Alexander wrote:
> Owen Taylor wrote:
> > We do actually have two much newer and nicer machines that are not at
> > capacity:
>  > [snip]
> 
> 	Oh yeah! Wow, those sound great!
> 
> > vbox.gnome.org is currently hosting (virtualized) git.gnome.org and
> > puppet.gnome.org; git currently has 20GB allocated to it, but that was
> > just some conversion tasks, it doesn't need nearly that much. We could
> > add an additional VM for something that was CPU/memory intensive but
> > without much IO load.
> 
> 	That's exactly what a Bugzilla web head is--CPU/memory intensive 
> without much I/O. Of course, if the DB goes onto a separate server, 
> there should ideally be a 1GB connection between the machines or 
> better--I assume that's the case between vbox and drawable?

I'm pretty sure yes - they are new machines racked next to each other
and connected to the same new switch.

> >  Do we want to have a single big "db.gnome.org" and host all the GNOME
> >  database needs on it, or do we want a VM specialized for bugzilla,
> >  and later add separate VM's for other database tasks.
> 
> 	I suspect that performance-wise, it'd be best to just have a single 
> enormous db.gnome.org. I've yet to see high-end database performance out 
> of a VM, and high-end performance is ideally what we'd like to see for 
> Bugzilla (not because it does thousands of operations per second, but 
> because Bugzilla's Search functionality is very intensive). 

The disk configuration on drawable.gnome.org wouldn't count as high-end
by any means - well, except by comparison to the current
button.gnome.org. So, I'm not sure it would be that limited by
virtualization. But certainly, if nothing else, it's easier to
analyze and think about performance without a virtualization layer.

I think our main method of getting high performance is going to have
almost everything in memory - the entire bugzilla.gnome.org database
after 9 years and 0.5M bugs is still only 8GB.

> An ideal 
> situation is for there to be two database servers--one master and one 
> slave (both for backup reasons and for performance reasons--Bugzilla can 
> do read-intensive operations on the slave), but again, I think the 
> performance penalty of having them in a VM might outweigh the advantage 
> of having two separate machines.

Presumably for the table-lock-avoidance purposes of shadowdb you could
just have two databases on the same MySQL instance.

That does double the memory requirements to get everything into memory,
but 2*8GB is still considerably less than 32GB.

(So given only one machine the question becomes: is it better to do that
with MyISAM tables, or to use InnoDB tables that have row locks and
shouldn't need shadowdb. But have other overheads and might suffer from
bugzilla not demarcating transactions.)

Reasons we might want to go with VMs anyways:

 - Allow having multiple mysqld instances, with different tuning
   parameters, and separately configured pools for the different
   applications.

   It is very possible to run multiple mysqld instances on the same
   machine/VM but then you can't use the standard config files,
   'service mysqld start/stop', and so forth.

 - Slightly easier to move things around between machines in the future

 - Some conceivable improvement in security. Though if the bugzilla
   user is restricted to access to the bugzilla database it's not
   a big difference.

 - We want "almost everything" to be VMs in the future. (IIRC,
   fedoraproject.org has everything but database servers virtualized.)

I think my basic instinct is the same as yours - to just set up drawable
directly as a single big mysqld server instead of virtualizing within
it. I'd certainly like input from others. I'm neither a DB expert not a
virtualization expert, and I'm hoping someone else will do most of the
setup anyways :-)

> > That's all we'd really need to know to allocate an IP and do the initial
> > system image install. Then for actually doing the setup there would be
> > additional information needed - e.g., InnoDB vs. MyISAM.
> 
> 	I'll see if I can get Mozilla's MySQL configuration that they use for 
> their current Bugzilla servers, for that last bit.

That would be useful. 

- Owen