Re: Build gnome org is down

[Jeff Schroeder <jeffschroeder computer org>]

On Thu, Sep 30, 2010 at 6:35 AM, Ray Wang <raywang gnome org> wrote:
> I cannot ssh into fixed.g.o as well.
> seems like sshd was down?

The builders generally cause the box to swap heavily. During heavy
memory pressure, it is common for the kernel thread writing memory out
to swap, kswapd, to go into IO wait. At this point, just about
everything on the box will seemingly hang as the kernel puts it into
IO wait until the memory can be written to swap.

sshd is up, but completely unresponsive due to this swap issue.
$ nc -v -z fixed.gnome.org 22
Connection to fixed.gnome.org 22 port [tcp/ssh] succeeded!

It is commonly called swapdeath. The best 2 ways out of it are a sysrq
kill all tasks or a reboot. Putting these builders on RHEL6 vms will
be a welcome move forward.

> On Thu, Sep 30, 2010 at 9:02 PM, Olav Vitters <olav vitters nl> wrote:
>> On Thu, Sep 30, 2010 at 11:35:51AM +0200, Piñeiro wrote:
>>> From: Olav Vitters <olav vitters nl>
>>>
>>> > On Thu, Sep 30, 2010 at 10:19:33AM +0200, Piñeiro wrote:
>>> >> subject is clear: b.g.o is down.
>>> >>
>>> >> As usual in this cases, I tried to login on fixed.gnome.org (ssh
>>> >> apinheiro fixed gnome org) to check if the master and slave were
>>> >> running and restarted again.
>>> >>
>>> >> But, although I get ping response from fixed.gnome.org, I'm not able
>>> >> to enter in the machine. I don't reach the option to enter the
>>> >> password, or any error. Just nothing.
>>> >>
>>> >> So, what we should do? Contact sys-admin? Olav?
>>> >
>>> > Sysadmins. We have multiple people now.
>>>
>>> When I said sys-admin, I was thinking specifically on
>>> gnome-sysadmin gnome org  I added here your name as you are also
>>> syndicated to build-bridage list in order to ask that. So I will try
>>> it again:
>>>
>>> Olav, should I forward this mail to gnome-sysadming gnome org?
>>
>> I already cc'ed gnome-infrastructure. Basically the same thing (though
>> you need to subscribe and a public list.
>>
>>> > Lately issue is that it just uses too much memory (*many* leftover dbus
>>> > processes)
>>>
>>> So the build process get stuck in the long term due this dbus
>>> processes? It is a fault of dbus or we would need to solve that on the
>>> gnome buildbot itself?
>>
>> jhbuilds results in dbus-* processes being left over. With RHEL6 you
>> could maybe do a cgroup and kill the whole process structure.
>>
>> --
>> Regards,
>> Olav
>> _______________________________________________
>> gnome-infrastructure mailing list
>> gnome-infrastructure gnome org
>> http://mail.gnome.org/mailman/listinfo/gnome-infrastructure
>>
>
>
>
> --
> Ray Wang
>   - Follow your dreams
> _______________________________________________
> gnome-infrastructure mailing list
> gnome-infrastructure gnome org
> http://mail.gnome.org/mailman/listinfo/gnome-infrastructure
>



-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com