Problems with jobqueue.pl falling over

[Owen Taylor <otaylor redhat com>]

So, after adding '-d' to the jobqueue.pl options on Max's advice, I was
able to figure out why the mail queue daemon has been falling over over
the last few days.

It's my fault (sort of).

The reason that the problem started showing up now, is that in order to
add links to reviews to outgoing mails for Splinter, I used the
 
 mailer-before_send

hook. (It's not ideal, but its the only hook available in Bugzilla 3.4
to affect outgoing mail, and I want to keep Splinter working with
Bugzilla 3.4 and not require new hook additions.)

Now, if anything in mailer-before_send uses Bugzilla->dbh, then a
database connection is going to be opened in jobqueue.pl and left open.
We had mysqld configured with:

 wait_timeout=820

(About 15 minutes), so if it the splinter hook didn't get run for 15
minutes, then the handle would be closed, and all the outgoing mail
would start producing errors like:

job failed: An error occurred processing hook mailer-before_send in extension
./extensions/splinter: DBD::mysql::st execute failed: MySQL server has
gone away [for Statement ... ]

As a temporary workaround for this I upped the wait timeout to 3600, 
and hopefully that will make the problem basically go away for us, but
it's not really the right fix. I have no real idea what the right fix is.

(Moving the Splinter code to a new hook that gets run when formatting
the mail rather than sending the mail would be a fix, but as above I'd
rather not do that to maintain compatibility with released Bugzilla.
And mailer-before_send hooks *should* be able to use Bugzilla->dbh,
it seems to me.)

Any ideas, Max?

- Owen