Ovirt devel - May 2008 - [PATCH] make taskomatic.rb robust to database outages

Hi ovirt,

En route to transaction support, and being of the "if it ain't broke,
break it!" school of code spelunking, I noticed that taskomatic.rb
dies if one stops postgresql on the appliance.

So I offer this first attempt at making taskomatic.rb handle this
situation somewhat more gracefully.

Key points:
0) move 'require dutils' above the call to daemonize, so that the
script will Fail outright if a connection cannot be established before
daemonization (not sure about this one really, but I thought the
sysadmin doing 'service ovirt-taskomatic start' would like to know
immediately that it was failing).
1) use ruby exception handling and extremely rudimentary logging for
exceptions in Task.find section.
2) wrap the entire 'case task.action' block (and the following Task
state updates) in a begin/rescue. This is meant as a first step toward
isolating/handling the many exceptions that can be raised by these
statements.

I am VERY open to feedback/suggestions on this; in particular, the
logging (does anyone know off-hand how to log line numbers in this
type of situation?), and making the script fail to start with the init
script if the DB appears unavailable (maybe it should just go daemon
and keep polling for a connection while logging its error?).

Oh, and I use the 'database_connect' method from active_record_env.rb
for the case (not sure if it can be reached) where ActiveRecord shows
that it's not connected (note: this seems to mean something different
from "there's no currently-functioning connection to the db").

I've tested this with database halts before and during execution, and
also restarting database during execution. Works for me.

Thanks,
Steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: safe_taskomatic.patch
Type: text/x-patch
Size: 3013 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/ovirt-devel/attachments/20080508/bd95f19f/attachment.bin>

On Thu, May 8, 2008 at 11:01 PM, steve linabery <slinabery at gmail.com>
wrote:
> Hi ovirt,
>
> En route to transaction support, and being of the "if it ain't
broke,
> break it!" school of code spelunking, I noticed that taskomatic.rb
> dies if one stops postgresql on the appliance.
>
> So I offer this first attempt at making taskomatic.rb handle this
> situation somewhat more gracefully.
>
> Key points:
> 0) move 'require dutils' above the call to daemonize, so that the
> script will Fail outright if a connection cannot be established before
> daemonization (not sure about this one really, but I thought the
> sysadmin doing 'service ovirt-taskomatic start' would like to know
> immediately that it was failing).
> 1) use ruby exception handling and extremely rudimentary logging for
> exceptions in Task.find section.
> 2) wrap the entire 'case task.action' block (and the following Task
> state updates) in a begin/rescue. This is meant as a first step toward
> isolating/handling the many exceptions that can be raised by these
> statements.
>
> I am VERY open to feedback/suggestions on this; in particular, the
> logging (does anyone know off-hand how to log line numbers in this
> type of situation?), and making the script fail to start with the init
> script if the DB appears unavailable (maybe it should just go daemon
> and keep polling for a connection while logging its error?).
>
> Oh, and I use the 'database_connect' method from
active_record_env.rb
> for the case (not sure if it can be reached) where ActiveRecord shows
> that it's not connected (note: this seems to mean something different
> from "there's no currently-functioning connection to the
db").
>
> I've tested this with database halts before and during execution, and
> also restarting database during execution. Works for me.
>
> Thanks,
> Steve
>
Hmm, I see from the libvirt list that perhaps I should have called
this "RFC" rather than "Patch", since I don't really
mean it's ready
for commit, but rather I'm looking for some feedback on it.

So, feedback at will.

Thanks,
Steve

Hi ovirt,

Resurrecting this patch that I wrote before I started having trouble
with my appliance. I decided (in the absence of comments) that it was
better for script to become a daemon and connect at earliest
opportunity if DB connection failed at initial runtime (logging this
fact, of course).

See you all tomorrow.

Steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: safe_taskomatic.patch
Type: text/x-patch
Size: 3727 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/ovirt-devel/attachments/20080521/489dc7fb/attachment.bin>