Ovirt devel - Feb 2009 - [PATCH server] Make taskomatic retry qpidd connection on failure.

This patch makes taskomatic keep trying to connect to qpidd in the
event of any kind of failure to do so or if the connection is lost
during runtime.

If others test this and it works out ok I'll do the same for
dbomatic.

Signed-off-by: Ian Main <imain at redhat.com>
---
 src/task-omatic/taskomatic.rb |   44 ++++++++++++++++++++++++++++++++++++----
 1 files changed, 39 insertions(+), 5 deletions(-)

diff --git a/src/task-omatic/taskomatic.rb b/src/task-omatic/taskomatic.rb
index e65db08..a5cf6ed 100755
--- a/src/task-omatic/taskomatic.rb
+++ b/src/task-omatic/taskomatic.rb
@@ -47,11 +47,7 @@ class TaskOmatic
     @nth_host = 0
 
     @session = Qpid::Qmf::Session.new()
-
-    server, port = get_srv('qpidd', 'tcp')
-    raise "Unable to determine qpid server from DNS SRV record" if
not server
-
-    @broker = @session.add_broker("amqp://#{server}:#{port}",
:mechanism => 'GSSAPI')
+    @broker = nil
 
     do_daemon = true
 
@@ -87,6 +83,42 @@ class TaskOmatic
       $stdout = lf
       $stderr = lf
     end
+
+    # this has to be after daemonizing now because it could take a LONG time to
+    # actually connect if qpidd isn't running yet etc.
+    qpid_ensure_connected
+
+  end
+
+  def qpid_ensure_connected()
+
+    return if @broker and @broker.connected?
+
+    sleepy = 2
+
+    while true do
+      begin
+        server, port = get_srv('qpidd', 'tcp')
+        raise "Unable to determine qpid server from DNS SRV record"
if not server
+
+        @broker = @session.add_broker("amqp://#{server}:#{port}",
:mechanism => 'GSSAPI')
+
+        # Connection succeeded, go about our business.
+        return
+      rescue Exception => msg
+        puts "Error connecting to qpidd: #{msg}"
+      end
+      sleep(sleepy)
+      sleepy *= 2
+      sleepy = 120 if sleepy > 120
+
+      begin
+        # Could also be a credentials problem?  Try to get them again..
+        get_credentials('qpidd')
+      rescue Exception => msg
+        puts "Error getting qpidd credentials: #{msg}"
+      end
+    end
   end
 
   def find_capable_host(db_vm)
@@ -755,6 +787,8 @@ class TaskOmatic
         end
       end
 
+      qpid_ensure_connected
+
       tasks.each do |task|
 
         task.time_started = Time.now
-- 
1.6.0.6

No error on startup, so ACK.  Of course more needs to be done to make sure
to log useful events, but as discussed, that will be in a separate patch.

On Mon, Feb 16, 2009 at 2:48 PM, Ian Main <imain at redhat.com> wrote:
> This patch makes taskomatic keep trying to connect to qpidd in the
> event of any kind of failure to do so or if the connection is lost
> during runtime.
>
> If others test this and it works out ok I'll do the same for
> dbomatic.
>
> Signed-off-by: Ian Main <imain at redhat.com>
> ---
>  src/task-omatic/taskomatic.rb |   44
> ++++++++++++++++++++++++++++++++++++----
>  1 files changed, 39 insertions(+), 5 deletions(-)
>
> diff --git a/src/task-omatic/taskomatic.rb b/src/task-omatic/taskomatic.rb
> index e65db08..a5cf6ed 100755
> --- a/src/task-omatic/taskomatic.rb
> +++ b/src/task-omatic/taskomatic.rb
> @@ -47,11 +47,7 @@ class TaskOmatic
>     @nth_host = 0
>
>     @session = Qpid::Qmf::Session.new()
> -
> -    server, port = get_srv('qpidd', 'tcp')
> -    raise "Unable to determine qpid server from DNS SRV record"
if not
> server
> -
> -    @broker = @session.add_broker("amqp://#{server}:#{port}",
:mechanism
> => 'GSSAPI')
> +    @broker = nil
>
>     do_daemon = true
>
> @@ -87,6 +83,42 @@ class TaskOmatic
>       $stdout = lf
>       $stderr = lf
>     end
> +
> +    # this has to be after daemonizing now because it could take a LONG
> time to
> +    # actually connect if qpidd isn't running yet etc.
> +    qpid_ensure_connected
> +
> +  end
> +
> +  def qpid_ensure_connected()
> +
> +    return if @broker and @broker.connected?
> +
> +    sleepy = 2
> +
> +    while true do
> +      begin
> +        server, port = get_srv('qpidd', 'tcp')
> +        raise "Unable to determine qpid server from DNS SRV
record" if not
> server
> +
> +        @broker =
@session.add_broker("amqp://#{server}:#{port}",
> :mechanism => 'GSSAPI')
> +
> +        # Connection succeeded, go about our business.
> +        return
> +      rescue Exception => msg
> +        puts "Error connecting to qpidd: #{msg}"
> +      end
> +      sleep(sleepy)
> +      sleepy *= 2
> +      sleepy = 120 if sleepy > 120
> +
> +      begin
> +        # Could also be a credentials problem?  Try to get them again..
> +        get_credentials('qpidd')
> +      rescue Exception => msg
> +        puts "Error getting qpidd credentials: #{msg}"
> +      end
> +    end
>   end
>
>   def find_capable_host(db_vm)
> @@ -755,6 +787,8 @@ class TaskOmatic
>         end
>       end
>
> +      qpid_ensure_connected
> +
>       tasks.each do |task|
>
>         task.time_started = Time.now
> --
> 1.6.0.6
>
> _______________________________________________
> Ovirt-devel mailing list
> Ovirt-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/ovirt-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/ovirt-devel/attachments/20090216/381750a7/attachment.htm>