Ovirt devel - Aug 2009 - [PATCH server] Fixed db-omatic so it doesn't die due to unhandled exceptions related to saving db objects in update_host_state and update_domain_state.

The underlying issue has to do with optimistic locking for active record,
and contention between db-omatic and task-omatic. That still needs to be
fixed, especially since contention issues might get worse when the two
daemons are made multithreaded.
---
 src/db-omatic/db_omatic.rb |   22 +++++++++++++++++++---
 1 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/src/db-omatic/db_omatic.rb b/src/db-omatic/db_omatic.rb
index d2540e1..819f67c 100755
--- a/src/db-omatic/db_omatic.rb
+++ b/src/db-omatic/db_omatic.rb
@@ -160,7 +160,9 @@ class DbOmatic < Qpid::Qmf::Console
         end
 
         begin
-          # find open vm host history for this vm,
+          # find open vm host history for this vm
+          # NOTE: db-omatic is currently the only user for VmHostHistory, so
+          # using optimistic locking is fine. Someday this might need changing.
           history = VmHostHistory.find(:first, :conditions => ["vm_id =
? AND time_ended is NULL", vm.id])
 
           if state == Vm::STATE_RUNNING
@@ -215,7 +217,14 @@ class DbOmatic < Qpid::Qmf::Console
         end
 
         vm.state = state
-        vm.save!
+        
+        begin
+            vm.save!
+        rescue
+            @logger.error "Optimistic locking failed; tried updating a
stale row for VM #{vm.description}"
+            @logger.error "For now, workaround is retrying."
+            return update_domain_state(domain, state_override)
+        end
 
         domain[:synced] = true
     end
@@ -234,7 +243,13 @@ class DbOmatic < Qpid::Qmf::Console
             #db_host.lock_version = 2
             # XXX: This would just be for init..
             #db_host.is_disabled = 0
-            db_host.save!
+            
+            begin
+                db_host.save!
+            rescue
+                @logger.error "Optimistic locking failure on host
#{host_info['hostname']}, retrying."
+                return update_host_state(host_info, state)
+            end
             host_info[:synced] = true
 
             if state == Host::STATE_AVAILABLE
@@ -266,6 +281,7 @@ class DbOmatic < Qpid::Qmf::Console
                         end
                     rescue Exception => e # just log any errors here
                         @logger.info "Exception checking for dead VMs
(could be normal): #{e.message}"
+                        @logger.info e.backtrace
                     end
                 end
             end
-- 
1.6.2.5