Ovirt devel - Aug 2009 - [PATCH server] Fixed db-omatic so it doesn't die due to an unhandled ActiveRecord::StaleObjectError exception.

The error is caused by the save! method updating an object that another
thread (most likely in taskomatic) already updated.

In the case of this race condition, we retry saving. However, a proper fix
would involve fixing the locking.
---
 src/db-omatic/db_omatic.rb |   23 ++++++++++++++++++++---
 1 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/src/db-omatic/db_omatic.rb b/src/db-omatic/db_omatic.rb
index d2540e1..60ea9f7 100755
--- a/src/db-omatic/db_omatic.rb
+++ b/src/db-omatic/db_omatic.rb
@@ -160,7 +160,9 @@ class DbOmatic < Qpid::Qmf::Console
         end
 
         begin
-          # find open vm host history for this vm,
+          # find open vm host history for this vm
+          # NOTE: db-omatic is currently the only user for VmHostHistory, so
+          # using optimistic locking is fine. Someday this might need changing.
           history = VmHostHistory.find(:first, :conditions => ["vm_id =
? AND time_ended is NULL", vm.id])
 
           if state == Vm::STATE_RUNNING
@@ -215,7 +217,14 @@ class DbOmatic < Qpid::Qmf::Console
         end
 
         vm.state = state
-        vm.save!
+        
+        begin
+            vm.save!
+        rescue ActiveRecord::StaleObjectError => e
+            @logger.error "Optimistic locking failed for VM
#{vm.description}, retrying."
+            @logger.error e.backtrace
+            return update_domain_state(domain, state_override)
+        end
 
         domain[:synced] = true
     end
@@ -234,7 +243,14 @@ class DbOmatic < Qpid::Qmf::Console
             #db_host.lock_version = 2
             # XXX: This would just be for init..
             #db_host.is_disabled = 0
-            db_host.save!
+            
+            begin
+                db_host.save!
+            rescue ActiveRecord::StaleObjectError => e
+                @logger.error "Optimistic locking failure on host
#{host_info['hostname']}, retrying."
+                @logger.error e.backtrace
+                return update_host_state(host_info, state)
+            end
             host_info[:synced] = true
 
             if state == Host::STATE_AVAILABLE
@@ -266,6 +282,7 @@ class DbOmatic < Qpid::Qmf::Console
                         end
                     rescue Exception => e # just log any errors here
                         @logger.info "Exception checking for dead VMs
(could be normal): #{e.message}"
+                        @logger.info e.backtrace
                     end
                 end
             end
-- 
1.6.2.5

Arjun Roy wrote:
> The error is caused by the save! method updating an object that another
> thread (most likely in taskomatic) already updated.
>
> In the case of this race condition, we retry saving. However, a proper fix
> would involve fixing the locking.
> ---
>  src/db-omatic/db_omatic.rb |   23 ++++++++++++++++++++---
>  1 files changed, 20 insertions(+), 3 deletions(-)
>
> diff --git a/src/db-omatic/db_omatic.rb b/src/db-omatic/db_omatic.rb
> index d2540e1..60ea9f7 100755
> --- a/src/db-omatic/db_omatic.rb
> +++ b/src/db-omatic/db_omatic.rb
> @@ -215,7 +217,14 @@ class DbOmatic < Qpid::Qmf::Console
>          end
>  
>          vm.state = state
> -        vm.save!
> +        
> +        begin
> +            vm.save!
> +        rescue ActiveRecord::StaleObjectError => e
> +            @logger.error "Optimistic locking failed for VM
#{vm.description}, retrying."
> +            @logger.error e.backtrace
> +            return update_domain_state(domain, state_override)
> +        end
>  
>          domain[:synced] = true
>      end
> @@ -234,7 +243,14 @@ class DbOmatic < Qpid::Qmf::Console
>              #db_host.lock_version = 2
>              # XXX: This would just be for init..
>              #db_host.is_disabled = 0
> -            db_host.save!
> +            
> +            begin
> +                db_host.save!
> +            rescue ActiveRecord::StaleObjectError => e
> +                @logger.error "Optimistic locking failure on host
#{host_info['hostname']}, retrying."
> +                @logger.error e.backtrace
> +                return update_host_state(host_info, state)
> +            end
>              host_info[:synced] = true
>  
>              if state == Host::STATE_AVAILABLE
>   
OK I pushed this but replacing the "retry" return statements with
blanke
return statements, as we have not yet been able to test the 
StaleObjectError retries.

Scott