Arjun Roy
2009-Aug-10 18:31 UTC
[Ovirt-devel] [PATCH server] Fixed db-omatic so it doesn't die due to an unhandled ActiveRecord::StaleObjectError exception.
The error is caused by the save! method updating an object that another thread (most likely in taskomatic) already updated. In the case of this race condition, we retry saving. However, a proper fix would involve fixing the locking. --- src/db-omatic/db_omatic.rb | 23 ++++++++++++++++++++--- 1 files changed, 20 insertions(+), 3 deletions(-) diff --git a/src/db-omatic/db_omatic.rb b/src/db-omatic/db_omatic.rb index d2540e1..60ea9f7 100755 --- a/src/db-omatic/db_omatic.rb +++ b/src/db-omatic/db_omatic.rb @@ -160,7 +160,9 @@ class DbOmatic < Qpid::Qmf::Console end begin - # find open vm host history for this vm, + # find open vm host history for this vm + # NOTE: db-omatic is currently the only user for VmHostHistory, so + # using optimistic locking is fine. Someday this might need changing. history = VmHostHistory.find(:first, :conditions => ["vm_id = ? AND time_ended is NULL", vm.id]) if state == Vm::STATE_RUNNING @@ -215,7 +217,14 @@ class DbOmatic < Qpid::Qmf::Console end vm.state = state - vm.save! + + begin + vm.save! + rescue ActiveRecord::StaleObjectError => e + @logger.error "Optimistic locking failed for VM #{vm.description}, retrying." + @logger.error e.backtrace + return update_domain_state(domain, state_override) + end domain[:synced] = true end @@ -234,7 +243,14 @@ class DbOmatic < Qpid::Qmf::Console #db_host.lock_version = 2 # XXX: This would just be for init.. #db_host.is_disabled = 0 - db_host.save! + + begin + db_host.save! + rescue ActiveRecord::StaleObjectError => e + @logger.error "Optimistic locking failure on host #{host_info['hostname']}, retrying." + @logger.error e.backtrace + return update_host_state(host_info, state) + end host_info[:synced] = true if state == Host::STATE_AVAILABLE @@ -266,6 +282,7 @@ class DbOmatic < Qpid::Qmf::Console end rescue Exception => e # just log any errors here @logger.info "Exception checking for dead VMs (could be normal): #{e.message}" + @logger.info e.backtrace end end end -- 1.6.2.5
Scott Seago
2009-Aug-10 18:55 UTC
[Ovirt-devel] [PATCH server] Fixed db-omatic so it doesn't die due to an unhandled ActiveRecord::StaleObjectError exception.
Arjun Roy wrote:> The error is caused by the save! method updating an object that another > thread (most likely in taskomatic) already updated. > > In the case of this race condition, we retry saving. However, a proper fix > would involve fixing the locking. > --- > src/db-omatic/db_omatic.rb | 23 ++++++++++++++++++++--- > 1 files changed, 20 insertions(+), 3 deletions(-) > > diff --git a/src/db-omatic/db_omatic.rb b/src/db-omatic/db_omatic.rb > index d2540e1..60ea9f7 100755 > --- a/src/db-omatic/db_omatic.rb > +++ b/src/db-omatic/db_omatic.rb > @@ -215,7 +217,14 @@ class DbOmatic < Qpid::Qmf::Console > end > > vm.state = state > - vm.save! > + > + begin > + vm.save! > + rescue ActiveRecord::StaleObjectError => e > + @logger.error "Optimistic locking failed for VM #{vm.description}, retrying." > + @logger.error e.backtrace > + return update_domain_state(domain, state_override) > + end > > domain[:synced] = true > end > @@ -234,7 +243,14 @@ class DbOmatic < Qpid::Qmf::Console > #db_host.lock_version = 2 > # XXX: This would just be for init.. > #db_host.is_disabled = 0 > - db_host.save! > + > + begin > + db_host.save! > + rescue ActiveRecord::StaleObjectError => e > + @logger.error "Optimistic locking failure on host #{host_info['hostname']}, retrying." > + @logger.error e.backtrace > + return update_host_state(host_info, state) > + end > host_info[:synced] = true > > if state == Host::STATE_AVAILABLE >OK I pushed this but replacing the "retry" return statements with blanke return statements, as we have not yet been able to test the StaleObjectError retries. Scott
Apparently Analagous Threads
- [PATCH server] Fixed db-omatic so it doesn't die due to unhandled exceptions related to saving db objects in update_host_state and update_domain_state.
- [PATCH] Fix dbomatic state changes.
- [PATCH server] Fixed db-omatic to update the host for a running vm when the object_props callback is fired.
- [PATCH server] Replace the occurence of the type @qmfc.object(Qmf::Query.new(:class => "xxx", 'key' => search_key)) for @qmfc.object(Qmf::Query.new(:class => "xxx"), 'key' => search_key) else the search on the key is not functionnal.
- [PATCH server] Fixed db-omatic so it doesn't segfault because of newer qmf api.