I switched from mod_passenger to unicorn on a fairly high traffic site and ran into a strange problem that forced me to move back to mod_passenger... it seems as if classes would sometimes get mixed up with each other. If I had two Rails models: class Foo < ActiveRecord::Base; end class Bar < ActiveRecord::Base; end Normally Foo.inspect and Bar.inspect would return: Foo(field1: integer, field2: integer) Bar(field3: integer, field4: integer) When things were "broken" within a process, sometimes I would see: Foo(field3: integer, field4: integer) <--- note field3/field4 actually belong to Bar, not Foo And because of that, wacky errors would appear in my logs like: Foo.find_by_field1(12345) --> not a method Foo.create(:field1 => 12345) --> column not found I also noticed the problem with field serializing in ActiveRecord... given: class Boz < ActiveRecord::Base serialize :some_data end When processes were working correctly, Boz.find(1).some_data would return an actual object (like a Hash). When things were broken, the raw serialized string from the database would be returned... almost as if the Boz class "forgot" that it''s supposed to deserialize "some_data". Could it be that class attributes are somehow being co-mingled when unicorn is starting up under high concurrency? Perhaps a mutex is missing somewhere?
El S?bado, 6 de Febrero de 2010, Warren Konkel escribi?:> I switched from mod_passenger to unicorn on a fairly high traffic site > and ran into a strange problem that forced me to move back to > mod_passenger... it seems as if classes would sometimes get mixed up > with each other. If I had two Rails models: > > class Foo < ActiveRecord::Base; end > class Bar < ActiveRecord::Base; end > > Normally Foo.inspect and Bar.inspect would return: > > Foo(field1: integer, field2: integer) > Bar(field3: integer, field4: integer) > > When things were "broken" within a process, sometimes I would see: > > Foo(field3: integer, field4: integer) <--- note field3/field4 > actually belong to Bar, not Foo > > And because of that, wacky errors would appear in my logs like: > > Foo.find_by_field1(12345) --> not a method > Foo.create(:field1 => 12345) --> column not found > > I also noticed the problem with field serializing in ActiveRecord... given: > > class Boz < ActiveRecord::Base > serialize :some_data > end > > When processes were working correctly, Boz.find(1).some_data would > return an actual object (like a Hash). When things were broken, the > raw serialized string from the database would be returned... almost as > if the Boz class "forgot" that it''s supposed to deserialize > "some_data". > > Could it be that class attributes are somehow being co-mingled when > unicorn is starting up under high concurrency? Perhaps a mutex is > missing somewhere?IMHO all your Unicorn workers are sharing the same DB connection (the same ActiveRecord instances) so the problem arises. Take a look to the configuration here: http://unicorn.bogomips.org/examples/unicorn.conf.rb You can see there how the ActiveRecord is disconnected at the beggining and started for each worker later. -- I?aki Baz Castillo <ibc at aliax.net>
I?aki Baz Castillo <ibc at aliax.net> wrote:> El S?bado, 6 de Febrero de 2010, Warren Konkel escribi?: > > Could it be that class attributes are somehow being co-mingled when > > unicorn is starting up under high concurrency? Perhaps a mutex is > > missing somewhere? > > IMHO all your Unicorn workers are sharing the same DB connection (the same > ActiveRecord instances) so the problem arises. > > Take a look to the configuration here: > http://unicorn.bogomips.org/examples/unicorn.conf.rb > You can see there how the ActiveRecord is disconnected at the beggining and > started for each worker later.Yes, you have to reconnect any connected TCP sockets since they have no defined atomicity semantics when they''re shared across processes/threads. Ruby Mutexes aren''t useful at all here, they''re only useful with threads in the same process. If you need to protect TCP client sockets from multiple processes, you''ll need SysV/POSIX semaphores or file locks (flock/fcntl), but you''re really better off using multiple TCP client sockets in the first place. -- Eric Wong