I switched from mod_passenger to unicorn on a fairly high traffic site
and ran into a strange problem that forced me to move back to
mod_passenger... it seems as if classes would sometimes get mixed up
with each other. If I had two Rails models:
class Foo < ActiveRecord::Base; end
class Bar < ActiveRecord::Base; end
Normally Foo.inspect and Bar.inspect would return:
Foo(field1: integer, field2: integer)
Bar(field3: integer, field4: integer)
When things were "broken" within a process, sometimes I would see:
Foo(field3: integer, field4: integer) <--- note field3/field4
actually belong to Bar, not Foo
And because of that, wacky errors would appear in my logs like:
Foo.find_by_field1(12345) --> not a method
Foo.create(:field1 => 12345) --> column not found
I also noticed the problem with field serializing in ActiveRecord... given:
class Boz < ActiveRecord::Base
serialize :some_data
end
When processes were working correctly, Boz.find(1).some_data would
return an actual object (like a Hash). When things were broken, the
raw serialized string from the database would be returned... almost as
if the Boz class "forgot" that it''s supposed to deserialize
"some_data".
Could it be that class attributes are somehow being co-mingled when
unicorn is starting up under high concurrency? Perhaps a mutex is
missing somewhere?
El S?bado, 6 de Febrero de 2010, Warren Konkel escribi?:> I switched from mod_passenger to unicorn on a fairly high traffic site > and ran into a strange problem that forced me to move back to > mod_passenger... it seems as if classes would sometimes get mixed up > with each other. If I had two Rails models: > > class Foo < ActiveRecord::Base; end > class Bar < ActiveRecord::Base; end > > Normally Foo.inspect and Bar.inspect would return: > > Foo(field1: integer, field2: integer) > Bar(field3: integer, field4: integer) > > When things were "broken" within a process, sometimes I would see: > > Foo(field3: integer, field4: integer) <--- note field3/field4 > actually belong to Bar, not Foo > > And because of that, wacky errors would appear in my logs like: > > Foo.find_by_field1(12345) --> not a method > Foo.create(:field1 => 12345) --> column not found > > I also noticed the problem with field serializing in ActiveRecord... given: > > class Boz < ActiveRecord::Base > serialize :some_data > end > > When processes were working correctly, Boz.find(1).some_data would > return an actual object (like a Hash). When things were broken, the > raw serialized string from the database would be returned... almost as > if the Boz class "forgot" that it''s supposed to deserialize > "some_data". > > Could it be that class attributes are somehow being co-mingled when > unicorn is starting up under high concurrency? Perhaps a mutex is > missing somewhere?IMHO all your Unicorn workers are sharing the same DB connection (the same ActiveRecord instances) so the problem arises. Take a look to the configuration here: http://unicorn.bogomips.org/examples/unicorn.conf.rb You can see there how the ActiveRecord is disconnected at the beggining and started for each worker later. -- I?aki Baz Castillo <ibc at aliax.net>
I?aki Baz Castillo <ibc at aliax.net> wrote:> El S?bado, 6 de Febrero de 2010, Warren Konkel escribi?: > > Could it be that class attributes are somehow being co-mingled when > > unicorn is starting up under high concurrency? Perhaps a mutex is > > missing somewhere? > > IMHO all your Unicorn workers are sharing the same DB connection (the same > ActiveRecord instances) so the problem arises. > > Take a look to the configuration here: > http://unicorn.bogomips.org/examples/unicorn.conf.rb > You can see there how the ActiveRecord is disconnected at the beggining and > started for each worker later.Yes, you have to reconnect any connected TCP sockets since they have no defined atomicity semantics when they''re shared across processes/threads. Ruby Mutexes aren''t useful at all here, they''re only useful with threads in the same process. If you need to protect TCP client sockets from multiple processes, you''ll need SysV/POSIX semaphores or file locks (flock/fcntl), but you''re really better off using multiple TCP client sockets in the first place. -- Eric Wong