Hope this is the right place to ask questions about unicorn. I''m not on the mailing list though, so please CC my email address. Since moving from phusion passenger to Unicorn, it seems like one functionality of the logging-rails gem (https://github.com/TwP/logging-rails) is (half) broken. Logging-rails supports an email appender, which aggregates log messages (typically ERROR and above) and then emails them. However, since moving to Unicorn we stopped getting those emails. The interesting thing however, is that when we USR2 signal our unicorn and it restarts itself, then those emails somehow get flushed out and get sent all at once... I tried sending USR1 signal to see if it has something to do with log flushing, but this does not seem to make any difference. Any ideas how to investigate this / make it work again?? Cheers Yoav
Yoav Aner <yoav at kenhub.com> wrote:> Hope this is the right place to ask questions about unicorn. I''m not > on the mailing list though, so please CC my email address.No problem!> Since moving from phusion passenger to Unicorn, it seems like one > functionality of the logging-rails gem > (https://github.com/TwP/logging-rails) is (half) broken. > > Logging-rails supports an email appender, which aggregates log > messages (typically ERROR and above) and then emails them. However, > since moving to Unicorn we stopped getting those emails. > > The interesting thing however, is that when we USR2 signal our unicorn > and it restarts itself, then those emails somehow get flushed out and > get sent all at once... I tried sending USR1 signal to see if it has > something to do with log flushing, but this does not seem to make any > difference.USR1 only affects logging to regular files, so is unlikely to help in your case. Are you using preload_app true or false? If true, perhaps some shared resource (socket or temporary file) isn''t being handled correctly?> Any ideas how to investigate this / make it work again??I''m not familiar with the "logging" or "logging-rails" gems. I took a brief look at them but couldn''t figure anything out, can you ask the author of those gems? If you do figure something out, please let the list know so it can possibly help others in the future.
Hi Eric, Thanks a bunch for getting back so quickly on this. I followed your suggestion and tried with `preload_app = false` and looks like this seems to fix this problem! Any idea what can go wrong when it''s set to true or how I can try go about fixing this?? I haven''t yet contacted the logging / logging-rails project. Perhaps that''s a good idea. Considering the gem did/does work fine on my dev environment and with phusion passenger (and now it seems also with Unicorn, albeit with preload_app = false), I wasn''t sure whether the problem is with this gem or elsewhere. Any tips on how to investigate this further or resolve this, or what information I can give the gem maintainer(s) would be much appreciated. Cheers Yoav On 30 November 2012 20:15, Eric Wong <normalperson at yhbt.net> wrote:> > Yoav Aner <yoav at kenhub.com> wrote: > > Hope this is the right place to ask questions about unicorn. I''m not > > on the mailing list though, so please CC my email address. > > No problem! > > > Since moving from phusion passenger to Unicorn, it seems like one > > functionality of the logging-rails gem > > (https://github.com/TwP/logging-rails) is (half) broken. > > > > Logging-rails supports an email appender, which aggregates log > > messages (typically ERROR and above) and then emails them. However, > > since moving to Unicorn we stopped getting those emails. > > > > The interesting thing however, is that when we USR2 signal our unicorn > > and it restarts itself, then those emails somehow get flushed out and > > get sent all at once... I tried sending USR1 signal to see if it has > > something to do with log flushing, but this does not seem to make any > > difference. > > USR1 only affects logging to regular files, so is unlikely to help > in your case. > > Are you using preload_app true or false? If true, perhaps some shared > resource (socket or temporary file) isn''t being handled correctly? > > > Any ideas how to investigate this / make it work again?? > > I''m not familiar with the "logging" or "logging-rails" gems. > I took a brief look at them but couldn''t figure anything out, > can you ask the author of those gems? > > If you do figure something out, please let the list know so it can > possibly help others in the future.
Yoav Aner <yoav at kenhub.com> wrote:> Hi Eric, > > Thanks a bunch for getting back so quickly on this. > > I followed your suggestion and tried with `preload_app = false` and looks > like this seems to fix this problem! Any idea what can go wrong when it''s > set to true or how I can try go about fixing this??preload_app is false by default because it''s the most likely to work for all gems. I suspect there''s a shared resource (file/socket) or some cached value that''s initialized before fork and loses state after forking. It''s a fairly common issue with database libraries> I haven''t yet contacted the logging / logging-rails project. Perhaps that''s > a good idea. Considering the gem did/does work fine on my dev environment > and with phusion passenger (and now it seems also with Unicorn, albeit with > preload_app = false), I wasn''t sure whether the problem is with this gem or > elsewhere. > > Any tips on how to investigate this further or resolve this, or what > information I can give the gem maintainer(s) would be much appreciated.I would definitely contact the maintainer of logging/logging-rails on how to reinitialize any state after forking.
Thanks Eric,
I think I might have found a solution :)
There was a hint in the config file I had for the logging-rails gem:
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
Logging.reopen if forked
end
end
I simply added `Logging.reopen` to the after_fork block and now it
looks like it''s working again!
Not sure I still fully (or even partially) understand what''s going on
there, but at least it''s working.
Thanks again for your help.
Cheers
Yoav
On 30 November 2012 21:05, Eric Wong <normalperson at yhbt.net>
wrote:> Yoav Aner <yoav at kenhub.com> wrote:
>> Hi Eric,
>>
>> Thanks a bunch for getting back so quickly on this.
>>
>> I followed your suggestion and tried with `preload_app = false` and
looks
>> like this seems to fix this problem! Any idea what can go wrong when
it''s
>> set to true or how I can try go about fixing this??
>
> preload_app is false by default because it''s the most likely to
work
> for all gems.
>
> I suspect there''s a shared resource (file/socket) or some cached
value
> that''s initialized before fork and loses state after forking.
>
> It''s a fairly common issue with database libraries
>
>> I haven''t yet contacted the logging / logging-rails project.
Perhaps that''s
>> a good idea. Considering the gem did/does work fine on my dev
environment
>> and with phusion passenger (and now it seems also with Unicorn, albeit
with
>> preload_app = false), I wasn''t sure whether the problem is
with this gem or
>> elsewhere.
>>
>> Any tips on how to investigate this further or resolve this, or what
>> information I can give the gem maintainer(s) would be much appreciated.
>
> I would definitely contact the maintainer of logging/logging-rails
> on how to reinitialize any state after forking.
Yoav Aner <yoav at kenhub.com> wrote:> I simply added `Logging.reopen` to the after_fork block and now it > looks like it''s working again!Awesome to know.> Not sure I still fully (or even partially) understand what''s going on > there, but at least it''s working.Reading a manpage for the fork() syscall should help you understand what''s going on.