Hi All, When I deploy new code I always go through the USR2 - QUIT process. However, after a few deployments it seems our base javascript file is missing. I''ve experienced this a few times now. We use rails 2.3.8, and have something like this in our layouts file: <%= javascript_include_tag ''webtoolkit'', ''jquery'', ''jquery.metadata'', ''jquery.url_utils'', ''jquery.form'', ''jquery.hotkeys'', ''jquery.livequery'', ''application'', :cache => ''base'' %> When it''s missing it doesn''t matter how often I kill with USR2 / QUIT, it won''t appear. If I stop the unicorn master process completely and then start again, then it does create the base.js file again. Anyone experienced this as well? Anyone know what could be the reason for this? Jimmy
Btw, we also had this weird thing last week where after a new deployment of code almost all field labels had the same value. We use I18n v0.4.1 with a cache using memory_store, like so: I18n::Backend::Simple.send(:include, I18n::Backend::Cache) I18n.cache_store = ActiveSupport::Cache.lookup_store(:memory_store) I18n.cache_store.silence! I get the feeling that memory from the previous master/workers somehow isn''t released and is used by the new unicorn master and/or worker processes. Jimmy
Jimmy Soho <jimmy.soho at gmail.com> wrote:> Hi All, > > When I deploy new code I always go through the USR2 - QUIT process. > However, after a few deployments it seems our base javascript file is > missing. I''ve experienced this a few times now. We use rails 2.3.8, > and have something like this in our layouts file: > > <%= javascript_include_tag ''webtoolkit'', ''jquery'', ''jquery.metadata'', > ''jquery.url_utils'', ''jquery.form'', ''jquery.hotkeys'', > ''jquery.livequery'', ''application'', :cache => ''base'' %> > > When it''s missing it doesn''t matter how often I kill with USR2 / QUIT, > it won''t appear. If I stop the unicorn master process completely and > then start again, then it does create the base.js file again. > > Anyone experienced this as well? Anyone know what could be the reason > for this?Hi Jimmy, Are you deploying with Capistrano and you originally started Unicorn in a working_directory that no longer exists on the filesystem? Under Linux, you can check the working directory of any process with `ls -l /proc/$PID/cwd` That''s the only thing I can think of at the moment. In your other email:> I get the feeling that memory from the previous master/workers somehow > isn''t released and is used by the new unicorn master and/or worker > processes.Shouldn''t be possible unless your OS is broken. A more likely explanation is that Unicorn process is pinned to the working directory of an old deploy. -- Eric Wong
> Hi Jimmy, ? Are you deploying with Capistrano and you originally started > Unicorn in a working_directory that no longer exists on the filesystem? > > Under Linux, you can check the working directory of any process > with `ls -l /proc/$PID/cwd`Ah! Yes. I use Capistrano, and it pruned releases. But definitely /after/ a new version was deployed. What I don''t understand then though is that I have this in my unicorn config file: working_directory "/srv/app1/current" How could the new master / workers still be pointing to the old directory? Isn''t what this working_directory directive is for? Jimmy
Jimmy Soho <jimmy.soho at gmail.com> wrote:> > Hi Jimmy, ? Are you deploying with Capistrano and you originally started > > Unicorn in a working_directory that no longer exists on the filesystem? > > > > Under Linux, you can check the working directory of any process > > with `ls -l /proc/$PID/cwd` > > Ah! Yes. I use Capistrano, and it pruned releases. But definitely > /after/ a new version was deployed. > > What I don''t understand then though is that I have this in my unicorn > config file: > > working_directory "/srv/app1/current" > > How could the new master / workers still be pointing to the old > directory? Isn''t what this working_directory directive is for?In your Capistrano deploy tasks, do you create/update the current symlink before you send USR2? You should have the symlink updated before sending USR2.. Other folks here may have more experience with Capistrano+Unicorn, maybe they have portable recipes they can share. -- Eric Wong
> In your Capistrano deploy tasks, do you create/update the current > symlink before you send USR2? ?You should have the symlink updated > before sending USR2..Yes I do. Meanwhile I''ve been able to indeed reproduce the issue I was seeing if I prune releases. This is what I do: - ls /srv/app/releases shows only: /srv/app/releases/20100818022900 - stop unicorn - start unicorn - deploy new release (same code actually) - unicorn now has a different PID, /proc/31761/cwd does point to the latest release directory: /proc/31761/cwd -> /srv/app/releases/20100818024514 - refresh browser, ok, no issue. - rm -Rf /srv/app/releases/20100818022900 - deploy another new release - refresh browser: javascripts/base.js is missing nothing in unicorn.stderr.log, but looking in unicorn.stdout.log it mentions: /srv/app/releases/20100818022900/Gemfile not found The unicorn process PID hasn''t changed, nor has proc/31761/cwd My /srv/app/current dir now points to the latest release. Which I guess is wrong as well, as unicorn failed to start with the latest release; so it (= capistrano) needs to rollback somehow (which is a different issue). If I now peek in /proc/31761/environ then I notice these: GEM_HOME=/srv/app/releases/20100818022900/vendor/bundler_gems PATH=/srv/app/releases/20100818022900/vendor/bundler_gems/bin:[... etc.] GEM_PATH=/srv/app/releases/20100818022900/vendor/bundler_gems BUNDLE_GEMFILE=/srv/app/releases/20100818022900/Gemfile which are all pointing to the working directory I pruned. Hence the message about Gemfile not found I guess. Any suggestions how / when to update those environment variables with correct values?> Other folks here may have more experience with Capistrano+Unicorn, > maybe they have portable recipes they can share.Maybe it''s more a gem and/or bundler thing? Though it seems like a general issue to me that environment variables can have references to old working directories, which need to be updated if you restart Unicorn with USR2 all the time. Is this something Unicorn could possibly take care off? Jimmy
Jimmy Soho <jimmy.soho at gmail.com> wrote:> If I now peek in /proc/31761/environ then I notice these: > > GEM_HOME=/srv/app/releases/20100818022900/vendor/bundler_gems > PATH=/srv/app/releases/20100818022900/vendor/bundler_gems/bin:[... etc.] > GEM_PATH=/srv/app/releases/20100818022900/vendor/bundler_gems > BUNDLE_GEMFILE=/srv/app/releases/20100818022900/Gemfile > > which are all pointing to the working directory I pruned. Hence the > message about Gemfile not found I guess. > > Any suggestions how / when to update those environment variables with > correct values?Aha! I believe there''s an option for Bundler to use the /srv/app/shared directory with Capistrano. That''s probably your best option.> > Other folks here may have more experience with Capistrano+Unicorn, > > maybe they have portable recipes they can share. > > Maybe it''s more a gem and/or bundler thing? Though it seems like a > general issue to me that environment variables can have references to > old working directories, which need to be updated if you restart > Unicorn with USR2 all the time. Is this something Unicorn could > possibly take care off?You can also try force the environment variables with the before_exec hook in your config file: before_exec do |server| ENV[''GEM_HOME''] = ''/srv/app/current/vendor/bundler_gems'' end But I think the best option is to use a shared directory for Bundler since it should speed up deploys, too. On the other hand, maybe Bundler should be taught to respect symlink directories... -- Eric Wong
> Aha! I believe there''s an option for Bundler to use the /srv/app/shared > directory with Capistrano. That''s probably your best option.I am using the shared bundler gems option, I use the example floating around the internets: run("cd #{release_path} && bundle install vendor/bundler_gems --without development test --disable-shared-gems")> You can also try force the environment variables with the > before_exec hook in your config file: > > before_exec do |server| > ENV[''GEM_HOME''] = ''/srv/app/current/vendor/bundler_gems'' > endAfter a few hours hacking I''ve come to the conclusion that''s the only way it can work, for now. Having delved into bundler I think Bundler.setup needs to change. I''ll file an issue for that with them. PS. I: for those interested, this is now the unicorn-conf.rb file I''m using, so I can now happily prune all old releases: http://gist.github.com/534668 PS. II: While I trying out various things I found that if I modified the before_exec hook, then deployed that code, send USR2, then it executes the before_exec hook of the old master, not the new master. So I then had to send a USR2 a second time to see the new code in before_exec taking effect. Is that intentional behaviour? PS. III: Haven''t tested it yet, but in case a USR2 fails, and unicorn reverts back to the old master, does the old master then have the new environment variable values? Or does it run the before_exec hook again for the old master? Jimmy
Jimmy Soho <jimmy.soho at gmail.com> wrote:> PS. II: While I trying out various things I found that if I modified > the before_exec hook, then deployed that code, send USR2, then it > executes the before_exec hook of the old master, not the new master. > So I then had to send a USR2 a second time to see the new code in > before_exec taking effect. Is that intentional behaviour?Yes. USR2 doesn''t reload the config file before by default. Send a HUP before USR2 if you want it to reload the config file. Just keep in mind if you add any new dependencies on your new deploy and HUP, things could break, too, especially if you''re using "preload_app true".> PS. III: Haven''t tested it yet, but in case a USR2 fails, and unicorn > reverts back to the old master, does the old master then have the new > environment variable values? Or does it run the before_exec hook again > for the old master?The old master doesn''t get affected by environment (or any other local) changes in before_exec. before_exec only runs in the new master process, so it can only affect the new master process (unless it sends signals to the old master). -- Eric Wong