In another thread, I had posted that we rotate our mongrel production
logs once a week as a means to keep the performance degradations that
happen with large log files at bay. A few of you asked me to post how
we do it, so here it is. We run our sites on RedHat Enterprise 4
machines. Our logrotate.conf file looks like this:
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# uncomment this if you want your log files compressed
#compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp -- we''ll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}
# system-specific logs may be also be configured here.
# mongrel logs
/path/to/rails/root/shared/log/*.log {
copytruncate
weekly
missingok
dateext
compress
sharedscripts
olddir /path/to/rails/root/old_mongrel_logs
rotate 28
postrotate
for i in `ls /path/to/rails/root/shared/log/mongrel*.pid`; do
kill -USR2 `cat $i`
done
endscript
}
Incidentally, we also use monit to monitor our various processes
(mongrel, Apache, ferretDRb MySQL, etc), so if for some odd reason
this log rotation would not bring one of our mongrels back up cleanly,
we''d be alerted and monit would try to restart it. So far, this has
not happened at all across any of our 6 production setups just like
this.
None of the sites are extremely high traffic, so it''s possible that
you''d want to rotate mongrel log files more frequently than weekly.
I hope this helps.
--
Sean Brown
seanmichaelbrown at gmail.com
On Thu, Feb 14, 2008 at 11:29 AM, Sean Brown <seanmichaelbrown at gmail.com> wrote:> postrotate > for i in `ls /path/to/rails/root/shared/log/mongrel*.pid`; do > kill -USR2 `cat $i` > done > endscript> Incidentally, we also use monit to monitor our various processes > (mongrel, Apache, ferretDRb MySQL, etc), so if for some odd reason > this log rotation would not bring one of our mongrels back up cleanly, > we''d be alerted and monit would try to restart it.Why not ask monit to restart your mongrels? It seems kind of nice to have monit be the only guy in town touching the processes it monitors.... but maybe there''s a technical reason why not? -Nate
On Thu, Feb 14, 2008 at 2:08 PM, Nate Vack <njvack at wisc.edu> wrote:> On Thu, Feb 14, 2008 at 11:29 AM, Sean Brown <seanmichaelbrown at gmail.com> wrote: > > > postrotate > > for i in `ls /path/to/rails/root/shared/log/mongrel*.pid`; do > > kill -USR2 `cat $i` > > done > > endscript > > > Incidentally, we also use monit to monitor our various processes > > (mongrel, Apache, ferretDRb MySQL, etc), so if for some odd reason > > this log rotation would not bring one of our mongrels back up cleanly, > > we''d be alerted and monit would try to restart it. > > Why not ask monit to restart your mongrels? It seems kind of nice to > have monit be the only guy in town touching the processes it > monitors.... but maybe there''s a technical reason why not? > > -Nate >No technical reason. Simply timing. Monit wakes itself every X minutes to check if processes are running, so there could be a delay of up X minutes until the mongrel process was restarted. The approach in the logrotate restarts the mongrels immediately. Interestingly, monit does realize that the process ID for each mongrel has changed and alerts of that fact, but since we know that will happen every Sunday at 3:00 AM, we can safely ignore it. -- Sean Brown seanmichaelbrown at gmail.com
On Thu, Feb 14, 2008 at 1:44 PM, Sean Brown <seanmichaelbrown at gmail.com> wrote:> On Thu, Feb 14, 2008 at 2:08 PM, Nate Vack <njvack at wisc.edu> wrote:> > Why not ask monit to restart your mongrels? It seems kind of nice to > > have monit be the only guy in town touching the processes it > > monitors.... but maybe there''s a technical reason why not?> No technical reason. Simply timing. Monit wakes itself every X > minutes to check if processes are running, so there could be a delay > of up X minutes until the mongrel process was restarted. The approach > in the logrotate restarts the mongrels immediately. Interestingly, > monit does realize that the process ID for each mongrel has changed > and alerts of that fact, but since we know that will happen every > Sunday at 3:00 AM, we can safely ignore it.I''m pretty sure that doing: monit restart all -g your_mongrel_group will wake monit up and have it restart your app before hitting its timeout. This also avoids the worried emails from monit; ignoring ''PID changed'' messages from monit (even if you''re expecting them) is a bad habit to get into. -n
Umm, why are you restarting your mongrels at all? copytruncate allows the processes to write to the log file without restarting The only issue is you may lose a few requests while it''s rotating, but in my case, I''m using the nginx logs to provide the stats anyway. On Thu, Feb 14, 2008 at 2:32 PM, Nate Vack <njvack at wisc.edu> wrote:> On Thu, Feb 14, 2008 at 1:44 PM, Sean Brown <seanmichaelbrown at gmail.com> wrote: > > > On Thu, Feb 14, 2008 at 2:08 PM, Nate Vack <njvack at wisc.edu> wrote: > > > > > Why not ask monit to restart your mongrels? It seems kind of nice to > > > have monit be the only guy in town touching the processes it > > > monitors.... but maybe there''s a technical reason why not? > > > > No technical reason. Simply timing. Monit wakes itself every X > > minutes to check if processes are running, so there could be a delay > > of up X minutes until the mongrel process was restarted. The approach > > in the logrotate restarts the mongrels immediately. Interestingly, > > monit does realize that the process ID for each mongrel has changed > > and alerts of that fact, but since we know that will happen every > > Sunday at 3:00 AM, we can safely ignore it. > > I''m pretty sure that doing: > > monit restart all -g your_mongrel_group > > will wake monit up and have it restart your app before hitting its > timeout. This also avoids the worried emails from monit; ignoring ''PID > changed'' messages from monit (even if you''re expecting them) is a bad > habit to get into. > > -n > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >