Hey folks. Using mongrel_rails, Is it possible to have multiple mongrel instances write to the same log file? I suppose that implementing this might incur a performance hit since each mongrel would need to lock and unlock the file in order to not weave the log messages together in an unreadable stream? Or perhaps all the log messages are single-line anyway. Thanks, john -- John Joseph Bachir http://blog.johnjosephbachir.org http://lyceum.ibiblio.org http://dissent.cc http://jjb.cc
On Feb 5, 2008, at 2:06 PM, John Joseph Bachir wrote:> Hey folks. > > Using mongrel_rails, Is it possible to have multiple mongrel instances > write to the same log file? I suppose that implementing this might > incur a performance hit since each mongrel would need to lock and > unlock the file in order to not weave the log messages together in an > unreadable stream? Or perhaps all the log messages are single-line > anyway. > > Thanks, > john >Hi John, You''ll want to use something like SyslogLogger. Regards, Jason
On 2/5/08, John Joseph Bachir <johnjosephbachir at gmail.com> wrote:> > Hey folks. > > Using mongrel_rails, Is it possible to have multiple mongrel instances > write to the same log file? I suppose that implementing this might > incur a performance hit since each mongrel would need to lock and > unlock the file in order to not weave the log messages together in an > unreadable stream? Or perhaps all the log messages are single-line > anyway.Not 100 % sure I see what you''re looking for, but anyway: In my out-of-the-box setup, my three mongrelled rails instances all write to log/production.log. There is no locking, and sometimes log messages are interleaved (interleft?). The mongrels write a little something to their respective log files during start and stop, but that''s all. You might want to have a look at Kirk Haines''s "Analogger", which I believe solves the interleaving problem. I remember Kirk advising us on it on the list, so you could have a look in the archive. /David -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080205/f25e0709/attachment.html
On Feb 5, 2008 3:06 PM, John Joseph Bachir <johnjosephbachir at gmail.com> wrote:> Hey folks. > > Using mongrel_rails, Is it possible to have multiple mongrel instances > write to the same log file? I suppose that implementing this might > incur a performance hit since each mongrel would need to lock and > unlock the file in order to not weave the log messages together in an > unreadable stream? Or perhaps all the log messages are single-line > anyway.Are you talking about the mongrel log itself (which usually doesn''t accumulate much), or your application''s log? With as little as Mongrel itself writes, I wouldn''t see any significant problem in just having all of your mongrels write to the same file, personally. If you are talking about application logs, then one way is to use an external logging service of some sort. I have one, Analogger (http://analogger.swiftcore.org). As of the release there it isn''t quite drop in compatible with Rails, but I''m releasing an update which provides an interface to allow drop in Rails compatibility. Kirk Haines
On Feb 5, 2008 5:19 PM, David Vrensk <david at vrensk.com> wrote:> Not 100 % sure I see what you''re looking for, but anyway: In my > out-of-the-box setup, my three mongrelled rails instances all write to > log/production.log.On Feb 5, 2008 5:24 PM, Kirk Haines <wyhaines at gmail.com> wrote:> Are you talking about the mongrel log itself (which usually doesn''t > accumulate much), or your application''s log? > > With as little as Mongrel itself writes, I wouldn''t see any > significant problem in just having all of your mongrels write to the > same file, personally.Thanks for the quick replies. With a mongrel_cluster config file like this: user: root log_file: /path/to/app/log/dir/mongrel.log port: "8000" environment: production group: apache address: 127.0.0.1 servers: 8 ... I end up with these log files: mongrel.8000.log mongrel.8001.log mongrel.8002.log mongrel.8003.log mongrel.8004.log mongrel.8005.log mongrel.8006.log mongrel.8007.log With, as has been stated, very little written to each, and the application logging going to production.log. It would be nice if I could keep things tidier and have just one mongrel.log Thanks! John -- John Joseph Bachir http://blog.johnjosephbachir.org http://lyceum.ibiblio.org http://dissent.cc http://jjb.cc
We did this on "accident" on our site, and the performance hit is horrible. You might not notice it when you''re not getting a lot of traffic, but when you do, the performance hit is really awful. Each mongrel needs to have its own log file and log only the messages you absolutely need (which is pretty much true for every web server). Cheers, Kevin Lawver On Feb 5, 2008 2:06 PM, John Joseph Bachir <johnjosephbachir at gmail.com> wrote:> Hey folks. > > Using mongrel_rails, Is it possible to have multiple mongrel instances > write to the same log file? I suppose that implementing this might > incur a performance hit since each mongrel would need to lock and > unlock the file in order to not weave the log messages together in an > unreadable stream? Or perhaps all the log messages are single-line > anyway. > > Thanks, > john > > -- > John Joseph Bachir > http://blog.johnjosephbachir.org > http://lyceum.ibiblio.org > http://dissent.cc > http://jjb.cc > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-- Kevin Lawver http://lawver.net
On Feb 5, 2008 5:23 PM, Kevin Lawver <lawver at gmail.com> wrote:> We did this on "accident" on our site, and the performance hit is > horrible. You might not notice it when you''re not getting a lot of > traffic, but when you do, the performance hit is really awful. Each > mongrel needs to have its own log file and log only the messages you > absolutely need (which is pretty much true for every web server).That doesn''t really make sense, though, since Mongrel logs almost nothing. It shouldn''t materially affect performance because it is almost never writing anything to the file. If, on the other hand, you accidentally pooled your app logs into the same file, then yeah, that will create a performance hit under load because every requests on each different process is trying to push to that same file and just depending on the OS to sort it out. Kirk Haines
On Feb 5, 2008 7:32 PM, Kirk Haines <wyhaines at gmail.com> wrote:> If, on the other hand, you accidentally pooled your app logs into the > same file, then yeah, that will create a performance hit under load > because every requests on each different process is trying to push to > that same file and just depending on the OS to sort it out.But the application log is already consolidated (to production.log) by default-- are you suggesting that this incurs a performance hit and the log should be split up? -- John Joseph Bachir http://blog.johnjosephbachir.org http://lyceum.ibiblio.org http://dissent.cc http://jjb.cc
On Feb 6, 2008, at 16:53 , John Joseph Bachir wrote:> But the application log is already consolidated (to production.log) by > default-- are you suggesting that this incurs a performance hit and > the log should be split up?At least, that''s the experiences we had with our Rails application; random slowdowns for no apparent reason. All went away after we moved to SyslogLogger. -- Jakob Skjerning - http://mentalized.net
I recently just blogged about my experiences with large logfiles effecting mongrel. It''s not hugely scientific, but there is definitely a connection between mongrel memory usage and the rails production.log. http://blog.edhickey.com/2008/02/08/mongrel-memory-usage/ On Feb 10, 2008 5:29 PM, Jakob Skjerning <jakob at mentalized.net> wrote:> > On Feb 6, 2008, at 16:53 , John Joseph Bachir wrote: > > > But the application log is already consolidated (to production.log) by > > default-- are you suggesting that this incurs a performance hit and > > the log should be split up? > > At least, that''s the experiences we had with our Rails application; > random slowdowns for no apparent reason. All went away after we moved > to SyslogLogger. > > -- > Jakob Skjerning - http://mentalized.net > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
On Thu, Feb 14, 2008 at 1:38 AM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:> On Mon, 11 Feb 2008 09:36:06 -0600 > "Ed Hickey" <bassnode at gmail.com> wrote: > > > I recently just blogged about my experiences with large logfiles > > effecting mongrel. It''s not hugely scientific, but there is > > definitely a connection between mongrel memory usage and the rails > > production.log. > > http://blog.edhickey.com/2008/02/08/mongrel-memory-usage/ > > That''s fucking weird. There''s definitely a correlation, but not sure > how mongrel would cause that. It doesn''t touch the logs or logger or > anything and leaves that all to Rails. > > How much you want to bet that hack job rails-core did on the fucking > Logger class causes this? Not sure how. That''s just weird. > > Next would to see if it happens in jruby since they use a different IO > setup. > > Anyway, looks like syslogger is the way to go for production. >This won''t address the cause, but can alleviate the symptoms. We use logrotate to rotate our production logs once a week and that definitely helps keep things snappy. You certainly set them to rotate more or less frequently if your application needs it. If anyone would like to see our logrotate configuration, let me know. -- Sean Brown seanmichaelbrown at gmail.com
On Feb 13, 2008 10:50 AM, Sean Brown <seanmichaelbrown at gmail.com> wrote:> If anyone would > like to see our logrotate configuration, let me know.I would be interested in seeing your config, thanks for offering. Perhaps it''s worthy of a new thread? -- John Joseph Bachir http://blog.johnjosephbachir.org http://lyceum.ibiblio.org http://dissent.cc http://jjb.cc
So, it''s the size of the LOG that is causing the issue? I''m rotating every night anyway using logrotate, but I was thinking about moving to syslogger if I could find a nice how-to. On Feb 13, 2008 1:11 PM, John Joseph Bachir <johnjosephbachir at gmail.com> wrote:> On Feb 13, 2008 10:50 AM, Sean Brown <seanmichaelbrown at gmail.com> wrote: > > > If anyone would > > like to see our logrotate configuration, let me know. > > I would be interested in seeing your config, thanks for offering. > Perhaps it''s worthy of a new thread? > > -- > John Joseph Bachir > http://blog.johnjosephbachir.org > http://lyceum.ibiblio.org > http://dissent.cc > http://jjb.cc > _______________________________________________ > > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
At least in my case, the issue is definitely related to the size of the log. All the sapped memory gets freed as soon as I truncate the production.log. I tried SyslogLogger today (http://rails-analyzer.rubyforge.org/hacks/classes/SyslogLogger.html) but couldn''t get the rails log to only show up in one place when following the README''s directions It was logging to /var/log/production.log as well as /var/log/messages. Server load jumped up to 2, I assume because of all the disk IO. Anyone who''s deployed SyslogLogger and has insight, comments welcome. Also, anyone tried Analogger? http://analogger.swiftcore.org/ ed On Feb 13, 2008 3:13 PM, Joey Geiger <jgeiger at gmail.com> wrote:> So, it''s the size of the LOG that is causing the issue? > > I''m rotating every night anyway using logrotate, but I was thinking > about moving to syslogger if I could find a nice how-to. > > > On Feb 13, 2008 1:11 PM, John Joseph Bachir <johnjosephbachir at gmail.com> wrote: > > On Feb 13, 2008 10:50 AM, Sean Brown <seanmichaelbrown at gmail.com> wrote: > > > > > If anyone would > > > like to see our logrotate configuration, let me know. > > > > I would be interested in seeing your config, thanks for offering. > > Perhaps it''s worthy of a new thread? > > > > -- > > John Joseph Bachir > > http://blog.johnjosephbachir.org > > http://lyceum.ibiblio.org > > http://dissent.cc > > http://jjb.cc > > _______________________________________________ > > > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
On 13 Feb 2008, at 21:56, Ed Hickey wrote:> At least in my case, the issue is definitely related to the size of > the log. All the sapped memory gets freed as soon as I truncate the > production.log. > > I tried SyslogLogger today > (http://rails-analyzer.rubyforge.org/hacks/classes/SyslogLogger.html) > but couldn''t get the rails log to only show up in one place when > following the README''s directions It was logging to > /var/log/production.log as well as /var/log/messages. Server load > jumped up to 2, I assume because of all the disk IO. > Anyone who''s deployed SyslogLogger and has insight, comments welcome. > > Also, anyone tried Analogger? http://analogger.swiftcore.org/I''m not running it in production yet, but we''re using it in a development environment for our application. I sent Kirk Haines a patch which implements a Logger style api which you can hand to the rails log config options. I believe this will be coming with the release that is coming up soon. It''s been working just fine under development for our group, and we will be going production with it shortly.> > > ed > > > On Feb 13, 2008 3:13 PM, Joey Geiger <jgeiger at gmail.com> wrote: >> So, it''s the size of the LOG that is causing the issue? >> >> I''m rotating every night anyway using logrotate, but I was thinking >> about moving to syslogger if I could find a nice how-to. >> >> >> On Feb 13, 2008 1:11 PM, John Joseph Bachir <johnjosephbachir at gmail.com >> > wrote: >>> On Feb 13, 2008 10:50 AM, Sean Brown <seanmichaelbrown at gmail.com> >>> wrote: >>> >>>> If anyone would >>>> like to see our logrotate configuration, let me know. >>> >>> I would be interested in seeing your config, thanks for offering. >>> Perhaps it''s worthy of a new thread? >>> >>> -- >>> John Joseph Bachir >>> http://blog.johnjosephbachir.org >>> http://lyceum.ibiblio.org >>> http://dissent.cc >>> http://jjb.cc >>> _______________________________________________ >>> >>> Mongrel-users mailing list >>> Mongrel-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/mongrel-users >>> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On Mon, 11 Feb 2008 09:36:06 -0600 "Ed Hickey" <bassnode at gmail.com> wrote:> I recently just blogged about my experiences with large logfiles > effecting mongrel. It''s not hugely scientific, but there is > definitely a connection between mongrel memory usage and the rails > production.log. > http://blog.edhickey.com/2008/02/08/mongrel-memory-usage/That''s fucking weird. There''s definitely a correlation, but not sure how mongrel would cause that. It doesn''t touch the logs or logger or anything and leaves that all to Rails. How much you want to bet that hack job rails-core did on the fucking Logger class causes this? Not sure how. That''s just weird. Next would to see if it happens in jruby since they use a different IO setup. Anyway, looks like syslogger is the way to go for production. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/