Hi, My department has a SAN and I''m wondering if it is safe to have all of my mongrel''s share log files stored on the SAN. More specifically, we have 4 machines, each running a couple of mongrel processes. Right now each machine has its own set of log files for mongrel. However, I am thinking that it might be better to put the log files on the SAN and let all the mongrels on all the different machines share 1 set of log files. I''d be interested to gets some feedback from other mongrelians on whether this is a good idea or a bad idea and why. Thanks, Steven
On Thu, 30 Nov 2006 10:36:47 -0800 Steven Hansen <runner at berkeley.edu> wrote:> > Hi, > > My department has a SAN and I''m wondering if it is safe to have all of > my mongrel''s share log files stored on the SAN. > > More specifically, we have 4 machines, each running a couple of mongrel > processes. Right now each machine has its own set of log files for > mongrel. However, I am thinking that it might be better to put the log > files on the SAN and let all the mongrels on all the different machines > share 1 set of log files. I''d be interested to gets some feedback from > other mongrelians on whether this is a good idea or a bad idea and why.I should probably just make this the norm, but I advocate that people setup pid and log files separate for each port they have, and probable each machine:port combination. This makes it easier to isolate one mongrel''s stuff and to track what is happening to a particular instance. If you do this it also means you can share it all you like. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Ezra Zygmuntowicz
2006-Nov-30 20:54 UTC
[Mongrel] Sharing Mongrel Log Files on SAN Storage
On Nov 30, 2006, at 10:36 AM, Steven Hansen wrote:> > Hi, > > My department has a SAN and I''m wondering if it is safe to have all of > my mongrel''s share log files stored on the SAN. > > More specifically, we have 4 machines, each running a couple of > mongrel > processes. Right now each machine has its own set of log files for > mongrel. However, I am thinking that it might be better to put the > log > files on the SAN and let all the mongrels on all the different > machines > share 1 set of log files. I''d be interested to gets some feedback > from > other mongrelians on whether this is a good idea or a bad idea and > why. > > Thanks, > Steven >Hey Steven- What filesystem re you using n the SAN? We have many Xen nodes running rails apps that share a GFS filesystem partition off the SAN. With GFS you can make hostname specific symlinks so that even though the nodes of the cluster are all still writing their logs to logs/ production.log, each node is actually writing to a symlink based on hostname. Let me illustrate. Say we have 2 Xen instances for one rails app. They both share a GFS partition mounted off the SAN at /data. So anything under /data is shared between these nodes. With a capistrano setup you make sure to deploy yoru apps to the /data partition. So to setup the hostname logging on both slices you woudl log into each one and run the following commands: app is here: /data/railsapp/current logs are here: /data/railsapp/shared/log We want both nodes to have their own log files but still have them both be writing to the same log file path. So we make hostname symlinks like this: $ cd /data/railsapp/shared $ rm -rf log $ mkdir `hostname` $ ln -s @hostname log You do that on all nodes that share the same /data partition. This gives a very dramatic increase in performance over all nodes writing to the same log file. Getting a file lock on shared storage is more troublesome then on non shared storage. So with multiple nodes all writing to the same log file, each node has to get a lock before it can write and other nodes wait their turn for the lock. Using this @hostname trick, each node has its own log file that no other nodes write to. But you don''t have to change the way rails or capistrano sets up the log files. Each node writes its logs to /data/railsapp/ shared/log/production.log, but in reality they are all writing to a symlink to a directory with the same name as the nodes hostname. We tried NFS and a few other filesystems and they all sucked. We ended up using the red hat clustering suite. But even though its a red hat suite it runs on any linux, we use it with gentoo. It includes GFS and the cluster fencing stuff that allows nodes to dynamically fence off any other nodes that are misbehaving. GFS works really well for shared storage off a SAN. I would definitely recommend not having all your nodes write their logs to the same file on the SAN. See if you can find a way to do like GFS does or as a last resort you may have to do some hacking to get rails to write its log files with the pid or some other unique identifier so that each node writes to its own log files. Doing this gave us a whopping 25% better performance when compared with all nodes sharing the same log file. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)
Ezra Zygmuntowicz wrote:> On Nov 30, 2006, at 10:36 AM, Steven Hansen wrote: > > >> Hi, >> >> My department has a SAN and I''m wondering if it is safe to have all of >> my mongrel''s share log files stored on the SAN. >> >> More specifically, we have 4 machines, each running a couple of >> mongrel >> processes. Right now each machine has its own set of log files for >> mongrel. However, I am thinking that it might be better to put the >> log >> files on the SAN and let all the mongrels on all the different >> machines >> share 1 set of log files. I''d be interested to gets some feedback >> from >> other mongrelians on whether this is a good idea or a bad idea and >> why. >> >> Thanks, >> Steven >> >> > > Hey Steven- > > What filesystem re you using n the SAN? We have many Xen nodes > running rails apps that share a GFS filesystem partition off the SAN. > With GFS you can make hostname specific symlinks so that even though > the nodes of the cluster are all still writing their logs to logs/ > production.log, each node is actually writing to a symlink based on > hostname. > > Let me illustrate. Say we have 2 Xen instances for one rails app. > They both share a GFS partition mounted off the SAN at /data. So > anything under /data is shared between these nodes. With a capistrano > setup you make sure to deploy yoru apps to the /data partition. So to > setup the hostname logging on both slices you woudl log into each one > and run the following commands: > > app is here: > /data/railsapp/current > > logs are here: > /data/railsapp/shared/log > > We want both nodes to have their own log files but still have them > both be writing to the same log file path. So we make hostname > symlinks like this: > > $ cd /data/railsapp/shared > $ rm -rf log > $ mkdir `hostname` > $ ln -s @hostname log > > You do that on all nodes that share the same /data partition. This > gives a very dramatic increase in performance over all nodes writing > to the same log file. Getting a file lock on shared storage is more > troublesome then on non shared storage. So with multiple nodes all > writing to the same log file, each node has to get a lock before it > can write and other nodes wait their turn for the lock. Using this > @hostname trick, each node has its own log file that no other nodes > write to. But you don''t have to change the way rails or capistrano > sets up the log files. Each node writes its logs to /data/railsapp/ > shared/log/production.log, but in reality they are all writing to a > symlink to a directory with the same name as the nodes hostname. > > We tried NFS and a few other filesystems and they all sucked. We > ended up using the red hat clustering suite. But even though its a > red hat suite it runs on any linux, we use it with gentoo. It > includes GFS and the cluster fencing stuff that allows nodes to > dynamically fence off any other nodes that are misbehaving. GFS works > really well for shared storage off a SAN. > > I would definitely recommend not having all your nodes write their > logs to the same file on the SAN. See if you can find a way to do > like GFS does or as a last resort you may have to do some hacking to > get rails to write its log files with the pid or some other unique > identifier so that each node writes to its own log files. Doing this > gave us a whopping 25% better performance when compared with all > nodes sharing the same log file. > > > Cheers- > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >Ezra and Zed, Thanks for responding. I feel a little foolish for even asking the question now, better safe than sorry though :-P I guess originally I was thinking that if I wanted to check logs for traffic or something, then having it all in one log would be easier. But as Zed pointed out, this would make tracking down problems with an offending mongrel instance, much more difficult. Not to mention that the issue with more mongrel threads fighting for the lock on the log files. Anyway, I''m not sure what type of file system we are using as I really don''t have much contact with the guys who are in charge of our machines. Their setup seems similar to yours, Ezra. When I ssh to a machine, I end up on 1 of (n) machines. My entire account''s file system--except a few directories--is stored on the SAN (at least this is how I understand it). The non SAN directories are symbolic links to the specific machine that I''m logged on to. So in my situation the directory /apps/local/#{account_name} will always point to the directory of a specific machine. I think I''ll be ok and have no problem giving each mongrel access to logs on its specific machine. In case, you''re interested, below is an overly detailed diagram of the setup. Thanks so much! -Steven ''%'' means filesystem is local to current machine, all other directories are on the SAN |--apps % | |--users | + | |--#{account_name} | | + | | |--bin | | |--rails | | | + | | | |--#{app_name} | | | | + | | | | |--current -> /users/#{account_name}/rails/#{app_name}/releases/rel_x | | | | | + | | | | | |--app | | | | | |--config | | | | | |--db | | | | | |--doc | | | | | |--lib | | | | | |--log -> /apps/local/#{account_name}/mongrel/log | | | | | |--public | | | | | |--script | | | | | |--test | | | | | |--tmp | | | | | | + | | | | | | |--pids -> /apps/local/#{account_name}/mongrel/pids % | | | | | | | | | | | |--vendor | | | | | | | | | |--releases | | | | | + | | | | | |--rel_1 | | | | | |--rel_2 | | | | | |--rel_3 | | | | | | | | | | |--apache2 | | | + | | | |--conf | | | |--lib | | | |--libexec -> /opt/csw/apache2/libexec % | | | |--logs -> /apps/local/#{account_name}/apache2/logs % | | | |--bin -> /opt/csw/apache2/bin % | | | | | |--include | | |--var | | |--local -> /apps/local/#{account_name} % | | | + | | | |--apache2 % | | | | + | | | | |--logs % | | | | |--tmp % | | | | | | | |--mongrel % | | | | + | | | | |--log % | | | | |--pids % | | | |
Are you suggesting that you have a separate mongrel.8001.log, mongrel.8002.log like the pids files or is there a way to somehow log like production-8001.log, production-8002.log through a mongrel setting? Having separate log files for the mongrel.log seems like overkill to me, since when my app is running, it''s not generating very many entries there... If you can create rails production logs named with the pid, could someone point me in a direction on how to set that up? Thanks.> I should probably just make this the norm, but I advocate that people setup pid and log files separate for each port they have, and probable each machine:port combination. This makes it easier to isolate one mongrel''s stuff and to track what is happening to a particular instance. > > If you do this it also means you can share it all you like.
On Thu, 30 Nov 2006 21:14:41 -0600 "Joey Geiger" <jgeiger at gmail.com> wrote:> Are you suggesting that you have a separate mongrel.8001.log, > mongrel.8002.log like the pids files or is there a way to somehow log > like production-8001.log, production-8002.log through a mongrel > setting? > > Having separate log files for the mongrel.log seems like overkill to > me, since when my app is running, it''s not generating very many > entries there... > > If you can create rails production logs named with the pid, could > someone point me in a direction on how to set that up? Thanks.It varies for different people. The norm is that Mongrel is very quiet except when it needs to tell you about how it''s configured and when things are going bad. In most situations people can safely ignore the mongrel logs and be happy. When things start going bad though you''ll run into situations where you can''t isolate which mongrel instance is doing it. In one configuration I actually configured the log files to be: mongrel-hostname-port.log for each instance, and the PID files as mongrel-hostname-port.pid. This was a customer server so changing it was cake, but you can accomplish the same thing with options. So, for most folks one log file is enough. For the paranoid and the broken creating separate logs per mongrel is necessary. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.