Hi Folks, I''ve been getting reports of people with serious stability problems in Mongrel. These problems go away instantly when they remove ruby-sendfile. If your mongrel starts up saying that you have "sendfile installed" and that it is using it then **you are wrong**. Remove the gem and don''t use it. I''ll be investigating why it doesn''t really work as advertised, and really it''s not worth the hassle for just a 20% perf boost on static files (and ONLY static files, no other requests at all). Really, if you need fast static files use a real web server instead. Thanks! P.S. Almost through my e-mail queue. Man. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.railsmachine.com/ -- Need Mongrel support?
On 03/08/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> I''ve been getting reports of people with serious stability problems in > Mongrel. These problems go away instantly when they remove > ruby-sendfile.You may want to nudge Coda to update his post [1] - I''d never heard of sendfile until I read that, and it''s often referred to in the community. [1] http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/ Cheers, -- Dave Murphy (Schwuk) http://schwuk.com
On Thu, 2006-08-03 at 08:13 +0100, Dave Murphy wrote:> On 03/08/06, Zed Shaw <zedshaw at zedshaw.com> wrote: > > I''ve been getting reports of people with serious stability problems in > > Mongrel. These problems go away instantly when they remove > > ruby-sendfile. > > You may want to nudge Coda to update his post [1] - I''d never heard of > sendfile until I read that, and it''s often referred to in the > community. > > [1] http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/Thanks, contacted him. That seems to be the source. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.railsmachine.com/ -- Need Mongrel support?
On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> I''ll be investigating why it doesn''t really work as advertised, and > really it''s not worth the hassle for just a 20% perf boost on static > files (and ONLY static files, no other requests at all). Really, if you > need fast static files use a real web server instead.In my case, most of my content is static and the performance boost is very significant. I''m using file_column to manage the uploads, so using a "real web server" around this is just a pain. I haven''t noticed any stability problems at all either. Is this something that people notice right away (perhaps due to something else in their environment), or will it sneak up on me? -- James
On Thu, 2006-08-03 at 07:53 -0500, James Ludlow wrote:> On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote: > > I''ll be investigating why it doesn''t really work as advertised, and > > really it''s not worth the hassle for just a 20% perf boost on static > > files (and ONLY static files, no other requests at all). Really, if you > > need fast static files use a real web server instead. > > In my case, most of my content is static and the performance boost is > very significant. I''m using file_column to manage the uploads, so > using a "real web server" around this is just a pain. >Measure the performance of both (with sendfile and without), you may find it''s not that great (20% is a high estimate).> I haven''t noticed any stability problems at all either. Is this > something that people notice right away (perhaps due to something else > in their environment), or will it sneak up on me?It sneaks up on you. Just one day you hit the right load and the OS starts canceling client requests at weird moments and Mongrel crashes for no reason. You''re on your own now man. :-) -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.railsmachine.com/ -- Need Mongrel support?
On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> On Thu, 2006-08-03 at 07:53 -0500, James Ludlow wrote: > > On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote: > > > I''ll be investigating why it doesn''t really work as advertised, and > > > really it''s not worth the hassle for just a 20% perf boost on static > > > files (and ONLY static files, no other requests at all). Really, if you > > > need fast static files use a real web server instead. > > > > In my case, most of my content is static and the performance boost is > > very significant. I''m using file_column to manage the uploads, so > > using a "real web server" around this is just a pain. > > > > Measure the performance of both (with sendfile and without), you may > find it''s not that great (20% is a high estimate).Setting the performance issue aside for a moment, if I have a long download (i.e. streaming video), I''m going to tie up a mongrel process for one user for a relatively long time. This isn''t a crisis, because I can just spawn more, and I''m more constrained by the bandwidth at the switch than by memory or CPU time. My use of the word "performance" was pretty vague and imprecise. I''ll try it without sendfile, but with more mongrel processes and see what happens.> > I haven''t noticed any stability problems at all either. Is this > > something that people notice right away (perhaps due to something else > > in their environment), or will it sneak up on me? > > It sneaks up on you. Just one day you hit the right load and the OS > starts canceling client requests at weird moments and Mongrel crashes > for no reason. > > You''re on your own now man. :-)I appreciate the warning. With what seems like a sudden influx of stability problems and now this announcement about sendfile, I''m wondering if something else might be going on though. I''m not saying that ruby-sendfile isn''t broken, but maybe something else changed recently that exposed this bug. -- James
On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> > You may want to nudge Coda to update his post [1] - I''d never heard of > > sendfile until I read that, and it''s often referred to in the > > community. > > > > [1] http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/ > > Thanks, contacted him. That seems to be the source.Nudged! Sorry for the problems--I figured it was a decent CYA maneuver if my proxying config let stuff through (which it did, at first), but I haven''t experienced any problems with it. Either way, my post has been updated with the current recommendation to stay the hell away from sendfile. Thanks for the heads-up, Zed. -- Coda Hale http://blog.codahale.com
On Thu, 2006-08-03 at 09:40 -0500, James Ludlow wrote:> On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote:<snip>> Setting the performance issue aside for a moment, if I have a long > download (i.e. streaming video), I''m going to tie up a mongrel process > for one user for a relatively long time. This isn''t a crisis, because > I can just spawn more, and I''m more constrained by the bandwidth at > the switch than by memory or CPU time. >I think you''re misunderstanding the purpose of install ruby-sendfile. Even *without* sendfile installed Mongrel won''t block on sending a file. It''ll still chug away on multiple concurrent files without stopping anything else. There''s a limit of course, but it''s not locked like with Rails. What sendfile does is this: http://www.die.net/doc/linux/man/man2/sendfile.2.html It gives ruby the sendfile API. Problem with the sendfile API is that''s it''s implemented wildly differently on all the OS and even different versions of the OS. On some it blocks. On others it can throw EINTERRUPT. On still others it doesn''t even do zero-copy. This is the main reason it causes instability in Mongrel.> My use of the word "performance" was pretty vague and imprecise. I''ll > try it without sendfile, but with more mongrel processes and see what > happens. >You shouldn''t need more Mongrel processes. <snip>> I appreciate the warning. With what seems like a sudden influx of > stability problems and now this announcement about sendfile, I''m > wondering if something else might be going on though. I''m not saying > that ruby-sendfile isn''t broken, but maybe something else changed > recently that exposed this bug.No, what actually happened is Coda Hale put sendfile in a list of gems to install. Previously most folks weren''t installing it. Now that they''re following those docs they were just installing it arbitrarily without understanding what it did. With an increase in different people using sendfile for different apps I started getting complaints about stability. Telling them to remove sendfile fixed all their problems. Also, in every OS that I''ve tried it on the errors are related to differences in how sendfile is implemented. For example, FreeBSD was aborting sendfile with EINTERRUPT and returning the amount sent requiring that you send the remainder (which isn''t very helpful). It was also doing this on exactly 32k boundaries, but only when the server is overloaded. Anyway, let me know how it works. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.railsmachine.com/ -- Need Mongrel support?
On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> On Thu, 2006-08-03 at 09:40 -0500, James Ludlow wrote: > > On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote: > <snip> > > Setting the performance issue aside for a moment, if I have a long > > download (i.e. streaming video), I''m going to tie up a mongrel process > > for one user for a relatively long time. This isn''t a crisis, because > > I can just spawn more, and I''m more constrained by the bandwidth at > > the switch than by memory or CPU time. > > > I think you''re misunderstanding the purpose of install ruby-sendfile. > Even *without* sendfile installed Mongrel won''t block on sending a file. > It''ll still chug away on multiple concurrent files without stopping > anything else. There''s a limit of course, but it''s not locked like with > Rails.You are correct about my misunderstanding. I was mistaken about how mongrel would handle this. Why didn''t I just check it out to begin with? Laziness apparently. Plus, sendfile "seemed" to be working great.> What sendfile does is this: > > http://www.die.net/doc/linux/man/man2/sendfile.2.html > > It gives ruby the sendfile API. Problem with the sendfile API is that''s > it''s implemented wildly differently on all the OS and even different > versions of the OS. On some it blocks. On others it can throw > EINTERRUPT. On still others it doesn''t even do zero-copy. > > This is the main reason it causes instability in Mongrel.Cool, thanks.> No, what actually happened is Coda Hale put sendfile in a list of gems > to install. Previously most folks weren''t installing it. Now that > they''re following those docs they were just installing it arbitrarily > without understanding what it did.Yeah, those were the instructions that I used. It really is a nice piece of documentation, despite the current problems being found.> Anyway, let me know how it works.OK -- James
Okay. So send file is bad and has been removed. Correct me if Iam wrong , but if you have apache installed and proxying through it, you dont really need sendfile . is that assumption correct? Sunder On 8/3/06, Coda Hale <coda.hale at gmail.com> wrote:> > On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote: > > > You may want to nudge Coda to update his post [1] - I''d never heard of > > > sendfile until I read that, and it''s often referred to in the > > > community. > > > > > > [1] > http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/ > > > > Thanks, contacted him. That seems to be the source. > > Nudged! > > Sorry for the problems--I figured it was a decent CYA maneuver if my > proxying config let stuff through (which it did, at first), but I > haven''t experienced any problems with it. Either way, my post has been > updated with the current recommendation to stay the hell away from > sendfile. > > Thanks for the heads-up, Zed. > > -- > Coda Hale > http://blog.codahale.com > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060803/5891a64f/attachment.html
On Thu, 2006-08-03 at 09:08 -0700, Sunder wrote:> Okay. So send file is bad and has been removed. Correct me if Iam > wrong , but if you have apache installed and proxying through it, you > dont really need sendfile . is that assumption correct?You are correct! -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.railsmachine.com/ -- Need Mongrel support?
> Okay. So send file is bad and has been removed. Correct me if Iam wrong , > but if you have apache installed and proxying through it, you dont really > need sendfile . is that assumption correct?Depends on how you''re proxying I think... If you pass everything to mongrel, well, everything will get handled by mongrel. If you hadd those re-write rules that check file existence thatn probably not as important. We''re running 0.3.13 on centos with sendfile doing a *lot* of traffic, and not seeing any issues... although I wouldn''t be surprised if sendfile was never being activated as we''ve got those rewrite rules in place for existing files... -philip> > Sunder > > On 8/3/06, Coda Hale <coda.hale at gmail.com> wrote: >> >> On 8/3/06, Zed Shaw <zedshaw at zedshaw.com> wrote: >> > > You may want to nudge Coda to update his post [1] - I''d never heard of >> > > sendfile until I read that, and it''s often referred to in the >> > > community. >> > > >> > > [1] >> http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/ >> > >> > Thanks, contacted him. That seems to be the source. >> >> Nudged! >> >> Sorry for the problems--I figured it was a decent CYA maneuver if my >> proxying config let stuff through (which it did, at first), but I >> haven''t experienced any problems with it. Either way, my post has been >> updated with the current recommendation to stay the hell away from >> sendfile. >> >> Thanks for the heads-up, Zed. >> >> -- >> Coda Hale >> http://blog.codahale.com >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> >
On 8/3/06, Sunder <talk2sunder at gmail.com> wrote:> Okay. So send file is bad and has been removed. Correct me if Iam wrong , > but if you have apache installed and proxying through it, you dont really > need sendfile . is that assumption correct?It depends on how you set up the proxy. In general, no, everything will go through mongrel, you would have to either explicitly map only certain paths to mongrel (for anything that''s not static) or explicitly exclude things that are static (can''t recall how or even if this is possible with mod_proxy using filetypes, but you can always use mod_rewrite to do more complicated stuff). Thomas
Just curious, if you are really need the "performance" for static files, why not use an asset_host ? i haven''t personally used it, nor do I fully know what it does, but on a cursory, half-hazard glance, it appears you can serve up content from another web server (lightty, apache, etc.) here for all your other stuff: http://asset.yourapp.com/images/zed_rules.gif just tossin'' it out there.... On 8/3/06, Thomas Lockney <tlockney at gmail.com> wrote:> On 8/3/06, Sunder <talk2sunder at gmail.com> wrote: > > Okay. So send file is bad and has been removed. Correct me if Iam wrong , > > but if you have apache installed and proxying through it, you dont really > > need sendfile . is that assumption correct? > > It depends on how you set up the proxy. In general, no, everything > will go through mongrel, you would have to either explicitly map only > certain paths to mongrel (for anything that''s not static) or > explicitly exclude things that are static (can''t recall how or even if > this is possible with mod_proxy using filetypes, but you can always > use mod_rewrite to do more complicated stuff). > > Thomas > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-- Charles Brian Quinn self-promotion: www.seebq.com highgroove studios: www.highgroove.com slingshot hosting: www.slingshothosting.com
> Just curious, if you are really need the "performance" for static > files, why not use an asset_host ? i haven''t personally used it, nor > do I fully know what it does, but on a cursory, half-hazard glance, it > appears you can serve up content from another web server (lightty, > apache, etc.) here for all your other stuff: > > http://asset.yourapp.com/images/zed_rules.gif > > just tossin'' it out there....Because sometimes you work with people who aren''t good about using the various _tag method that make this work... heh. but yeah, it would help.> > On 8/3/06, Thomas Lockney <tlockney at gmail.com> wrote: >> On 8/3/06, Sunder <talk2sunder at gmail.com> wrote: >>> Okay. So send file is bad and has been removed. Correct me if Iam wrong , >>> but if you have apache installed and proxying through it, you dont really >>> need sendfile . is that assumption correct? >> >> It depends on how you set up the proxy. In general, no, everything >> will go through mongrel, you would have to either explicitly map only >> certain paths to mongrel (for anything that''s not static) or >> explicitly exclude things that are static (can''t recall how or even if >> this is possible with mod_proxy using filetypes, but you can always >> use mod_rewrite to do more complicated stuff). >> >> Thomas >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > > > -- > Charles Brian Quinn > self-promotion: www.seebq.com > highgroove studios: www.highgroove.com > slingshot hosting: www.slingshothosting.com > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Possibly Parallel Threads
- Mongrel proxied through Apache won''t serve images
- Strange local requests with Apache 2.2, mod_proxy_balancer, and Mongrel 0.3.13.2
- The Debian Plan
- New automated Capistrano setup for Apache2.1+ and mongrel_cluster
- [SEC] Mongrel Temporary Fix For cgi.rb 99% CPU DoS Attack