Mongrel users - May 2008 - Unix Domain Sockets + Fork for improved scalability

Hello All,

Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
changes to mongrel that I''ve written on behalf of
my employer, Raritan Computer, Inc. Also, these changes are released
under the existing Ruby license used for Mongrel.

Raritan is using mongrel internally for product development and was
not able to get the level of scalability desired due to limitations
Mongrel/Rails/Ruby. In particular the single threaded nature of
processing rails request is a serious bottleneck when one or more long
running requests could occur. Originally we used mongrel cluster to
spin up a small number of mongrel rails instance; Apache mod_proxy was
used to hand off requests to the various instances of mongrel.

Unfortunately a long running request could cause our entire web
application to "hang".  As these requests are being processing one or
more mongrel instances continued to accept and enqueue new connections
it can''t immediately service.

Mongrel cluster didn''t provide any features for dynamically sizing the
number of mongrel_rails instances necessary to keep the application
running smoothly. Also, mongrels nature of continuing to enqueue
connections while processing a long running request was a problem for
our application. In essence we needed a smarter mongrel that would
know which instances of a "cluster" were busy or free; if none were
free then it should spin up a new instance.

The attached patch does just that. It''s definitely tailored for a Unix
derivative platform and depends on the functionality of the UNIXSocket
class to pass file descriptors between instances. It works in a
master-slave type of server setup. The "master" is the initial
instance of mongrel rails which goes through its normal initialization
gyrations before forking mulitple "slave" instances to do the actual
rails processing. The server listens on the defined TCP/IP address and
port accepting request (from Apache in our case), determines which
child if any is available, and then hands off the file descriptor for
processing.

The number of children can dynamically grow and shrink depending on
how busy the application is being kept. The patch has fixed our issue
with having long running requests hang the web application.

The patch isn''t without any warts. Due to the way that mongrel
preforms initialization I found it difficult to follow the startup
logic. Because of the convoluted
contortions that occur during startup I was unable to
make deeper changes due to time constraints which means that the
following problem exists.

* If you start the server as root and the configuration causes a
setuid/setgid call to a non-root user then during shutdown mongrel may
not be able to remove its pid file.

I''m not aware of any other lurking problem, but I wouldn''t be
surprised if one or more exists.

If you have any further question just send me a note or post to the
list. I''ll stay subscribed for a few weeks at least. Also, for the
maintainers: I''ve added a copyright notice for Raritan Computer on
those files that had extensive changes. I believe that is the correct
way to handle major changes, let me know if I''m incorrect in my
understanding.

Enjoy

-- Brian Weaver

*sigh* here is the rest of the text I''m required to include

THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.



-- 

/* insert witty comment here */

Hey Brian-

	This is pretty kick ass, I''ll definitely take a look.

-Ezra	

On May 5, 2008, at 8:37 PM, Brian Weaver wrote:
> Hello All,
>
> Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
> changes to mongrel that I''ve written on behalf of
> my employer, Raritan Computer, Inc. Also, these changes are released
> under the existing Ruby license used for Mongrel.
>
> Raritan is using mongrel internally for product development and was
> not able to get the level of scalability desired due to limitations
> Mongrel/Rails/Ruby. In particular the single threaded nature of
> processing rails request is a serious bottleneck when one or more long
> running requests could occur. Originally we used mongrel cluster to
> spin up a small number of mongrel rails instance; Apache mod_proxy was
> used to hand off requests to the various instances of mongrel.
>
> Unfortunately a long running request could cause our entire web
> application to "hang".  As these requests are being processing
one or
> more mongrel instances continued to accept and enqueue new connections
> it can''t immediately service.
>
> Mongrel cluster didn''t provide any features for dynamically sizing
the
> number of mongrel_rails instances necessary to keep the application
> running smoothly. Also, mongrels nature of continuing to enqueue
> connections while processing a long running request was a problem for
> our application. In essence we needed a smarter mongrel that would
> know which instances of a "cluster" were busy or free; if none
were
> free then it should spin up a new instance.
>
> The attached patch does just that. It''s definitely tailored for a
Unix
> derivative platform and depends on the functionality of the UNIXSocket
> class to pass file descriptors between instances. It works in a
> master-slave type of server setup. The "master" is the initial
> instance of mongrel rails which goes through its normal initialization
> gyrations before forking mulitple "slave" instances to do the
actual
> rails processing. The server listens on the defined TCP/IP address and
> port accepting request (from Apache in our case), determines which
> child if any is available, and then hands off the file descriptor for
> processing.
>
> The number of children can dynamically grow and shrink depending on
> how busy the application is being kept. The patch has fixed our issue
> with having long running requests hang the web application.
>
> The patch isn''t without any warts. Due to the way that mongrel
> preforms initialization I found it difficult to follow the startup
> logic. Because of the convoluted
> contortions that occur during startup I was unable to
> make deeper changes due to time constraints which means that the
> following problem exists.
>
> * If you start the server as root and the configuration causes a
> setuid/setgid call to a non-root user then during shutdown mongrel may
> not be able to remove its pid file.
>
> I''m not aware of any other lurking problem, but I
wouldn''t be
> surprised if one or more exists.
>
> If you have any further question just send me a note or post to the
> list. I''ll stay subscribed for a few weeks at least. Also, for the
> maintainers: I''ve added a copyright notice for Raritan Computer on
> those files that had extensive changes. I believe that is the correct
> way to handle major changes, let me know if I''m incorrect in my
> understanding.
>
> Enjoy
>
> -- Brian Weaver
>
> *sigh* here is the rest of the text I''m required to include
>
> THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
IMPLIED
> WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
> MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
> -- 
>
> /* insert witty comment here */
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

Thanks....

I was just reviewing the patch and I''ve notice several spelling errors
(and grammar problems) in my log messages that need to be fixed. Also,
there isn''t any logic handling any kind of IO error on the TCP Server
socket for the UnixDispatchServer. That could be a problem, but I
don''t really expect any IO errors on the actual TCP socket for
accepting incoming request from Apache (mod_proxy).

I''ll try to review the code changes again after I get some sleep (I
really should have been in bed hours ago) and submit a new patch with
any fixes that I make; cosmetic or otherwise.

-- Weave

On Tue, May 6, 2008 at 12:30 AM, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
> Hey Brian-
>
>         This is pretty kick ass, I''ll definitely take a look.
>
>  -Ezra
>
>
>  On May 5, 2008, at 8:37 PM, Brian Weaver wrote:
>
> >
> >
> >
> > Hello All,
> >
> > Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
> > changes to mongrel that I''ve written on behalf of
> > my employer, Raritan Computer, Inc. Also, these changes are released
> > under the existing Ruby license used for Mongrel.
> >
> > Raritan is using mongrel internally for product development and was
> > not able to get the level of scalability desired due to limitations
> > Mongrel/Rails/Ruby. In particular the single threaded nature of
> > processing rails request is a serious bottleneck when one or more long
> > running requests could occur. Originally we used mongrel cluster to
> > spin up a small number of mongrel rails instance; Apache mod_proxy was
> > used to hand off requests to the various instances of mongrel.
> >
> > Unfortunately a long running request could cause our entire web
> > application to "hang".  As these requests are being
processing one or
> > more mongrel instances continued to accept and enqueue new connections
> > it can''t immediately service.
> >
> > Mongrel cluster didn''t provide any features for dynamically
sizing the
> > number of mongrel_rails instances necessary to keep the application
> > running smoothly. Also, mongrels nature of continuing to enqueue
> > connections while processing a long running request was a problem for
> > our application. In essence we needed a smarter mongrel that would
> > know which instances of a "cluster" were busy or free; if
none were
> > free then it should spin up a new instance.
> >
> > The attached patch does just that. It''s definitely tailored
for a Unix
> > derivative platform and depends on the functionality of the UNIXSocket
> > class to pass file descriptors between instances. It works in a
> > master-slave type of server setup. The "master" is the
initial
> > instance of mongrel rails which goes through its normal initialization
> > gyrations before forking mulitple "slave" instances to do
the actual
> > rails processing. The server listens on the defined TCP/IP address and
> > port accepting request (from Apache in our case), determines which
> > child if any is available, and then hands off the file descriptor for
> > processing.
> >
> > The number of children can dynamically grow and shrink depending on
> > how busy the application is being kept. The patch has fixed our issue
> > with having long running requests hang the web application.
> >
> > The patch isn''t without any warts. Due to the way that
mongrel
> > preforms initialization I found it difficult to follow the startup
> > logic. Because of the convoluted
> > contortions that occur during startup I was unable to
> > make deeper changes due to time constraints which means that the
> > following problem exists.
> >
> > * If you start the server as root and the configuration causes a
> > setuid/setgid call to a non-root user then during shutdown mongrel may
> > not be able to remove its pid file.
> >
> > I''m not aware of any other lurking problem, but I
wouldn''t be
> > surprised if one or more exists.
> >
> > If you have any further question just send me a note or post to the
> > list. I''ll stay subscribed for a few weeks at least. Also,
for the
> > maintainers: I''ve added a copyright notice for Raritan
Computer on
> > those files that had extensive changes. I believe that is the correct
> > way to handle major changes, let me know if I''m incorrect in
my
> > understanding.
> >
> > Enjoy
> >
> > -- Brian Weaver
> >
> > *sigh* here is the rest of the text I''m required to include
> >
> > THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
IMPLIED
> > WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
> > MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
> >
> >
> >
> > --
> >
> > /* insert witty comment here */
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
>  _______________________________________________
>  Mongrel-users mailing list
>  Mongrel-users at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 

/* insert witty comment here */

How does this work with the AR database connections? Are they closed  
and reopened in the children? Or do the children load rails after they  
fork?

-Ezra



On May 5, 2008, at 10:36 PM, Brian Weaver wrote:
> Thanks....
>
> I was just reviewing the patch and I''ve notice several spelling
errors
> (and grammar problems) in my log messages that need to be fixed. Also,
> there isn''t any logic handling any kind of IO error on the TCP
Server
> socket for the UnixDispatchServer. That could be a problem, but I
> don''t really expect any IO errors on the actual TCP socket for
> accepting incoming request from Apache (mod_proxy).
>
> I''ll try to review the code changes again after I get some sleep
(I
> really should have been in bed hours ago) and submit a new patch with
> any fixes that I make; cosmetic or otherwise.
>
> -- Weave
>
> On Tue, May 6, 2008 at 12:30 AM, Ezra Zygmuntowicz  
> <ezmobius at gmail.com> wrote:
>> Hey Brian-
>>
>>        This is pretty kick ass, I''ll definitely take a look.
>>
>> -Ezra
>>
>>
>> On May 5, 2008, at 8:37 PM, Brian Weaver wrote:
>>
>>>
>>>
>>>
>>> Hello All,
>>>
>>> Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
>>> changes to mongrel that I''ve written on behalf of
>>> my employer, Raritan Computer, Inc. Also, these changes are
released
>>> under the existing Ruby license used for Mongrel.
>>>
>>> Raritan is using mongrel internally for product development and was
>>> not able to get the level of scalability desired due to limitations
>>> Mongrel/Rails/Ruby. In particular the single threaded nature of
>>> processing rails request is a serious bottleneck when one or more  
>>> long
>>> running requests could occur. Originally we used mongrel cluster to
>>> spin up a small number of mongrel rails instance; Apache mod_proxy
>>> was
>>> used to hand off requests to the various instances of mongrel.
>>>
>>> Unfortunately a long running request could cause our entire web
>>> application to "hang".  As these requests are being
processing one
>>> or
>>> more mongrel instances continued to accept and enqueue new  
>>> connections
>>> it can''t immediately service.
>>>
>>> Mongrel cluster didn''t provide any features for
dynamically sizing
>>> the
>>> number of mongrel_rails instances necessary to keep the application
>>> running smoothly. Also, mongrels nature of continuing to enqueue
>>> connections while processing a long running request was a problem  
>>> for
>>> our application. In essence we needed a smarter mongrel that would
>>> know which instances of a "cluster" were busy or free; if
none were
>>> free then it should spin up a new instance.
>>>
>>> The attached patch does just that. It''s definitely
tailored for a
>>> Unix
>>> derivative platform and depends on the functionality of the  
>>> UNIXSocket
>>> class to pass file descriptors between instances. It works in a
>>> master-slave type of server setup. The "master" is the
initial
>>> instance of mongrel rails which goes through its normal  
>>> initialization
>>> gyrations before forking mulitple "slave" instances to do
the actual
>>> rails processing. The server listens on the defined TCP/IP address
>>> and
>>> port accepting request (from Apache in our case), determines which
>>> child if any is available, and then hands off the file descriptor  
>>> for
>>> processing.
>>>
>>> The number of children can dynamically grow and shrink depending on
>>> how busy the application is being kept. The patch has fixed our  
>>> issue
>>> with having long running requests hang the web application.
>>>
>>> The patch isn''t without any warts. Due to the way that
mongrel
>>> preforms initialization I found it difficult to follow the startup
>>> logic. Because of the convoluted
>>> contortions that occur during startup I was unable to
>>> make deeper changes due to time constraints which means that the
>>> following problem exists.
>>>
>>> * If you start the server as root and the configuration causes a
>>> setuid/setgid call to a non-root user then during shutdown mongrel
>>> may
>>> not be able to remove its pid file.
>>>
>>> I''m not aware of any other lurking problem, but I
wouldn''t be
>>> surprised if one or more exists.
>>>
>>> If you have any further question just send me a note or post to the
>>> list. I''ll stay subscribed for a few weeks at least. Also,
for the
>>> maintainers: I''ve added a copyright notice for Raritan
Computer on
>>> those files that had extensive changes. I believe that is the  
>>> correct
>>> way to handle major changes, let me know if I''m incorrect
in my
>>> understanding.
>>>
>>> Enjoy
>>>
>>> -- Brian Weaver
>>>
>>> *sigh* here is the rest of the text I''m required to
include
>>>
>>> THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS
OR IMPLIED
>>> WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES
OF
>>> MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
>>>
>>>
>>>
>>> --
>>>
>>> /* insert witty comment here */
>>> _______________________________________________
>>> Mongrel-users mailing list
>>> Mongrel-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>
>>
>>
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>
>
>
> -- 
>
> /* insert witty comment here */
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com

On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver
wrote:
> Hello All,
> 
> Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
> changes to mongrel that I''ve written on behalf of
> my employer, Raritan Computer, Inc. Also, these changes are released
> under the existing Ruby license used for Mongrel.
Interesting Brian, glad you finally got the patch into the queue. :-)
> The attached patch does just that. It''s definitely tailored for a
Unix
> derivative platform and depends on the functionality of the UNIXSocket
> class to pass file descriptors between instances. It works in a
> master-slave type of server setup. The "master" is the initial
> instance of mongrel rails which goes through its normal initialization
> gyrations before forking mulitple "slave" instances to do the
actual
> rails processing. The server listens on the defined TCP/IP address and
> port accepting request (from Apache in our case), determines which
> child if any is available, and then hands off the file descriptor for
> processing.
So, I''ve just read through this patch, and correct me if I''m
wrong, but
your patch removes the ability for Mongrel to run as a plain web server
right?  There also seems to be no tests, so does it just use the
existing tests but within the confines of the new forking setup?
 
> The patch isn''t without any warts. Due to the way that mongrel
> preforms initialization I found it difficult to follow the startup
> logic. Because of the convoluted
> contortions that occur during startup I was unable to
> make deeper changes due to time constraints which means that the
> following problem exists.
Can you describe what was confusing?  The code''s not too large so maybe
it just wasn''t documented really clearly.

Also, had you tried just subclassing HttpServer and implementing the
alternative mechanism for both the master and slaves?
 
> * If you start the server as root and the configuration causes a
> setuid/setgid call to a non-root user then during shutdown mongrel may
> not be able to remove its pid file.
The pid file management has always been a mess because of ruby''s very
poor process management, and just the variablility in POSIX anyway.
Eventually it''ll just have to be a C api that does it right I think.
> I''m not aware of any other lurking problem, but I
wouldn''t be
> surprised if one or more exists.
> 
> If you have any further question just send me a note or post to the
> list. I''ll stay subscribed for a few weeks at least. Also, for the
> maintainers: I''ve added a copyright notice for Raritan Computer on
> those files that had extensive changes. I believe that is the correct
> way to handle major changes, let me know if I''m incorrect in my
> understanding.
Actually, and this is a big misnomer among open source folks, the
copyright isn''t on a file, it''s on the project.  It''s
kind of like
saying if you edit a page in my book with a marker you get to put your
copyright on that page.  Additionally there isn''t enough change to that
file to warrant an entire copyright change as it''s still primarily
written by myself and other project contributors.

What you can do is either officially donate it back to the project under
the original copyright with a small notice giving credit to yourself
and/or Raritan that you did the master/slave/unixsocket hack, or pull
this out into a separate, completely and wholely written by you/raritan
file that has none of my code and then release under your own license
however you like.

Talk it over with your corporate masters, but considering you have an
entire product based on other people''s generously donated free work it
might be a really great nice thing to give back.

Makes people all warm and fuzzy and like your products in exchange.

---
Evil: http://www.zedshaw.com/
Utu : http://savingtheinternetwithhate.com/

On Tue, May 06, 2008 at 01:36:32AM -0400, Brian Weaver
wrote:
> Thanks....
> 
> I was just reviewing the patch and I''ve notice several spelling
errors
> (and grammar problems) in my log messages that need to be fixed. Also,
> there isn''t any logic handling any kind of IO error on the TCP
Server
> socket for the UnixDispatchServer. That could be a problem, but I
> don''t really expect any IO errors on the actual TCP socket for
> accepting incoming request from Apache (mod_proxy).
Well, experience on the project has shown that the second you toss this
thing on FreeBSD you get a whole new cluster of IOError you never knew
existed.  Then put it on OSX and you get 10 more.  Each all an exception
of wonderful clarity.

I''m pretty sure as people try this out you''ll find them all
though.
 
> I''ll try to review the code changes again after I get some sleep
(I
> really should have been in bed hours ago) and submit a new patch with
> any fixes that I make; cosmetic or otherwise.
No! Work now! Sleep later! :-)
 
---
Evil: http://www.zedshaw.com/
Utu : http://savingtheinternetwithhate.com/

On Tue, May 6, 2008 at 1:59 AM, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
>  Actually, and this is a big misnomer among open source folks, the
>  copyright isn''t on a file, it''s on the project. 
It''s kind of like
>  saying if you edit a page in my book with a marker you get to put your
>  copyright on that page.  Additionally there isn''t enough change
to that
>  file to warrant an entire copyright change as it''s still
primarily
>  written by myself and other project contributors.
>
>  What you can do is either officially donate it back to the project under
>  the original copyright with a small notice giving credit to yourself
>  and/or Raritan that you did the master/slave/unixsocket hack, or pull
>  this out into a separate, completely and wholely written by you/raritan
>  file that has none of my code and then release under your own license
>  however you like.
We haven''t required copyright assignment from any of the other
contributors, so there''s no way we could relicense anyway without
stripping out their code. So I think it is fine for Raritan to keep
the original copyright, since it''s under the same license.

I agree that there''s no point in the additional copyright notice,
though; Raritan is just one of the many (some anonymous now)
contributors.
>
>  Talk it over with your corporate masters, but considering you have an
>  entire product based on other people''s generously donated free
work it
>  might be a really great nice thing to give back.
>
>  Makes people all warm and fuzzy and like your products in exchange.
>
>  ---
>  Evil: http://www.zedshaw.com/
>  Utu : http://savingtheinternetwithhate.com/
>
>
>
>  _______________________________________________
>  Mongrel-users mailing list
>  Mongrel-users at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
Evan Weaver

See the comments embedded.....

On Tue, May 6, 2008 at 1:59 AM, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
>  > Hello All,
>  >
>  > Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
>  > changes to mongrel that I''ve written on behalf of
>  > my employer, Raritan Computer, Inc. Also, these changes are released
>  > under the existing Ruby license used for Mongrel.
>
>  Interesting Brian, glad you finally got the patch into the queue. :-)
>
>
>  > The attached patch does just that. It''s definitely tailored
for a Unix
>  > derivative platform and depends on the functionality of the
UNIXSocket
>  > class to pass file descriptors between instances. It works in a
>  > master-slave type of server setup. The "master" is the
initial
>  > instance of mongrel rails which goes through its normal
initialization
>  > gyrations before forking mulitple "slave" instances to do
the actual
>  > rails processing. The server listens on the defined TCP/IP address
and
>  > port accepting request (from Apache in our case), determines which
>  > child if any is available, and then hands off the file descriptor for
>  > processing.
>
>  So, I''ve just read through this patch, and correct me if
I''m wrong, but
>  your patch removes the ability for Mongrel to run as a plain web server
>  right?  There also seems to be no tests, so does it just use the
>  existing tests but within the confines of the new forking setup?
No, the patch should not remove the ability for mongel to run a a
plain web server. I basically split the HttpServer class into two
classes: Server and HttpServer. I attempted to move most of what would
be common code between the existing HttpServer class and the new
UnixDispatchServer class into the base Server class. If you specify
''--unixmaster'' or put ''unixmaster = true'' in
the mongrel yaml
configuration file then it runs an instance of the UnixDispatchServer;
otherwise it should run the default HttpServer as it was originally
written to do,

Your right there are currently no test. Yes you can flame me all you
want for that; I''ll get to it when I have time. I figured in the
spirit of most open source projects I would go ahead and "release
early" to let people play with it so that I could get feedback. The
primary development platform was Linux (CentOS 4 actually) and I
tested it heavily with our application before I posted it. It''s far
from perfect, but it is a start.

Besides other people might feel like contributing, I wouldn''t want to
deprive them of a decent opportunity by writing all the test myself.
That would seem kind of greedy. ;-)
>
>
>  > The patch isn''t without any warts. Due to the way that
mongrel
>  > preforms initialization I found it difficult to follow the startup
>  > logic. Because of the convoluted
>  > contortions that occur during startup I was unable to
>  > make deeper changes due to time constraints which means that the
>  > following problem exists.
>
>  Can you describe what was confusing?  The code''s not too large so
maybe
>  it just wasn''t documented really clearly.
>
>  Also, had you tried just subclassing HttpServer and implementing the
>  alternative mechanism for both the master and slaves?
>
>
>  > * If you start the server as root and the configuration causes a
>  > setuid/setgid call to a non-root user then during shutdown mongrel
may
>  > not be able to remove its pid file.
>
>  The pid file management has always been a mess because of ruby''s
very
>  poor process management, and just the variablility in POSIX anyway.
>  Eventually it''ll just have to be a C api that does it right I
think.
>
Almost all systems support the ''unlink'' command. At issues is
when you
drop privilege levels from root after creating a pid file then it''s
almost impossible to remove. I''ve gotten around this issue by using a
service script (/etc/init.d/mongrel_rails) on our system to check for
the existence of a pid file, ensure that the process really isn''t
running, and then removing the pid before starting up mongrel. It''s
not an insurmountable problem, I just figured I would state it up
front to save other developers and users some time.

One could always change the code to save the privilege level allowing
for a new call to setuid/setgid, but this could come back to bite if
the process has a vulnerability. It''s a small problem to do the checks
before starting or after stopping mongrel for the added security of
running as a non-privileged user.
>
>  > I''m not aware of any other lurking problem, but I
wouldn''t be
>  > surprised if one or more exists.
>  >
>  > If you have any further question just send me a note or post to the
>  > list. I''ll stay subscribed for a few weeks at least. Also,
for the
>  > maintainers: I''ve added a copyright notice for Raritan
Computer on
>  > those files that had extensive changes. I believe that is the correct
>  > way to handle major changes, let me know if I''m incorrect in
my
>  > understanding.
>
>  Actually, and this is a big misnomer among open source folks, the
>  copyright isn''t on a file, it''s on the project. 
It''s kind of like
>  saying if you edit a page in my book with a marker you get to put your
>  copyright on that page.  Additionally there isn''t enough change
to that
>  file to warrant an entire copyright change as it''s still
primarily
>  written by myself and other project contributors.
I''m not a copyright lawyer, but I bet if you talk to a few you would
get some variance on how it really should be done. The point is the
work I''ve done has been for Raritan. If the changes are such that
copyright law would assign the ownership, for all or part of the
changes, to Raritan then that''s who owns it. Raritan is simply trying
to follow the Ruby license that Mongrel is using....

Specifically section 2a of the license at
http://www.ruby-lang.org/en/LICENSE.txt
>
>  What you can do is either officially donate it back to the project under
>  the original copyright with a small notice giving credit to yourself
>  and/or Raritan that you did the master/slave/unixsocket hack, or pull
>  this out into a separate, completely and wholely written by you/raritan
>  file that has none of my code and then release under your own license
>  however you like.
>
>  Talk it over with your corporate masters, but considering you have an
>  entire product based on other people''s generously donated free
work it
>  might be a really great nice thing to give back.
>
>  Makes people all warm and fuzzy and like your products in exchange.
I know all about warm and fuzzy, giving to the community, paying for
the privilege, and being out of work with no income at the end of the
day. Luckily the work lives own and my previous company was able to
find someone to keep the open source fork going while we tried our
commercial endeavor. The commercial endeavor was a bust, but that''s
life. If you have any interest in network management then you should
hop on over to OpenNMS (http://www.opennms.org/) and you can see my
first dabbling with open source.

-- Weave

-- 

/* insert witty comment here */

> Mongrel cluster didn''t provide any features for dynamically sizing
the
> number of mongrel_rails instances necessary to keep the application
> running smoothly. Also, mongrels nature of continuing to enqueue
> connections while processing a long running request was a problem for
> our application. In essence we needed a smarter mongrel that would
> know which instances of a "cluster" were busy or free; if none
were
> free then it should spin up a new instance.
Reminds me of swiftiply vaguely.
-- 
Posted via http://www.ruby-forum.com/.

On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver
wrote:
> Unfortunately a long running request could cause our entire web
> application to "hang".  As these requests are being processing
one or
> more mongrel instances continued to accept and enqueue new connections
> it can''t immediately service.
Another solution: 

You could stick HAproxy in front of your mongrel cluster with a the
configuration somewhat like:

  listen application *:80
      balance roundrobin
      server  mongrel0 127.0.0.1:5000 minconn 1 maxconn 1 check
      server  mongrel1 127.0.0.1:5001 minconn 1 maxconn 1 check
      server  mongrel2 127.0.0.1:5002 minconn 1 maxconn 1 check
      server  mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
      server  mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check

The ''minconn 1 maxconn 1'' will force haproxy to queue the
results within
itself instead of mongrel, and the ''check'' will take the
mongrel out of
rotation if it goes down, and add it back into the rotation as soon as
it comes back up.  As soon as one mongrel is finished with a request,
haproxy will give it one of the ones it has queued.

This is a simple version of what I''m doing to load balance clusters of
mongrels these days.

enjoy,

-jeremy

-- 
======================================================================= Jeremy
Hinegardner                              jeremy at hinegardner.org

Jeremy Hinegardner <jeremy at hinegardner.org>
wrote:
> On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
> > Unfortunately a long running request could cause our entire web
> > application to "hang".  As these requests are being
processing one or
> > more mongrel instances continued to accept and enqueue new connections
> > it can''t immediately service.
Heh, this is also the exact problem I designed qrp around, too.
> Another solution: 
> 
> You could stick HAproxy in front of your mongrel cluster with a the
> configuration somewhat like:
> 
>   listen application *:80
>       balance roundrobin
>       server  mongrel0 127.0.0.1:5000 minconn 1 maxconn 1 check
>       server  mongrel1 127.0.0.1:5001 minconn 1 maxconn 1 check
>       server  mongrel2 127.0.0.1:5002 minconn 1 maxconn 1 check
>       server  mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
>       server  mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check
> 
> The ''minconn 1 maxconn 1'' will force haproxy to queue the
results within
> itself instead of mongrel, and the ''check'' will take the
mongrel out of
> rotation if it goes down, and add it back into the rotation as soon as
> it comes back up.  As soon as one mongrel is finished with a request,
> haproxy will give it one of the ones it has queued.
> 
> This is a simple version of what I''m doing to load balance
clusters of
> mongrels these days.
I initially tried doing this with haproxy before I made qrp, but it made
monitoring/testing something against an individual backend Mongrel
process far more difficult as I require Mongrel to accept connections
haproxy didn''t forward; so I disabled concurrency on the Mongrel
side instead.

-- 
Eric Wong

On Thu, May 08, 2008 at 12:50:25AM -0700, Eric Wong
wrote:
> Jeremy Hinegardner <jeremy at hinegardner.org> wrote:
> > On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
> > > Unfortunately a long running request could cause our entire web
> > > application to "hang".  As these requests are being
processing one or
> > > more mongrel instances continued to accept and enqueue new
connections
> > > it can''t immediately service.
> 
> Heh, this is also the exact problem I designed qrp around, too.
> 
> > Another solution: 
> > 
> > You could stick HAproxy in front of your mongrel cluster with a the
> > configuration somewhat like:
> > 
> >   listen application *:80
> >       balance roundrobin
> >       server  mongrel0 127.0.0.1:5000 minconn 1 maxconn 1 check
> >       server  mongrel1 127.0.0.1:5001 minconn 1 maxconn 1 check
> >       server  mongrel2 127.0.0.1:5002 minconn 1 maxconn 1 check
> >       server  mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
> >       server  mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check
> > 
> > The ''minconn 1 maxconn 1'' will force haproxy to
queue the results within
> > itself instead of mongrel, and the ''check'' will take
the mongrel out of
> > rotation if it goes down, and add it back into the rotation as soon as
> > it comes back up.  As soon as one mongrel is finished with a request,
> > haproxy will give it one of the ones it has queued.
> > 
> > This is a simple version of what I''m doing to load balance
clusters of
> > mongrels these days.
> 
> I initially tried doing this with haproxy before I made qrp, but it made
> monitoring/testing something against an individual backend Mongrel
> process far more difficult as I require Mongrel to accept connections
> haproxy didn''t forward; so I disabled concurrency on the Mongrel
> side instead.
> 
And we can''t forget about ngx_http_upstream_fair_module.c either.

Thanks Grzegorz and Ezra and anyone else who worked on that.

enjoy,

-jeremy


-- 
======================================================================= Jeremy
Hinegardner                              jeremy at hinegardner.org

Brian,

I found your patch while looking for a solution that we''ve needed to
decrease startup times, take advantage of COW, have completely graceful
restarts ( < 0.1 seconds downtime max, new server must be ready to
immediately start processing requests before the old server is booted
out ), and scheduling of garbage collection between results.

Your patch turned out to be a very good fit for all 4 of these arenas :)

Though, I did need to do some refactoring and additional work on what
you''ve started.

The main parts of my improvements are the "-R" flag
(Mongrel::UnixDispatchServer only), graceful shutdowns, and ability to
specify pre/post fork hooks.  The -R flag is like a hostile takeover
restart.  A new server boots up, gets all loaded and ready to go, then
sends a TERM signal to the old process.  The old process IMMEDIATELY
stops listening on it''s port, and waits for it''s children to
finish
their requests.  As soon as they are all done, the server shuts down.

My changes are available here in my fork of mongrel:

http://github.com/timcharper/mongrel/tree/super_cow_mongrel

Here''s the script I use to start up mongrel:

http://pastie.org/245416

I''d still like to see some additional cleanup and get some input on the
changes I''ve made.  Additionally, I did all this while working for my
company Lead Media Partners, so they should get a plug for this
contribution.

Peace and harmony,

Tim
-- 
Posted via http://www.ruby-forum.com/.

This looks interesting.

A question: can this version be made to run in the foreground, like `httpd
-DFOREGROUND''? I''d like to use daemontools to control the
master process,
which requires the master to not fork and background itself but stay in the
foregound and respond to signals.

-- 
Jos Backus
jos at catnook.com

Jos Backus wrote:
> This looks interesting.
> 
> A question: can this version be made to run in the foreground, like 
> `httpd
> -DFOREGROUND''? I''d like to use daemontools to control the
master
> process,
> which requires the master to not fork and background itself but stay in 
> the
> foregound and respond to signals.
It''s run fine for me either in the foreground or daemonized.
-- 
Posted via http://www.ruby-forum.com/.

On Fri, Aug 01, 2008 at 08:28:18AM +0200, Tim Harper
wrote:
> It''s run fine for me either in the foreground or daemonized.
Great, thanks Tim.

-- 
Jos Backus
jos at catnook.com