Mongrel users - Aug 2006 - Setup with Apache 2.2 proxy balancer : timeout during uploads

Hi all,

I encounter timeout errors during file uploads in my Rails application.
I think the problem may be in the Mongrel configuration. Here is my setup:

* Load balancer (Cisco CSM, my provider''s Catalyst 6509)
* Apache 2.2.3 with mod_proxy_balancer
* Mongrel 0.3.13.3
* Rails 1.1.6
* Ruby 1.8.4
* MySQL 5.0.22
* file_column

I host an application in which users may upload medium files (around
2-5MB each) using an AJAX form POST, one large file at a time. Some
users succeed to upload, others do not. There seem to be no relation
between the failing users and their browsers.

I see these error messages in Apache''s error_log:

[Tue Aug 29 21:28:27 2006] [error] (70007)The timeout specified has
expired: proxy: pass request body failed to 127.0.0.1:8012 (127.0.0.1)
from THE.CLIENT.IP.HERE ()

Joining these error messages with the access_log''s, the failing
requests
are the file uploads.

At the beginning, I had 5 Mongrel processes. I tried to increase this
value to 20 but I still see such errors. Is there any problem with
concurrency here? Does this setup do not handle a small/medium number of
concurrent uploads? The site is not yet heavily loaded, but it may be in
the near future. I hope I''ll be able to fix this very soon :)

I''m not quite sure to fully understand how a request is handled by this
setup, particularly at which step the full request may be cached and
passed by to Mongrel and/or Rails fully, without waiting for the client
to finish his upload.

Also note that the system''s load average is near zero, the system is
responsive, the bandwidth fully available (although the bottleneck is at
the client''s end on this point).

I hope I''ve been clear enough and don''t hesitate to ask for
more logs
and debugs ! Unfortunately, I can''t reproduce this problem locally and
I
haven''t tried Mongrel 0.3.13.4 on the production system yet.

Maybe this problem is related to the Rails code handling the upload
(which is the file_column plugin)? In this case, I will ask it on the
rails list.

Any suggestion would be greatly appreciated!

Philippe J.

On Aug 29, 2006, at 12:45 PM, Philippe Jayet wrote:
> I encounter timeout errors during file uploads in my Rails  
> application.
How big are the uploads?

- Jason

> How big are the uploads?
Around 2-5MB. Not very large ... :/

Philippe J.

On 8/29/06, Philippe Jayet <hip at lan-attack.ch>
wrote:
> Hi all,
>
> I encounter timeout errors during file uploads in my Rails application.
> I think the problem may be in the Mongrel configuration. Here is my setup:
>
> * Load balancer (Cisco CSM, my provider''s Catalyst 6509)
> * Apache 2.2.3 with mod_proxy_balancer
> * Mongrel 0.3.13.3
> * Rails 1.1.6
> * Ruby 1.8.4
> * MySQL 5.0.22
> * file_column
I''ve started seeing this too recently.  I upgraded to the latest
mongrel prerelease 0.3.13.4 and still had an issue this morning.  The
uploaded files are usually at least 20MB when this happens.  Here''s my
config:

Apache 2 with mod_proxy_balancer
mongrel 0.3.13.4
Edge Rails
Ruby 1.8.4

It doesn''t seem to be rails itself.  The production logs show nothing
peculiar.  I''ve turned on debug logging to see if it happens again.

-- 
Rick Olson
http://weblog.techno-weenie.net
http://mephistoblog.com

On Tue, 29 Aug 2006, Philippe Jayet wrote:
> Hi all,
>
> I encounter timeout errors during file uploads in my Rails application.
> I think the problem may be in the Mongrel configuration. Here is my setup:
>
> * Load balancer (Cisco CSM, my provider''s Catalyst 6509)
> * Apache 2.2.3 with mod_proxy_balancer
> * Mongrel 0.3.13.3
> * Rails 1.1.6
> * Ruby 1.8.4
> * MySQL 5.0.22
> * file_column
>
> I host an application in which users may upload medium files (around
> 2-5MB each) using an AJAX form POST, one large file at a time. Some
> users succeed to upload, others do not. There seem to be no relation
> between the failing users and their browsers.
>
We have almost the same setup... different load balancer (fortinet) and 
not at rails 1.1.6 (patched though) and not using file_column...

I''ve tested uploads with 80mb files (audio shows). Takes awhile, but 
never had a timeout.

I''ve done this with both lighty/fastcgi and apache/mongrel...

anyway, not sure that helps or not :/

I also get time outs on uploads every so often.

I''m running on FreeBSD 6.1
Apache 2.2.2 with mod_proxy_balancer
Mongrel 0.13.3
Rails 1.1.6
Ruby 1.8.4
MySQL 5.0.22

But it happens pretty rarely.

-carl

On Aug 29, 2006, at 3:58 PM, Philippe Jayet wrote:
>
>> How big are the uploads?
>
> Around 2-5MB. Not very large ... :/
>
> Philippe J.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

Thanks all for your feedback.
> I''ve started seeing this too recently.  I upgraded to the latest
> mongrel prerelease 0.3.13.4 and still had an issue this morning.  The
> uploaded files are usually at least 20MB when this happens. 
Here''s my
> config:
Apparently in my setup, there is no relation with the file size, but one
actor has a timeout of 10 minutes, it appears to be quite constant in
the log files. But I really can''t figure out where it may come from :(

Any idea someone ? I''ll try to investigate the Apache configuration.

Philippe J.

Hi all,
> Apparently in my setup, there is no relation with the file size, but one
> actor has a timeout of 10 minutes, it appears to be quite constant in
> the log files. But I really can''t figure out where it may come
from :(
> 
> Any idea someone ? I''ll try to investigate the Apache
configuration.
That wasn''t it. :) Although the ten minutes timeout was from Apache,
this isn''t the problem.

In fact the source of the problem was coming from the provider''s load
balancer ! No idea why, I''ll have to further investigate. But accessing
my server directly, I have no problem uploading files. I should have
thought trying that earlier :/

So ... really not related to Mongrel, which makes this thread slightly OT :)

Anyway thanks all for your help.

Philippe J.

Hello Mongrel gurus!

We are about to deploy our Rails application to production, and even  
though the last couple of weeks of testing went really well, today we  
are seeing a new critical issue - the browser just hangs sometimes  
trying to load a page. When this happens, it seems that one or more  
of the mongrel_rails processes is hanging and not responding, while  
others are (so depending on which process the request is routed to,  
the page loads or not).  The only remedy from this point onward  
appears to be brutal kill and restart of mongrel processes.

I found the following lines in mongrel.log:

Wed Aug 30 14:03:22 PDT 2006: Reaping 1 threads for slow workers  
because of ''shutdown''
Thread #<Thread:0xb79d382c sleep> is too old, killing.
Waiting for 1 requests to finish, could take 60 seconds.Wed Aug 30  
14:03:23 PDT 2006: Reaping 1 threads for slow workers because of  
''shutdown''
Thread #<Thread:0xb78a03ac sleep> is too old, killing.

And apache error log has this:

[Wed Aug 30 14:46:59 2006] [error] [client 64.161.139.66] proxy:  
Error reading from remote server returned by /bookstore, referer: ht
tp://wp01.my-secret-domain.com/create/book

Apache''s SSL error log has this:

[Wed Aug 30 14:42:35 2006] [error] [client 64.161.139.66] proxy:  
error reading status line from remote server 127.0.0.1
[Wed Aug 30 14:42:35 2006] [error] [client 64.161.139.66] proxy:  
Error reading from remote server returned by /my/account/save_edit_a
ddress/80


Restarting mongrel_cluster in a regular fashion leaves some of those  
processes running, and subsequent processes fail to start.  Killing  
mongrel_rails processes with ''kill -9'' actually clears the
cluster
and after a restart it appears to work fine.  This happened a couple  
of times today, and I am wondering if anyone has seen this behavior -  
and if there is a recommended fix?

- Should I upgrade to the latest unreleased version of Mongrel?  I am  
a little worried that it''s also unstable.
- We did not change anything about Rails session handling.  Could  
mongrel be locking up because it is dead-locking on the session files?
- How do I tell what''s really going on?
- Can I enable more mongrel logging that is acceptable for production  
use (ie - not the full debugging info)
- Can I turn on PID and date/time logging to mongrel.log so that I  
can see which process is having an issue?

Any advice on how to approach this issue is much appreciated.

Our environment:

Apache 2.2.3 + mod_proxy_balancer + mod_ssl, etc...
Mongrel 0.3.13.3
Ruby 1.8.4
Rails 1.1.6, SslRequirements, etc
Linux 2.6.9-42.ELsmp (RedHat ES4 with all up-to-date patches) on Intel

Mongrel Config:
---
cwd: /data/apps/app1/current
port: "5000"
environment: production
address: 127.0.0.1
pid_file: log/mongrel.pid
servers: 10


Apache Info:

# httpd -v

Server version: Apache/2.2.3
Server built:   Aug 22 2006 10:26:14
wp01[root]# httpd -V
Server version: Apache/2.2.3
Server built:   Aug 22 2006 10:26:14
Server''s Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   32-bit
Server MPM:     Prefork
   threaded:     no
     forked:     yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache-2.2.3"
-D SUEXEC_BIN="/usr/local/apache-2.2.3/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

Apache Module Info:

# httpd -l
Compiled in modules:
   core.c
   mod_authn_file.c
   mod_authn_default.c
   mod_authz_host.c
   mod_authz_groupfile.c
   mod_authz_user.c
   mod_authz_default.c
   mod_auth_basic.c
   mod_include.c
   mod_filter.c
   mod_log_config.c
   mod_env.c
   mod_headers.c
   mod_setenvif.c
   mod_proxy.c
   mod_proxy_connect.c
   mod_proxy_ftp.c
   mod_proxy_http.c
   mod_proxy_ajp.c
   mod_proxy_balancer.c
   mod_ssl.c
   prefork.c
   http_core.c
   mod_mime.c
   mod_status.c
   mod_autoindex.c
   mod_asis.c
   mod_cgi.c
   mod_negotiation.c
   mod_dir.c
   mod_actions.c
   mod_userdir.c
   mod_alias.c
   mod_rewrite.c
   mod_so.c


Restarting using capistrano with a few hanging mongrel processes:


 > cap restart
     loading configuration /usr/local/lib/ruby/gems/1.8/gems/ 
capistrano-1.1.0/lib/capistrano/recipes/standard.rb
     loading configuration ./config/deploy.rb
     loading configuration #<Proc:0x003566d0@/usr/local/lib/ruby/gems/ 
1.8/gems/mongrel_cluster-0.2.0/lib/mongrel_cluster/recipes.rb:1>
   * executing task restart
   * executing task restart_mongrel_cluster
   * executing task stop_mongrel_cluster
   * executing "mongrel_rails cluster::stop -C /data/apps/app1/ 
current/config/mongrel_cluster.yml"
     servers: ["wp01.my-secret-domain.com"]
     [wp01.my-secret-domain.com] executing command
** [out :: wp01.my-secret-domain.com] Stopping 10 Mongrel servers...
     command finished
   * executing task start_mongrel_cluster
   * executing "mongrel_rails cluster::start -C /data/apps/app1/ 
current/config/mongrel_cluster.yml"
     servers: ["wp01.my-secret-domain.com"]
     [wp01.my-secret-domain.com] executing command
** [out :: wp01.my-secret-domain.com] Starting 10 Mongrel servers...
** [out :: wp01.my-secret-domain.com] ** !!! PID file log/mongrel. 
5000.pid already exists.  Mongrel could be running already. Check  
your log/mongrel.log for errors.
** [out :: wp01.my-secret-domain.com] ** !!! PID file log/mongrel. 
5001.pid already exists.  Mongrel could be running already. Check  
your log/mongrel.log for errors.
** [out :: wp01.my-secret-domain.com] **
** [out :: wp01.my-secret-domain.com] !!! PID file log/mongrel. 
5004.pid already exists.  Mongrel could be running alrea
** [out :: wp01.my-secret-domain.com] dy. Check your log/mongrel.log  
for errors.
** [out :: wp01.my-secret-domain.com] **
** [out :: wp01.my-secret-domain.com] !!! PID file log/mongrel. 
5005.pid already exists.  Mongrel could be running alrea
** [out :: wp01.my-secret-domain.com] dy. Check your log/mongrel.log  
for errors.
** [out :: wp01.my-secret-domain.com] **
** [out :: wp01.my-secret-domain.com] !!! PID file log/mongrel. 
5007.pid already exists.  Mongrel could be running alrea
** [out :: wp01.my-secret-domain.com] dy. Check your log/mongrel.log  
for errors.
     command finished

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060830/1d9d33a8/attachment-0001.html

On Wed, 2006-08-30 at 15:10 -0700, kigsteronline at mac.com
wrote:
> Hello Mongrel gurus!
> 
> 
> We are about to deploy our Rails application to production, and even
> though the last couple of weeks of testing went really well, today we
> are seeing a new critical issue - the browser just hangs sometimes
> trying to load a page. When this happens, it seems that one or more of
> the mongrel_rails processes is hanging and not responding, while
> others are (so depending on which process the request is routed to,
> the page loads or not).  The only remedy from this point onward
> appears to be brutal kill and restart of mongrel processes.
> 
Ah, you might be hitting the CLOSE_WAIT bug.  This makes me think more
and more that Apache is doing something funky.  This bug and a 99% CPU
bug is driving down right insane since I can''t replicate it to save my
life.  I''ve got mongrel serving ISO images to an RFuzz script that
randomly closes sockets and does nasty requests while thrashing mongrel
with httperf with a short timeout.

Nothing.  Not even one CLOSE_WAIT socket, no memory leaks (anymore), and
no CPU thrashing.  It''s really driving me nuts.

What I''ve done is created a special "debug version" of the
mongrel
pre-release that I''d like you to run.  Here''s what you do:

1) download http://www.zedshaw.com/mongrel-0.3.13.4.gem
2) install:  gem install mongrel-0.3.13.4.gem
3) start your app just like it is.
4) Hit mongrels with USR1:  killall -USR1 mongrel_rails
5) Let mongrel run until it dies horribly.
6) Just before you kill -9 mongrel, run:  lsof -i -P | grep CLOSE_WAIT >
close_wait.log
7) Stop mongrel and send me log/mongrel.log and close_wait.log to
zedshaw at zedshaw.com.

Hopefully I can nail this for the next release.  The CLOSE_WAIT used to
be really bad but I cut it down quite a bit, so it should take longer to
detroy it.

Also make sure to tell me if the CPU usage for mongrel is above 90%
(it''s pegged).

Thanks for the help.



-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Zed and friends,

Turns out our locking up issue was related to the corrupt ruby stack  
in our application.

We were able to reproduce this bug in Webrick and LightHTTPD, so this  
is not Mongrel specific.

Thanks Zed for all the help,
Konstantin


On Aug 30, 2006, at 5:42 PM, Zed Shaw wrote:
> On Wed, 2006-08-30 at 15:10 -0700, kigsteronline at mac.com wrote:
>> Hello Mongrel gurus!
>>
>>
>> We are about to deploy our Rails application to production, and even
>> though the last couple of weeks of testing went really well, today we
>> are seeing a new critical issue - the browser just hangs sometimes
>> trying to load a page. When this happens, it seems that one or  
>> more of
>> the mongrel_rails processes is hanging and not responding, while
>> others are (so depending on which process the request is routed to,
>> the page loads or not).  The only remedy from this point onward
>> appears to be brutal kill and restart of mongrel processes.
>>
>
> Ah, you might be hitting the CLOSE_WAIT bug.  This makes me think more
> and more that Apache is doing something funky.  This bug and a 99% CPU
> bug is driving down right insane since I can''t replicate it to
save my
> life.  I''ve got mongrel serving ISO images to an RFuzz script that
> randomly closes sockets and does nasty requests while thrashing  
> mongrel
> with httperf with a short timeout.
>
> Nothing.  Not even one CLOSE_WAIT socket, no memory leaks  
> (anymore), and
> no CPU thrashing.  It''s really driving me nuts.
>
> What I''ve done is created a special "debug version" of
the mongrel
> pre-release that I''d like you to run.  Here''s what you
do:
>
> 1) download http://www.zedshaw.com/mongrel-0.3.13.4.gem
> 2) install:  gem install mongrel-0.3.13.4.gem
> 3) start your app just like it is.
> 4) Hit mongrels with USR1:  killall -USR1 mongrel_rails
> 5) Let mongrel run until it dies horribly.
> 6) Just before you kill -9 mongrel, run:  lsof -i -P | grep  
> CLOSE_WAIT >
> close_wait.log
> 7) Stop mongrel and send me log/mongrel.log and close_wait.log to
> zedshaw at zedshaw.com.
>
> Hopefully I can nail this for the next release.  The CLOSE_WAIT  
> used to
> be really bad but I cut it down quite a bit, so it should take  
> longer to
> detroy it.
>
> Also make sure to tell me if the CPU usage for mongrel is above 90%
> (it''s pegged).
>
> Thanks for the help.
>
>
>
> -- 
> Zed A. Shaw
> http://www.zedshaw.com/
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

Hi all,
> I''m not quite sure to fully understand how a request is handled by
this
> setup, particularly at which step the full request may be cached and
> passed by to Mongrel and/or Rails fully, without waiting for the client
> to finish his upload.
Concerning this point, I''m not sure of the following: does Mongrel
receive the full request before passing it to Rails ?

Just to be sure.

Thanks in advance for your answers!

Philippe

>
> Concerning this point, I''m not sure of the following: does Mongrel
> receive the full request before passing it to Rails ?
>
Yes it does.

On Thu, 2006-08-31 at 15:12 +0200, Philippe Jayet wrote:
> Hi all,
> 
> > I''m not quite sure to fully understand how a request is
handled by this
> > setup, particularly at which step the full request may be cached and
> > passed by to Mongrel and/or Rails fully, without waiting for the
client
> > to finish his upload.
> 
> Concerning this point, I''m not sure of the following: does Mongrel
> receive the full request before passing it to Rails ?
Yes, Mongrel processes the full request, including uploaded body before
calling rails (and locking it).  If the request is large then it also
streams the result to a tmpfile in order to not use up ram.

The response Rails generates is placed into an internal buffer and then
when Rails is done Mongrel unlocks rails and spends the rest of it''s
time sending back the response.  This is necessary so that Rails can be
locked for the shortest amount of time.

-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

> Yes, Mongrel processes the full request, including uploaded body before
> calling rails (and locking it).  If the request is large then it also
> streams the result to a tmpfile in order to not use up ram.
You''re doing an amazing job Zed :)
> The response Rails generates is placed into an internal buffer and then
> when Rails is done Mongrel unlocks rails and spends the rest of
it''s
> time sending back the response.  This is necessary so that Rails can be
> locked for the shortest amount of time.
How does it interact with send_file and send_data? Which method is the
best to use to serve files of serveral MB?

Thanks for your help,

Philippe J.

On Fri, 2006-09-01 at 14:42 +0200, Philippe Jayet wrote:
> > Yes, Mongrel processes the full request, including uploaded body
before
> > calling rails (and locking it).  If the request is large then it also
> > streams the result to a tmpfile in order to not use up ram.
> 
> You''re doing an amazing job Zed :)
> 
Thanks!
> > The response Rails generates is placed into an internal buffer and
then
> > when Rails is done Mongrel unlocks rails and spends the rest of
it''s
> > time sending back the response.  This is necessary so that Rails can
be
> > locked for the shortest amount of time.
> 
> How does it interact with send_file and send_data? Which method is the
> best to use to serve files of serveral MB?
Ah, yes, problem with those is that they''re run inside Rails, so
they''re
done during the lock and streamed out into the buffer mongrel hands to
rails.

Not a good combo.  You''re actually *best* off using mod_xsendfile from
apache, which lets you write the giganto data to a file and then put an
"X-Sendfile" header in the response pointing at this file.  Apache
will
then pick the file up as the response body and shoot it to the client
without bothering you.

Lighttpd also has this in the latest (unstable) version.


-- 
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.