Hi, I found the following page describing how to stream data from the server to the client: http://api.rubyonrails.com/classes/ActionController/Streaming.html I want to do the opposite, streaming data from the client to the server, letting the controller saving the data while it''s being received. In particular, I want to upload large files to the RailsDav (http://www.liverail.net/railsdav) plugin, without having to store the entire file in memory during the operation. This seems to be a Mongrel problem, so I started looking at this plugin: http://mongrel.rubyforge.org/docs/upload_progress.html How do I get Mongrel to store the incoming POST data in a tempfile, passing a handler to that file as the raw_post field in the Rails request? Is it even possible? How is the "request" parameter to the process() method related to the "request" object in the Rails controllers? /Daniel
On 10/15/07, Daniel Brahneborg <basic70 at gmail.com> wrote:> > I want to do the opposite, streaming data from the client to the server, > letting the controller saving the data while it''s being received. > In particular, I want to upload large files to the RailsDav > (http://www.liverail.net/railsdav) plugin, without having to store > the entire file in memory during the operation. > > This seems to be a Mongrel problem, [...] >I don''t have the solution to your problem, but I think you are looking in the wrong place. Unless you plan to have mongrel listening on a public port, this is not a mongrel problem. The standard way to use mongrel, with Rails or not, is to set up a few of them behind a reverse proxy like nginx or apache+mod_proxy_balancer. In those cases, the proxy will not forward the data to mongrel before it has received it all, and once it has all the data, the latency between nginx and mongrel is negligible. On the other hand, the reverse proxy (I''m only speaking from experience with nginx here) saves the data to file while it is receiving it. I think you should look for a solution that lets nginx pass the path to that file to mongrel rather than sending the data again. I believe I have read about such a solution on this very list a few months ago. So, not the help you wanted, but I hope it leads you in the right direction. Well, if you actually intend to have mongrel face the public, then just forget what I said. Good luck! /David -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20071015/af934fb6/attachment.html
A while ago I wrote a plugin to limit uploads in Mongrel. Mongrel does save
uploads to a tempfile on disk - if the upload is bigger than about 12 KB - using
a Ruby TempFile-object. (which stores in the $TMPDIR, /tmp on most systems).
The request is handed over to Rails after it''s fully received by
Mongrel.
I''m not sure if this "saving to disk" works the same with
chunked uploads (uploads without a Content-Length header).
Maybe my plugin can help you: http://slasaus.netsend.nl/articles/show/7
(warning: it works, but is not very elegant).
Gl,
Tim
----- Original Message ----
From: Daniel Brahneborg <basic70 at gmail.com>
To: mongrel-users at rubyforge.org
Sent: Monday, October 15, 2007 2:00:20 PM
Subject: [Mongrel] POST with huge HTTP body
Hi,
I found the following page describing how to stream data from the
server to the client:
http://api.rubyonrails.com/classes/ActionController/Streaming.html
I want to do the opposite, streaming data from the client to the
server,
letting the controller saving the data while it''s being received.
In particular, I want to upload large files to the RailsDav
(http://www.liverail.net/railsdav) plugin, without having to store
the entire file in memory during the operation.
This seems to be a Mongrel problem, so I started looking at this
plugin:
http://mongrel.rubyforge.org/docs/upload_progress.html
How do I get Mongrel to store the incoming POST data in a tempfile,
passing a handler to that file as the raw_post field in the Rails
request?
Is it even possible?
How is the "request" parameter to the process() method related to the
"request" object in the Rails controllers?
/Daniel
_______________________________________________
Mongrel-users mailing list
Mongrel-users at rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users
____________________________________________________________________________________
Shape Yahoo! in your own image. Join our Network Research Panel today!
http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7
"... The standard way to use mongrel, with Rails or not, is to set up a few
of them behind a reverse proxy like nginx or apache+mod_proxy_balancer. In
those cases, the proxy will not forward the data to mongrel before it has
received it all, and once it has all the data, the latency between nginx and
mongrel is negligible. ..."
My Apache 2.2 load balancing reverse proxy forwards all data immediately,
without storing it in an "intermediate-file". I believe this is
standard behavior. So this should not be the problem.
I think the problem is in the fact that Mongrel stores the whole file-object
before passing it through to Rails, right? Unfortunately I have no clue how to
fix that.
Tim
----- Original Message ----
From: David Vrensk <david at vrensk.com>
To: mongrel-users at rubyforge.org
Sent: Monday, October 15, 2007 2:49:27 PM
Subject: Re: [Mongrel] POST with huge HTTP body
On 10/15/07, Daniel Brahneborg <basic70 at gmail.com> wrote: I want to do
the opposite, streaming data from the client to the server,
letting the controller saving the data while it''s being received.
In particular, I want to upload large files to the RailsDav
( http://www.liverail.net/railsdav) plugin, without having to store
the entire file in memory during the operation.
This seems to be a Mongrel problem, [...]
I don''t have the solution to your problem, but I think you are looking
in the wrong place.
Unless you plan to have mongrel listening on a public port, this is not a
mongrel problem. The standard way to use mongrel, with Rails or not, is to set
up a few of them behind a reverse proxy like nginx or apache+mod_proxy_balancer.
In those cases, the proxy will not forward the data to mongrel before it has
received it all, and once it has all the data, the latency between nginx and
mongrel is negligible.
On the other hand, the reverse proxy (I''m only speaking from experience
with nginx here) saves the data to file while it is receiving it. I think you
should look for a solution that lets nginx pass the path to that file to mongrel
rather than sending the data again. I believe I have read about such a solution
on this very list a few months ago.
So, not the help you wanted, but I hope it leads you in the right direction.
Well, if you actually intend to have mongrel face the public, then just forget
what I said.
Good luck!
/David
____________________________________________________________________________________
Fussy? Opinionated? Impossible to please? Perfect. Join Yahoo!''s user
panel and lay it on us.
http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7
Thanks for the replies. The fact that Mongrel shouldn''t be the first one to get the POST/PUT data was a good point that I hadn''t thought of. However, even when talking directly to Mongrel, the memory consumption of the application increases until it crashes if too much data is sent. In my case, I was sending a 500MB file from a Webdav client. At some point all of it was loaded into RAM, which doesn''t work. Are there any more productive alternatives than writing the entire thing from scratch in C? Please? :) /Daniel On 10/15/07, Tim Kuijsten <kuijsten at yahoo.com> wrote:> A while ago I wrote a plugin to limit uploads in Mongrel. Mongrel does save uploads to a tempfile on disk - if the upload is bigger than about 12 KB - using a Ruby TempFile-object. (which stores in the $TMPDIR, /tmp on most systems). > > > The request is handed over to Rails after it''s fully received by Mongrel. > > > I''m not sure if this "saving to disk" works the same with chunked uploads (uploads without a Content-Length header). > > > > Maybe my plugin can help you: http://slasaus.netsend.nl/articles/show/7 (warning: it works, but is not very elegant). > > > Gl, > > > Tim > > ----- Original Message ---- > From: Daniel Brahneborg <basic70 at gmail.com> > To: mongrel-users at rubyforge.org > Sent: Monday, October 15, 2007 2:00:20 PM > Subject: [Mongrel] POST with huge HTTP body > > > Hi, > > I found the following page describing how to stream data from the > server to the client: > > http://api.rubyonrails.com/classes/ActionController/Streaming.html > > I want to do the opposite, streaming data from the client to the > server, > letting the controller saving the data while it''s being received. > In particular, I want to upload large files to the RailsDav > (http://www.liverail.net/railsdav) plugin, without having to store > the entire file in memory during the operation. > > This seems to be a Mongrel problem, so I started looking at this > plugin: > > http://mongrel.rubyforge.org/docs/upload_progress.html > > How do I get Mongrel to store the incoming POST data in a tempfile, > passing a handler to that file as the raw_post field in the Rails > request? > Is it even possible? > How is the "request" parameter to the process() method related to the > "request" object in the Rails controllers? > > /Daniel > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > ____________________________________________________________________________________ > Shape Yahoo! in your own image. Join our Network Research Panel today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Since a vanilla Mongrel will store uploaded data on disk in a tempfile, maybe it has something to do with your Railsdav plugin? I just tested an 80 MB upload with my Apache 2.2.6 reverse proxy and Mongrel 1.0.1 cluster and it does save the object first to /tmp on disk and not in memory. The memory used by the mongrel process before I started uploading was 50 MB. While uploading it peaked at 57 MB but was mostly stable around 50-52 MB. And after the upload was done it was back again at 50 MB. Maybe your $TMPDIR is not writable for the user you are running mongrel under? Just before the upload was finished: $ ls -hl /tmp/ total 160288 -rw------- 1 mongrel wheel 78.2M Oct 15 16:31 mongrel.25.0 I would start looking in the Railsdav plugin you are using, seems like they changed the default of saving to disk, to memory. Cheers. ----- Original Message ---- From: Daniel Brahneborg <basic70 at gmail.com> To: mongrel-users at rubyforge.org Sent: Monday, October 15, 2007 4:10:06 PM Subject: Re: [Mongrel] POST with huge HTTP body Thanks for the replies. The fact that Mongrel shouldn''t be the first one to get the POST/PUT data was a good point that I hadn''t thought of. However, even when talking directly to Mongrel, the memory consumption of the application increases until it crashes if too much data is sent. In my case, I was sending a 500MB file from a Webdav client. At some point all of it was loaded into RAM, which doesn''t work. Are there any more productive alternatives than writing the entire thing from scratch in C? Please? :) /Daniel On 10/15/07, Tim Kuijsten <kuijsten at yahoo.com> wrote:> A while ago I wrote a plugin to limit uploads in Mongrel. Mongreldoes save uploads to a tempfile on disk - if the upload is bigger than about 12 KB - using a Ruby TempFile-object. (which stores in the $TMPDIR, /tmp on most systems).> > > The request is handed over to Rails after it''s fully received byMongrel.> > > I''m not sure if this "saving to disk" works the same with chunkeduploads (uploads without a Content-Length header).> > > > Maybe my plugin can help you:http://slasaus.netsend.nl/articles/show/7 (warning: it works, but is not very elegant).> > > Gl, > > > Tim > > ----- Original Message ---- > From: Daniel Brahneborg <basic70 at gmail.com> > To: mongrel-users at rubyforge.org > Sent: Monday, October 15, 2007 2:00:20 PM > Subject: [Mongrel] POST with huge HTTP body > > > Hi, > > I found the following page describing how to stream data from the > server to the client: > > http://api.rubyonrails.com/classes/ActionController/Streaming.html > > I want to do the opposite, streaming data from the client to the > server, > letting the controller saving the data while it''s being received. > In particular, I want to upload large files to the RailsDav > (http://www.liverail.net/railsdav) plugin, without having to store > the entire file in memory during the operation. > > This seems to be a Mongrel problem, so I started looking at this > plugin: > > http://mongrel.rubyforge.org/docs/upload_progress.html > > How do I get Mongrel to store the incoming POST data in a tempfile, > passing a handler to that file as the raw_post field in the Rails > request? > Is it even possible? > How is the "request" parameter to the process() method related to the > "request" object in the Rails controllers? > > /Daniel > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > >____________________________________________________________________________________> Shape Yahoo! in your own image. Join our Network Research Paneltoday! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7> > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >_______________________________________________ Mongrel-users mailing list Mongrel-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users ____________________________________________________________________________________ Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. http://mobile.yahoo.com/go?refer=1GNXIC
At Mon, 15 Oct 2007 14:00:20 +0200, "Daniel Brahneborg" <basic70 at gmail.com> wrote:> > Hi, > > I found the following page describing how to stream data from the > server to the client: > > http://api.rubyonrails.com/classes/ActionController/Streaming.html > > I want to do the opposite, streaming data from the client to the > server, letting the controller saving the data while it''s being > received. In particular, I want to upload large files to the > RailsDav (http://www.liverail.net/railsdav) plugin, without having > to store the entire file in memory during the operation. > > This seems to be a Mongrel problem, so I started looking at this plugin: > > http://mongrel.rubyforge.org/docs/upload_progress.html > > How do I get Mongrel to store the incoming POST data in a tempfile, > passing a handler to that file as the raw_post field in the Rails request? > Is it even possible?Hi Daniel, Mongrel is going to put a large POSTed body into a tempfile. Examine the mongrel.rb file around line 220. In this case your request.body is going to be a File object. Why do you think that there is a mongrel problem here? See also what Tim Kuijsten said in this thread.> How is the "request" parameter to the process() method related to > the "request" object in the Rails controllers?request in mongrel (as sent to process) is an instance of Mongrel::HttpRequest. In Rails it is an instance of ActionController::CgiRequest. best, Erik Hetzner ;; Erik Hetzner, California Digital Library ;; gnupg key id: 1024D/01DB07E3 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20071015/b71fcda3/attachment.bin
On Mon, 15 Oct 2007 16:10:06 +0200 "Daniel Brahneborg" <basic70 at gmail.com> wrote:> Thanks for the replies. > > The fact that Mongrel shouldn''t be the first one to get the POST/PUT data > was a good point that I hadn''t thought of. However, even when talking > directly to Mongrel, the memory consumption of the application increases > until it crashes if too much data is sent. In my case, I was sending a 500MB > file from a Webdav client. At some point all of it was loaded into RAM, which > doesn''t work. > > Are there any more productive alternatives than writing the entire thing > from scratch in C? Please? :)Daniel, Mongrel is a library. Not a server. I just wrote a server with the library. Look past mongrel_rails and into the heart of the RDoc. You will find your answer there. (Probably somewhere in the place where it mentions handlers and filters.) -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/
On Mon, 15 Oct 2007 07:37:57 -0700 (PDT) Tim Kuijsten <kuijsten at yahoo.com> wrote:> Since a vanilla Mongrel will store uploaded data on disk in a tempfile, maybe it has something to do with your Railsdav plugin?Yes, I believe this is true as well. Railsdav pretty much blows. My experience with it is that none of the windows clients work well (all three) and it did some stupid stuff like used send_file in rails and processed the whole file on input. Ehem, there''s this company called Xythos. They make a really good WebDAV server and client. Their WebDAV server is written in Java but it''s not insanely architected. Instead it''s very well done and using Jakarta Slide you can JRuby yourself a little Ruby lib that looks like Fileutils in about an hour. It''s also a very flexible server, and very conforming since most of the people who worked on the RFCs works for Xythos. It''s possible with Xythos to swap out nearly everything as well. For example, if you need to use your crappy "rails acts_as_lamely_authenticated" authentication, then you can write your own auth filter for Xythos and it''ll go and look in your database. If you need to do something special to the files then you can do a filter for that (it''s got a sample that uses Clam to do AV). They also give you all the source to their (ugly) web front end for free so you can see how every operation is done, even the weirdo admin stuff. It''s a good product, and if you''re dumb enough to work with WebDAV (since obviously you like Microsoft raping you over a barrel), then go check it out. Good luck, I''ve seen WebDAV kill off nearly every Rails project that''s come near it. Not sure what it is, but it''s like the black hole with a cache of gold in the center. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/
It looks like it''s CGI::QueryExtension#read_body in ActionPack that is the first one to cause problems. When a PUT or POST request arrives, it reads the entire thing into memory. Even if that method is patched, there are tons of other methods that wants to mess around with the request body. According to the documentation, the CGI class is supposed to handle Tempfile objects, but obviously not in this case. Seems like the way to go is to just use Mongrel and write a HttpHandler as a standalone server without both Rails and Railsdav, at least for a first version. Lots of thanks for pointing me in the right direction. /Daniel On 10/15/07, Tim Kuijsten <kuijsten at yahoo.com> wrote:> Since a vanilla Mongrel will store uploaded data on disk in a tempfile, maybe it has something to do with your Railsdav plugin? > > > I just tested an 80 MB upload with my Apache 2.2.6 reverse proxy and Mongrel 1.0.1 cluster and it does save the object first to /tmp on disk and not in memory. The memory used by the mongrel process before I started uploading was 50 MB. While uploading it peaked at 57 MB but was mostly stable around 50-52 MB. And after the upload was done it was back again at 50 MB. > > > Maybe your $TMPDIR is not writable for the user you are running mongrel under? > > > Just before the upload was finished: > $ ls -hl /tmp/ > total 160288 > -rw------- 1 mongrel wheel 78.2M Oct 15 16:31 mongrel.25.0 > > > > > > I would start looking in the Railsdav plugin you are using, seems like they changed the default of saving to disk, to memory. > > > > Cheers. > > ----- Original Message ---- > From: Daniel Brahneborg <basic70 at gmail.com> > To: mongrel-users at rubyforge.org > Sent: Monday, October 15, 2007 4:10:06 PM > Subject: Re: [Mongrel] POST with huge HTTP body > > > Thanks for the replies. > > The fact that Mongrel shouldn''t be the first one to get the POST/PUT > data > was a good point that I hadn''t thought of. However, even when talking > directly to Mongrel, the memory consumption of the application > increases > until it crashes if too much data is sent. In my case, I was sending a > 500MB > file from a Webdav client. At some point all of it was loaded into RAM, > which > doesn''t work. > > Are there any more productive alternatives than writing the entire > thing > from scratch in C? Please? :) > > /Daniel > > On 10/15/07, Tim Kuijsten <kuijsten at yahoo.com> wrote: > > A while ago I wrote a plugin to limit uploads in Mongrel. Mongrel > does save uploads to a tempfile on disk - if the upload is bigger than > about 12 KB - using a Ruby TempFile-object. (which stores in the $TMPDIR, > /tmp on most systems). > > > > > > The request is handed over to Rails after it''s fully received by > Mongrel. > > > > > > I''m not sure if this "saving to disk" works the same with chunked > uploads (uploads without a Content-Length header). > > > > > > > > Maybe my plugin can help you: > http://slasaus.netsend.nl/articles/show/7 (warning: it works, but is > not very elegant). > > > > > > Gl, > > > > > > Tim > > > > ----- Original Message ---- > > From: Daniel Brahneborg <basic70 at gmail.com> > > To: mongrel-users at rubyforge.org > > Sent: Monday, October 15, 2007 2:00:20 PM > > Subject: [Mongrel] POST with huge HTTP body > > > > > > Hi, > > > > I found the following page describing how to stream data from the > > server to the client: > > > > http://api.rubyonrails.com/classes/ActionController/Streaming.html > > > > I want to do the opposite, streaming data from the client to the > > server, > > letting the controller saving the data while it''s being received. > > In particular, I want to upload large files to the RailsDav > > (http://www.liverail.net/railsdav) plugin, without having to store > > the entire file in memory during the operation. > > > > This seems to be a Mongrel problem, so I started looking at this > > plugin: > > > > http://mongrel.rubyforge.org/docs/upload_progress.html > > > > How do I get Mongrel to store the incoming POST data in a tempfile, > > passing a handler to that file as the raw_post field in the Rails > > request? > > Is it even possible? > > How is the "request" parameter to the process() method related to the > > "request" object in the Rails controllers? > > > > /Daniel > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > > > > > > > ____________________________________________________________________________________ > > Shape Yahoo! in your own image. Join our Network Research Panel > today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 > > > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > ____________________________________________________________________________________ > Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. > http://mobile.yahoo.com/go?refer=1GNXIC > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >