I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does not have mongrel installed. It does proxy requests to several other mongrel-only servers (each running 2 mongrel processes). Each mongrel node has the same rails code-base and it''s working perfectly. However, my question is when I add an upload file form... where is it going to physically put that file? I mean since it''s hitting either one node or the other, so how does mongrel deal with that? and how or where do I tell it to accept large files (100mb+) ? I have read on having an upload-only mongrel process, but again how will all the other mongrels know to post the file to it or to a specific folder somewhere? Direction on this would be greatly appreciated! Thanks for your help on this. I searched the forum and didn''t find anything specifically dealing with this. I have also checked the mongrel docs and most of the setups talk about mongrel_upload_progress plugin and/or the mongrel_cluster setup...(I may have missed something :-/) cheers, -rjs- p.s. mongrel ROCKS!
If you''re using Rails with Mongrel, file upload currently sucks. If you have two mongrels running (say) and two users upload two large files, your entire app will be hung just serving those requests. This is because Rails can''t run safely multi-threaded, and it''s not Mongrel''s fault. An upload only Mongrel could (and should) be written in a framework other than Rails. Both Camping and Merb are good choices. In this instance you don''t need many mongrels since even 1 can simultaneously serve many file uploads. So you''ll ideally have 1 server where you have one or more mongrels running your file upload app, and use that to accept uploads. This is what we do at SlideShare: we have a custom Camping app to accept file uploads. Your frontend webserver or load balancer would redirect all requests to /upload (say) to your uploader mongrel running at it''s own port. And all other requests can be served some other way. We do this through lighttpd proxying. HTH Vish On 11/28/06, Rogelio J. Samour <rogelio.samour at gmail.com> wrote:> > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > not have mongrel installed. It does proxy requests to several other > mongrel-only servers (each running 2 mongrel processes). Each mongrel > node has the same rails code-base and it''s working perfectly. > > However, my question is when I add an upload file form... where is it > going to physically put that file? I mean since it''s hitting either one > node or the other, so how does mongrel deal with that? and how or where > do I tell it to accept large files (100mb+) ? > > I have read on having an upload-only mongrel process, but again how will > all the other mongrels know to post the file to it or to a specific > folder somewhere? Direction on this would be greatly appreciated! > > Thanks for your help on this. I searched the forum and didn''t find > anything specifically dealing with this. I have also checked the mongrel > docs and most of the setups talk about mongrel_upload_progress plugin > and/or the mongrel_cluster setup...(I may have missed something :-/) > > > cheers, > > -rjs- > > p.s. mongrel ROCKS! > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/1fc96cbb/attachment.html
On Nov 28, 2006, at 11:40 AM, Rogelio J. Samour wrote:> I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > not have mongrel installed. It does proxy requests to several other > mongrel-only servers (each running 2 mongrel processes). Each mongrel > node has the same rails code-base and it''s working perfectly. > > However, my question is when I add an upload file form... where is it > going to physically put that file? I mean since it''s hitting either > one > node or the other, so how does mongrel deal with that? and how or > where > do I tell it to accept large files (100mb+) ? > > I have read on having an upload-only mongrel process, but again how > will > all the other mongrels know to post the file to it or to a specific > folder somewhere? Direction on this would be greatly appreciated! > > Thanks for your help on this. I searched the forum and didn''t find > anything specifically dealing with this. I have also checked the > mongrel > docs and most of the setups talk about mongrel_upload_progress plugin > and/or the mongrel_cluster setup...(I may have missed something :-/)I''m starting work on a way for Mongrel to better handle really large file uploads (20+ MB). In the meantime, it''s still pretty easy to setup a dedicated mongrel instance for file uploads. Assuming you are using mod_proxy_balancer, you first need to setup an exception for it. For example, the balancer setup lines look like this: ProxyPass / balancer://mongrel_cluster/ ProxyPassReverse / balancer://mongrel_cluster Put an exception line before the first call to ProxyPass like so: ProxyPass /uploads ! # proxy will not rewrite if it contains this URI Then, further down in your Apache configuration file, put in a rewrite rule to send the connection to the mongrel dedicated to uploads: RewriteEngine On RewriteCond %{REQUEST_URI} ^/uploads.* RewriteRule .* http://example.com:some_port% {REQUEST_URI} [P,QSA,L] (That RewriteRule is untested but should work.) There are probably other ways to skin this particular cat. cr
I would like to combine what you guys are suggesting. Namely, have a merb app (mongrel) dealio... be the sole file uploader listener. I understand the ReWrite rules and ProxyPass magic. However, this poses a new question: how would I go about downloading these uploaded files? I''m assuming I would want this mongrel node to also be a file server, correct? My only experience with setting up mongrel is through mongrel_rails. cremes: are you saying this upload-only mongrel is it''s own setup? if so, is there docs on setting up something like this? Thanks a lot for your time amigos! -rjs- On 11/28/06, cremes.devlist at mac.com <cremes.devlist at mac.com> wrote:> > > On Nov 28, 2006, at 11:40 AM, Rogelio J. Samour wrote: > > > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > > not have mongrel installed. It does proxy requests to several other > > mongrel-only servers (each running 2 mongrel processes). Each mongrel > > node has the same rails code-base and it''s working perfectly. > > > > However, my question is when I add an upload file form... where is it > > going to physically put that file? I mean since it''s hitting either > > one > > node or the other, so how does mongrel deal with that? and how or > > where > > do I tell it to accept large files (100mb+) ? > > > > I have read on having an upload-only mongrel process, but again how > > will > > all the other mongrels know to post the file to it or to a specific > > folder somewhere? Direction on this would be greatly appreciated! > > > > Thanks for your help on this. I searched the forum and didn''t find > > anything specifically dealing with this. I have also checked the > > mongrel > > docs and most of the setups talk about mongrel_upload_progress plugin > > and/or the mongrel_cluster setup...(I may have missed something :-/) > > I''m starting work on a way for Mongrel to better handle really large > file uploads (20+ MB). > > In the meantime, it''s still pretty easy to setup a dedicated mongrel > instance for file uploads. > > Assuming you are using mod_proxy_balancer, you first need to setup an > exception for it. > > For example, the balancer setup lines look like this: > > ProxyPass / balancer://mongrel_cluster/ > ProxyPassReverse / balancer://mongrel_cluster > > Put an exception line before the first call to ProxyPass like so: > > ProxyPass /uploads ! # proxy will not rewrite if it contains this URI > > Then, further down in your Apache configuration file, put in a > rewrite rule to send the connection to the mongrel dedicated to uploads: > > RewriteEngine On > > RewriteCond %{REQUEST_URI} ^/uploads.* > RewriteRule .* http://example.com:some_port% > {REQUEST_URI} [P,QSA,L] > > (That RewriteRule is untested but should work.) There are probably > other ways to skin this particular cat. > > cr > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061128/fde9625f/attachment.html
If you use something like mongrel_upload_progress, the single-threadedness of Rails is no problem because Mongrel will intercept the upload and not hand it off to Rails until it''s complete. That''s what I''m doing at the moment, and my app runs just fine on one Mongrel. If you''re running large file uploads, I would recommend looking into it anyway, because your users will surely want some kind of update about how their upload is going. Matt On 11/29/06, Vishnu Gopal <g.vishnu at gmail.com> wrote:> > If you''re using Rails with Mongrel, file upload currently sucks. If you > have two mongrels running (say) and two users upload two large files, your > entire app will be hung just serving those requests. This is because Rails > can''t run safely multi-threaded, and it''s not Mongrel''s fault. > > An upload only Mongrel could (and should) be written in a framework other > than Rails. Both Camping and Merb are good choices. In this instance you > don''t need many mongrels since even 1 can simultaneously serve many file > uploads. So you''ll ideally have 1 server where you have one or more mongrels > running your file upload app, and use that to accept uploads. This is what > we do at SlideShare: we have a custom Camping app to accept file uploads. > > Your frontend webserver or load balancer would redirect all requests to > /upload (say) to your uploader mongrel running at it''s own port. And all > other requests can be served some other way. We do this through lighttpd > proxying. > > HTH > Vish > > On 11/28/06, Rogelio J. Samour <rogelio.samour at gmail.com> wrote: > > > > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > > not have mongrel installed. It does proxy requests to several other > > mongrel-only servers (each running 2 mongrel processes). Each mongrel > > node has the same rails code-base and it''s working perfectly. > > > > However, my question is when I add an upload file form... where is it > > going to physically put that file? I mean since it''s hitting either one > > node or the other, so how does mongrel deal with that? and how or where > > do I tell it to accept large files (100mb+) ? > > > > I have read on having an upload-only mongrel process, but again how will > > all the other mongrels know to post the file to it or to a specific > > folder somewhere? Direction on this would be greatly appreciated! > > > > Thanks for your help on this. I searched the forum and didn''t find > > anything specifically dealing with this. I have also checked the mongrel > > docs and most of the setups talk about mongrel_upload_progress plugin > > and/or the mongrel_cluster setup...(I may have missed something :-/) > > > > > > cheers, > > > > -rjs- > > > > p.s. mongrel ROCKS! > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > >-- Matt White ----------------------- Thermal Creative http://blog.thermalcreative.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/13b552dc/attachment-0001.html
Great work guys, I''m testing this with a client and we''ll be updating the apache docs and linking over to the mongrel upload pages from there. On 11/28/06, cremes.devlist at mac.com <cremes.devlist at mac.com> wrote:> > On Nov 28, 2006, at 11:40 AM, Rogelio J. Samour wrote: > > > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > > not have mongrel installed. It does proxy requests to several other > > mongrel-only servers (each running 2 mongrel processes). Each mongrel > > node has the same rails code-base and it''s working perfectly. > > > > However, my question is when I add an upload file form... where is it > > going to physically put that file? I mean since it''s hitting either > > one > > node or the other, so how does mongrel deal with that? and how or > > where > > do I tell it to accept large files (100mb+) ? > > > > I have read on having an upload-only mongrel process, but again how > > will > > all the other mongrels know to post the file to it or to a specific > > folder somewhere? Direction on this would be greatly appreciated! > > > > Thanks for your help on this. I searched the forum and didn''t find > > anything specifically dealing with this. I have also checked the > > mongrel > > docs and most of the setups talk about mongrel_upload_progress plugin > > and/or the mongrel_cluster setup...(I may have missed something :-/) > > I''m starting work on a way for Mongrel to better handle really large > file uploads (20+ MB). > > In the meantime, it''s still pretty easy to setup a dedicated mongrel > instance for file uploads. > > Assuming you are using mod_proxy_balancer, you first need to setup an > exception for it. > > For example, the balancer setup lines look like this: > > ProxyPass / balancer://mongrel_cluster/ > ProxyPassReverse / balancer://mongrel_cluster > > Put an exception line before the first call to ProxyPass like so: > > ProxyPass /uploads ! # proxy will not rewrite if it contains this URI > > Then, further down in your Apache configuration file, put in a > rewrite rule to send the connection to the mongrel dedicated to uploads: > > RewriteEngine On > > RewriteCond %{REQUEST_URI} ^/uploads.* > RewriteRule .* http://example.com:some_port% > {REQUEST_URI} [P,QSA,L] > > (That RewriteRule is untested but should work.) There are probably > other ways to skin this particular cat. > > cr > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-- Charles Brian Quinn self-promotion: www.seebq.com highgroove studios: www.highgroove.com slingshot hosting: www.slingshothosting.com
> I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > not have mongrel installed. It does proxy requests to several other > mongrel-only servers (each running 2 mongrel processes). Each mongrel > node has the same rails code-base and it''s working perfectly. > > However, my question is when I add an upload file form... where is it > going to physically put that file? I mean since it''s hitting either one > node or the other, so how does mongrel deal with that? and how or where > do I tell it to accept large files (100mb+) ?Others have given good advice on how to do it, but wanted to say you don''t need to do anything to get it to accept 100mb files. We do that quite a bit and while it takes awhile works just fine. It does tie up a mongrel process, but we''re okay with that.> > I have read on having an upload-only mongrel process, but again how will > all the other mongrels know to post the file to it or to a specific > folder somewhere? Direction on this would be greatly appreciated! > > Thanks for your help on this. I searched the forum and didn''t find > anything specifically dealing with this. I have also checked the mongrel > docs and most of the setups talk about mongrel_upload_progress plugin > and/or the mongrel_cluster setup...(I may have missed something :-/) > > > cheers, > > -rjs- > > p.s. mongrel ROCKS! > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Philip: at this point I want to upload something... anything. I''m following the example on the mongrel_upload_plugin and though it says finished I don''t see the file anywhere. I Turned on --debug and I can only see that it''s at least getting the headers (content-type, etc) on the uploaded file... but nothing on the whereabouts. Like I mentioned before, I''m very green when it comes to mongrel... however I''m decently versed in sysadmin *nix stuff... apache and friends. I''m just needing a nudge in the right direction. :-) cheers, -rjs- On 11/28/06, Philip Hallstrom <mongrel at philip.pjkh.com> wrote:> > > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > > not have mongrel installed. It does proxy requests to several other > > mongrel-only servers (each running 2 mongrel processes). Each mongrel > > node has the same rails code-base and it''s working perfectly. > > > > However, my question is when I add an upload file form... where is it > > going to physically put that file? I mean since it''s hitting either one > > node or the other, so how does mongrel deal with that? and how or where > > do I tell it to accept large files (100mb+) ? > > Others have given good advice on how to do it, but wanted to say you don''t > need to do anything to get it to accept 100mb files. We do that quite a > bit and while it takes awhile works just fine. It does tie up a mongrel > process, but we''re okay with that. > > > > > I have read on having an upload-only mongrel process, but again how will > > all the other mongrels know to post the file to it or to a specific > > folder somewhere? Direction on this would be greatly appreciated! > > > > Thanks for your help on this. I searched the forum and didn''t find > > anything specifically dealing with this. I have also checked the mongrel > > docs and most of the setups talk about mongrel_upload_progress plugin > > and/or the mongrel_cluster setup...(I may have missed something :-/) > > > > > > cheers, > > > > -rjs- > > > > p.s. mongrel ROCKS! > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061128/3538951a/attachment.html
On 28.11.2006, at 21.30, Matt White wrote:> If you use something like mongrel_upload_progress, the single- > threadedness of Rails is no problem because Mongrel will intercept > the upload and not hand it off to Rails until it''s complete. That''s > what I''m doing at the moment, and my app runs just fine on one > Mongrel. > > If you''re running large file uploads, I would recommend looking > into it anyway, because your users will surely want some kind of > update about how their upload is going.Out of curiosity, has anyone managed to pair mongrel_upload_progress (or any of the other options) with MogileFS? //jarkko -- Jarkko Laine http://jlaine.net http://dotherightthing.com http://www.railsecommerce.com http://odesign.fi -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20061128/e5aa7db4/attachment.bin
Matt: I definitely want to use the mongrel_upload_progress* (I realize I call it something different earlier :-p) Can you post your setup and how you have this running with one mongrel? Are you using DRb? (I am and I just ran ruby lib/upload.rb and it printed the following: 1164740868: Added 1164740868: Finished 1164740868: Added 1164740868: Marking 1164740868: Checking ... Last 2 lines Many times 1164740868: Finished But again. I have no idea where the file went. :-/ -rjs- On 11/28/06, Matt White <stockliasteroid at gmail.com> wrote:> > If you use something like mongrel_upload_progress, the single-threadedness > of Rails is no problem because Mongrel will intercept the upload and not > hand it off to Rails until it''s complete. That''s what I''m doing at the > moment, and my app runs just fine on one Mongrel. > > If you''re running large file uploads, I would recommend looking into it > anyway, because your users will surely want some kind of update about how > their upload is going. > > Matt > > On 11/29/06, Vishnu Gopal <g.vishnu at gmail.com> wrote: > > > > If you''re using Rails with Mongrel, file upload currently sucks. If you > > have two mongrels running (say) and two users upload two large files, your > > entire app will be hung just serving those requests. This is because Rails > > can''t run safely multi-threaded, and it''s not Mongrel''s fault. > > > > An upload only Mongrel could (and should) be written in a framework > > other than Rails. Both Camping and Merb are good choices. In this instance > > you don''t need many mongrels since even 1 can simultaneously serve many file > > uploads. So you''ll ideally have 1 server where you have one or more mongrels > > running your file upload app, and use that to accept uploads. This is what > > we do at SlideShare: we have a custom Camping app to accept file uploads. > > > > Your frontend webserver or load balancer would redirect all requests to > > /upload (say) to your uploader mongrel running at it''s own port. And all > > other requests can be served some other way. We do this through lighttpd > > proxying. > > > > HTH > > Vish > > > > On 11/28/06, Rogelio J. Samour < rogelio.samour at gmail.com> wrote: > > > > > > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > > > not have mongrel installed. It does proxy requests to several other > > > mongrel-only servers (each running 2 mongrel processes). Each mongrel > > > node has the same rails code-base and it''s working perfectly. > > > > > > However, my question is when I add an upload file form... where is it > > > going to physically put that file? I mean since it''s hitting either > > > one > > > node or the other, so how does mongrel deal with that? and how or > > > where > > > do I tell it to accept large files (100mb+) ? > > > > > > I have read on having an upload-only mongrel process, but again how > > > will > > > all the other mongrels know to post the file to it or to a specific > > > folder somewhere? Direction on this would be greatly appreciated! > > > > > > Thanks for your help on this. I searched the forum and didn''t find > > > anything specifically dealing with this. I have also checked the > > > mongrel > > > docs and most of the setups talk about mongrel_upload_progress plugin > > > and/or the mongrel_cluster setup...(I may have missed something :-/) > > > > > > > > > cheers, > > > > > > -rjs- > > > > > > p.s. mongrel ROCKS! > > > > > > _______________________________________________ > > > Mongrel-users mailing list > > > Mongrel-users at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > -- > Matt White > ----------------------- > Thermal Creative > http://blog.thermalcreative.com > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061128/12cc91e5/attachment.html
Rogelio,
It will arrive in Rails just like any other uploaded file... It will show up
in params[:model][:file_field], or whatever you''ve named your input
file
field. Until you do something else with it, it only lives as a file in
/tmp/. You need to move it somewhere else, though, because Rails will do
away with it once the request is complete :)
Something like:
@file = params[:model][:file_field]
File.open("/path/to/#{@file.original_filename}", "wb") {
|file| file.write(@
file.read) }
Would copy the file from it''s temp location to
/path/to/original_filename on
your box. As you were wondering, it will end up on whatever machine is
hosting the mongrel that the client was connected to. So, you may need to
scp it to an assets server or something when it''s done...
That''s as far as I
can go without knowing more about your setup ;)
Hope this helps!
Matt
On 11/29/06, Rogelio Samour <rogelio.samour at gmail.com>
wrote:>
> Philip: at this point I want to upload something... anything. I''m
> following the example on the mongrel_upload_plugin and though it says
> finished I don''t see the file anywhere. I Turned on --debug and I
can only
> see that it''s at least getting the headers (content-type, etc) on
the
> uploaded file... but nothing on the whereabouts.
>
> Like I mentioned before, I''m very green when it comes to
mongrel...
> however I''m decently versed in sysadmin *nix stuff... apache and
friends.
> I''m just needing a nudge in the right direction. :-)
>
> cheers,
>
> -rjs-
>
> On 11/28/06, Philip Hallstrom <mongrel at philip.pjkh.com> wrote:
> >
> > > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that
does
> > > not have mongrel installed. It does proxy requests to several
other
> > > mongrel-only servers (each running 2 mongrel processes). Each
mongrel
> > > node has the same rails code-base and it''s working
perfectly.
> > >
> > > However, my question is when I add an upload file form... where
is it
> > > going to physically put that file? I mean since it''s
hitting either
> > one
> > > node or the other, so how does mongrel deal with that? and how or
> > where
> > > do I tell it to accept large files (100mb+) ?
> >
> > Others have given good advice on how to do it, but wanted to say you
> > don''t
> > need to do anything to get it to accept 100mb files. We do that quite
a
> >
> > bit and while it takes awhile works just fine. It does tie up a
mongrel
> > process, but we''re okay with that.
> >
> > >
> > > I have read on having an upload-only mongrel process, but again
how
> > will
> > > all the other mongrels know to post the file to it or to a
specific
> > > folder somewhere? Direction on this would be greatly appreciated!
> > >
> > > Thanks for your help on this. I searched the forum and
didn''t find
> > > anything specifically dealing with this. I have also checked the
> > mongrel
> > > docs and most of the setups talk about mongrel_upload_progress
plugin
> > > and/or the mongrel_cluster setup...(I may have missed something
:-/)
> > >
> > >
> > > cheers,
> > >
> > > -rjs-
> > >
> > > p.s. mongrel ROCKS!
> > >
> > > _______________________________________________
> > > Mongrel-users mailing list
> > > Mongrel-users at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
>
--
Matt White
-----------------------
Thermal Creative
http://blog.thermalcreative.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/02b23577/attachment-0001.html
I don''t know anything about the mongrel_upload_plugin... sorry... We do our uploads through standard rails... see their wiki or info... On Tue, 28 Nov 2006, Rogelio Samour wrote:> Philip: at this point I want to upload something... anything. I''m following > the example on the mongrel_upload_plugin and though it says finished I don''t > see the file anywhere. I Turned on --debug and I can only see that it''s at > least getting the headers (content-type, etc) on the uploaded file... but > nothing on the whereabouts. > > Like I mentioned before, I''m very green when it comes to mongrel... however > I''m decently versed in sysadmin *nix stuff... apache and friends. I''m just > needing a nudge in the right direction. :-) > > cheers, > > -rjs- > > On 11/28/06, Philip Hallstrom <mongrel at philip.pjkh.com> wrote: >> >> > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does >> > not have mongrel installed. It does proxy requests to several other >> > mongrel-only servers (each running 2 mongrel processes). Each mongrel >> > node has the same rails code-base and it''s working perfectly. >> > >> > However, my question is when I add an upload file form... where is it >> > going to physically put that file? I mean since it''s hitting either one >> > node or the other, so how does mongrel deal with that? and how or where >> > do I tell it to accept large files (100mb+) ? >> >> Others have given good advice on how to do it, but wanted to say you don''t >> need to do anything to get it to accept 100mb files. We do that quite a >> bit and while it takes awhile works just fine. It does tie up a mongrel >> process, but we''re okay with that. >> >> > >> > I have read on having an upload-only mongrel process, but again how will >> > all the other mongrels know to post the file to it or to a specific >> > folder somewhere? Direction on this would be greatly appreciated! >> > >> > Thanks for your help on this. I searched the forum and didn''t find >> > anything specifically dealing with this. I have also checked the mongrel >> > docs and most of the setups talk about mongrel_upload_progress plugin >> > and/or the mongrel_cluster setup...(I may have missed something :-/) >> > >> > >> > cheers, >> > >> > -rjs- >> > >> > p.s. mongrel ROCKS! >> > >> > _______________________________________________ >> > Mongrel-users mailing list >> > Mongrel-users at rubyforge.org >> > http://rubyforge.org/mailman/listinfo/mongrel-users >> > >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> >
On Nov 28, 2006, at 1:26 PM, Rogelio Samour wrote:> I would like to combine what you guys are suggesting. Namely, have > a merb app (mongrel) dealio... be the sole file uploader listener. > I understand the ReWrite rules and ProxyPass magic. However, this > poses a new question: how would I go about downloading these > uploaded files? I''m assuming I would want this mongrel node to also > be a file server, correct? > > My only experience with setting up mongrel is through > mongrel_rails. cremes: are you saying this upload-only mongrel is > it''s own setup? if so, is there docs on setting up something like > this? >Well, there aren''t any specific docs explaining how to do this configuration. There are docs for setting up a standalone Mongrel. Then there are other docs which show how to create a mongrel cluster and proxy it with Apache. Just combine the two techniques and you''re done. It really is as simple as it sounds. If you have specific problems that you need the group''s help to overcome, post here and we''ll try. cr
> If you use something like mongrel_upload_progress, the single-threadedness > of Rails is no problem because Mongrel will intercept the upload and not > hand it off to Rails until it''s complete. That''s what I''m doing at the > moment, and my app runs just fine on one Mongrel.I haven''t used this plugin, but just gave it a quick read through... one thing I didn''t see mentioned... can I skip the upload progress stuff and just get the benefits of it not blocking mongrel/rails?> > If you''re running large file uploads, I would recommend looking into it > anyway, because your users will surely want some kind of update about how > their upload is going. > > Matt > > On 11/29/06, Vishnu Gopal <g.vishnu at gmail.com> wrote: >> >> If you''re using Rails with Mongrel, file upload currently sucks. If you >> have two mongrels running (say) and two users upload two large files, your >> entire app will be hung just serving those requests. This is because Rails >> can''t run safely multi-threaded, and it''s not Mongrel''s fault. >> >> An upload only Mongrel could (and should) be written in a framework other >> than Rails. Both Camping and Merb are good choices. In this instance you >> don''t need many mongrels since even 1 can simultaneously serve many file >> uploads. So you''ll ideally have 1 server where you have one or more >> mongrels >> running your file upload app, and use that to accept uploads. This is what >> we do at SlideShare: we have a custom Camping app to accept file uploads. >> >> Your frontend webserver or load balancer would redirect all requests to >> /upload (say) to your uploader mongrel running at it''s own port. And all >> other requests can be served some other way. We do this through lighttpd >> proxying. >> >> HTH >> Vish >> >> On 11/28/06, Rogelio J. Samour <rogelio.samour at gmail.com> wrote: >> > >> > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does >> > not have mongrel installed. It does proxy requests to several other >> > mongrel-only servers (each running 2 mongrel processes). Each mongrel >> > node has the same rails code-base and it''s working perfectly. >> > >> > However, my question is when I add an upload file form... where is it >> > going to physically put that file? I mean since it''s hitting either one >> > node or the other, so how does mongrel deal with that? and how or where >> > do I tell it to accept large files (100mb+) ? >> > >> > I have read on having an upload-only mongrel process, but again how will >> > all the other mongrels know to post the file to it or to a specific >> > folder somewhere? Direction on this would be greatly appreciated! >> > >> > Thanks for your help on this. I searched the forum and didn''t find >> > anything specifically dealing with this. I have also checked the mongrel >> > docs and most of the setups talk about mongrel_upload_progress plugin >> > and/or the mongrel_cluster setup...(I may have missed something :-/) >> > >> > >> > cheers, >> > >> > -rjs- >> > >> > p.s. mongrel ROCKS! >> > >> > _______________________________________________ >> > Mongrel-users mailing list >> > Mongrel-users at rubyforge.org >> > http://rubyforge.org/mailman/listinfo/mongrel-users >> > >> >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> >> > > > -- > Matt White > ----------------------- > Thermal Creative > http://blog.thermalcreative.com >
On Nov 28, 2006, at 1:50 PM, Matt White wrote:> Rogelio, > > It will arrive in Rails just like any other uploaded file... It > will show up in params[:model][:file_field], or whatever you''ve > named your input file field. Until you do something else with it, > it only lives as a file in /tmp/. You need to move it somewhere > else, though, because Rails will do away with it once the request > is complete :) > > Something like: > > @file = params[:model][:file_field] > File.open("/path/to/#{@file.original_filename}", "wb") { |file| > file.write(@file.read) } > > Would copy the file from it''s temp location to /path/to/ > original_filename on your box. As you were wondering, it will end > up on whatever machine is hosting the mongrel that the client was > connected to. So, you may need to scp it to an assets server or > something when it''s done... That''s as far as I can go without > knowing more about your setup ;) >This is what I''d like to improve in Mongrel. It''s really wasteful to write the file to a TempFile and then rewrite it again elsewhere. For really big files (multi-gigabyte) the performance hit would be tremendous. I''m thinking furiously about how to properly manage this. If anyone has any ideas, feel free to share them on the list. cr
On Tue, 28 Nov 2006 13:35:32 -0600 (CST) Philip Hallstrom <mongrel at philip.pjkh.com> wrote:> > I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > > not have mongrel installed. It does proxy requests to several other > > mongrel-only servers (each running 2 mongrel processes). Each mongrel > > node has the same rails code-base and it''s working perfectly. > > > > However, my question is when I add an upload file form... where is it > > going to physically put that file? I mean since it''s hitting either one > > node or the other, so how does mongrel deal with that? and how or where > > do I tell it to accept large files (100mb+) ? > > Others have given good advice on how to do it, but wanted to say you don''t > need to do anything to get it to accept 100mb files. We do that quite a > bit and while it takes awhile works just fine. It does tie up a mongrel > process, but we''re okay with that.Quick clarification to what Philip said: The actual upload doesn''t block mongrel since it''s done in a thread before rails gets the uploaded body. Now, once Rails gets it the cgi.rb multipart mime processing is run on the resulting file which can eat your CPU for about 20 seconds/100mb of file (based on my observations). -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Tue, 28 Nov 2006 11:40:20 -0600 "Rogelio J. Samour" <rogelio.samour at gmail.com> wrote:> I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > not have mongrel installed. It does proxy requests to several other > mongrel-only servers (each running 2 mongrel processes). Each mongrel > node has the same rails code-base and it''s working perfectly. > > However, my question is when I add an upload file form... where is it > going to physically put that file? I mean since it''s hitting either one > node or the other, so how does mongrel deal with that? and how or where > do I tell it to accept large files (100mb+) ?You really want to look at mongrel_upload_progress and check out how you can do your own standalone upload service. Several folks proposed different apache configs for you to try out. When you break a small standalone Mongrel out of the main Rails setup and use it as the upload target you can basically handle hundreds of uploads concurrently without clustering. I''m gonna do a small write-up on this since I just did it for travelistic.com, but the main thing you need to know (which is found in the simple mongrel_upload_progress code) is that the file is streamed to /tmp/ and then handed to your rails app as a complete file. You don''t need to do your own copying inside Ruby (as someone else suggested) but instead can just do a simple move to where you want it. Next thing to know is that mongrel streams the file in small chunks of about 16k, so if the file is 200mb you don''t actually load 200mb into ram. Now, cgi.rb might load all 200mb and I know fastcgi does load the whole file. Then the files are uploaded as multipart content, and you need to use cgi.rb to process them since that''s about the best there is for ruby right now. I''ve got code in the works that uses a different algorithm and will make multipart mime much more efficient. Until then you''ve gotta put up with the sudden CPU bursts you get from cgi.rb after the file is upload. Finally, and this is very important, while the file is written to disk in small chunks and this upload doesn''t block Rails actions going on, when you finally pass this to Rails you''ll block that Mongrel process while cgi.rb is going. *THIS* is why you should make a separate Mongrel handler to do all of your upload processing and file preparation before you pass the fully cooked stuff to Rails. Mongrel running cgi.rb in a thread is much more efficient than Rails running cgi.rb inside a lock. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Tue, 28 Nov 2006 14:20:06 -0600 cremes.devlist at mac.com wrote:> > On Nov 28, 2006, at 1:50 PM, Matt White wrote: > > > Rogelio, > > > > It will arrive in Rails just like any other uploaded file... It > > will show up in params[:model][:file_field], or whatever you''ve > > named your input file field. Until you do something else with it, > > it only lives as a file in /tmp/. You need to move it somewhere > > else, though, because Rails will do away with it once the request > > is complete :) > > > > Something like: > > > > @file = params[:model][:file_field] > > File.open("/path/to/#{@file.original_filename}", "wb") { |file| > > file.write(@file.read) }Matt, don''t do this. In fact, if there''s code in Mongrel doing this then point it out to me. You already have the fully formed file handle, all you need to do is use fileutils to just rename it. This is an ultra fast operation. Don''t copy it in a loop inside ruby.> This is what I''d like to improve in Mongrel. It''s really wasteful to > write the file to a TempFile and then rewrite it again elsewhere. For > really big files (multi-gigabyte) the performance hit would be > tremendous.Matt''s mistaken Cremes, this isn''t how it''s done.> I''m thinking furiously about how to properly manage this. If anyone > has any ideas, feel free to share them on the list.You probably have been missing the entire chunk of code I and Ezra have been working on to solve this problem. Including a modified string search algorithm that will find the mime boundaries. If you wanna help on this, e-mail me off line with your ideas and I''ll help you check out the source. I haven''t been ignoring your other ideas, they''re just kind of secondary to the Mongrel 1.0 RC1 release. I''m pushing for this stuff in more of a Mongrel 1.1 or later release. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Tue, 28 Nov 2006 14:17:37 -0600 cremes.devlist at mac.com wrote:> > On Nov 28, 2006, at 1:26 PM, Rogelio Samour wrote: > > > I would like to combine what you guys are suggesting. Namely, have > > a merb app (mongrel) dealio... be the sole file uploader listener. > > I understand the ReWrite rules and ProxyPass magic. However, this > > poses a new question: how would I go about downloading these > > uploaded files? I''m assuming I would want this mongrel node to also > > be a file server, correct? > > > > My only experience with setting up mongrel is through > > mongrel_rails. cremes: are you saying this upload-only mongrel is > > it''s own setup? if so, is there docs on setting up something like > > this? > > > > Well, there aren''t any specific docs explaining how to do this > configuration. There are docs for setting up a standalone Mongrel. > Then there are other docs which show how to create a mongrel cluster > and proxy it with Apache. Just combine the two techniques and you''re > done. It really is as simple as it sounds. > > If you have specific problems that you need the group''s help to > overcome, post here and we''ll try.Yep, it''s not hard, the main thing is to read up on the Configurator and to create your own startup script. It''s maybe 20-60 lines of code depending on how complex you want to get. Take a look at the mongrel_rails script and the Configurator documentation in the RDoc: http://mongrel.rubyforge.org/rdoc/ I''ve also already done this for a company and they''ve given me permission to write it up and release it for other people to use. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Tue, 28 Nov 2006 14:20:00 -0600 (CST) Philip Hallstrom <mongrel at philip.pjkh.com> wrote:> > If you use something like mongrel_upload_progress, the single-threadedness > > of Rails is no problem because Mongrel will intercept the upload and not > > hand it off to Rails until it''s complete. That''s what I''m doing at the > > moment, and my app runs just fine on one Mongrel. > > I haven''t used this plugin, but just gave it a quick read through... one > thing I didn''t see mentioned... can I skip the upload progress stuff and > just get the benefits of it not blocking mongrel/rails?Yep, actually that''s an incredibly trivial thing to do. You''d probably be better off just starting with a small handler that moved the files where you wanted after the upload, and then expand it to your needs. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Tue, 28 Nov 2006 21:46:06 +0200 Jarkko Laine <jarkko at jlaine.net> wrote:> On 28.11.2006, at 21.30, Matt White wrote: > > > If you use something like mongrel_upload_progress, the single- > > threadedness of Rails is no problem because Mongrel will intercept > > the upload and not hand it off to Rails until it''s complete. That''s > > what I''m doing at the moment, and my app runs just fine on one > > Mongrel. > > > > If you''re running large file uploads, I would recommend looking > > into it anyway, because your users will surely want some kind of > > update about how their upload is going. > > Out of curiosity, has anyone managed to pair mongrel_upload_progress > (or any of the other options) with MogileFS?That''s been mentioned in the past, but I believe it wouldn''t work since the file has to be streamed to the local disk and then pushed up to MogileFS. Since disks are so cheap it turns out to be cheaper to just upload the file to a big disk and then use a fast web server to serve the files. Now, if you have a post processing stage you could benefit from this. I know several folks that, after they process the files, push them up to Amazon S3. Similar things can be done with MogileFS too. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Ha! I wondered if I would manage to incur the wrath of Zed...
He is, as always, correct... That''s what I get for having outdated
Rails
wiki code stuck in my head.
require ''fileutils''
@file = params[:model][:file_field]
FileUtils.mv @file.local_path, "/path/to/#{@file.original_filename}"
I assume this is what I should do instead, correct?
And yeah, I''ll second Zed''s comment on S3... That''s
what I''m doing, in
conjunction with Backgroundrb, because of some post-processing. Works great.
Matt
On 11/29/06, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:>
> On Tue, 28 Nov 2006 14:20:06 -0600
> cremes.devlist at mac.com wrote:
>
> >
> > On Nov 28, 2006, at 1:50 PM, Matt White wrote:
> >
> > > Rogelio,
> > >
> > > It will arrive in Rails just like any other uploaded file... It
> > > will show up in params[:model][:file_field], or whatever
you''ve
> > > named your input file field. Until you do something else with it,
> > > it only lives as a file in /tmp/. You need to move it somewhere
> > > else, though, because Rails will do away with it once the request
> > > is complete :)
> > >
> > > Something like:
> > >
> > > @file = params[:model][:file_field]
> > > File.open("/path/to/#{@file.original_filename}",
"wb") { |file|
> > > file.write(@file.read) }
>
> Matt, don''t do this. In fact, if there''s code in Mongrel
doing this then
> point it out to me.
>
> You already have the fully formed file handle, all you need to do is use
> fileutils to just rename it. This is an ultra fast operation.
Don''t copy
> it in a loop inside ruby.
>
> > This is what I''d like to improve in Mongrel. It''s
really wasteful to
> > write the file to a TempFile and then rewrite it again elsewhere. For
> > really big files (multi-gigabyte) the performance hit would be
> > tremendous.
>
> Matt''s mistaken Cremes, this isn''t how it''s
done.
>
> > I''m thinking furiously about how to properly manage this. If
anyone
> > has any ideas, feel free to share them on the list.
>
> You probably have been missing the entire chunk of code I and Ezra have
> been working on to solve this problem. Including a modified string search
> algorithm that will find the mime boundaries. If you wanna help on this,
> e-mail me off line with your ideas and I''ll help you check out the
source.
>
> I haven''t been ignoring your other ideas, they''re just
kind of secondary
> to the Mongrel 1.0 RC1 release. I''m pushing for this stuff in
more of a
> Mongrel 1.1 or later release.
>
> --
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
--
Matt White
-----------------------
Thermal Creative
http://blog.thermalcreative.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/886817fa/attachment.html
On 28.11.2006, at 22.41, Zed A. Shaw wrote:> On Tue, 28 Nov 2006 21:46:06 +0200 > Jarkko Laine <jarkko at jlaine.net> wrote: >> Out of curiosity, has anyone managed to pair mongrel_upload_progress >> (or any of the other options) with MogileFS? > > That''s been mentioned in the past, but I believe it wouldn''t work > since the file has to be streamed to the local disk and then pushed > up to MogileFS. Since disks are so cheap it turns out to be > cheaper to just upload the file to a big disk and then use a fast > web server to serve the files.The problem with this, I think, is that you won''t get the benefits of no single point of failure (unless you use an expensive RAID array or SAN setup). Also, if you have app servers on multiple machines, you would have to setup NFS or something if you have anything else than the file to be added through the form (i.e. you can''t send the entire form to a process dedicated just for file uploads). So I guess something like a BackgrounDRB process running the Robot Co-Op''s Mogile library might be the way to go. //jarkko -- Jarkko Laine http://jlaine.net http://dotherightthing.com http://www.railsecommerce.com http://odesign.fi -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20061128/ee37b16d/attachment-0001.bin
>>> I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does >>> not have mongrel installed. It does proxy requests to several other >>> mongrel-only servers (each running 2 mongrel processes). Each mongrel >>> node has the same rails code-base and it''s working perfectly. >>> >>> However, my question is when I add an upload file form... where is it >>> going to physically put that file? I mean since it''s hitting either one >>> node or the other, so how does mongrel deal with that? and how or where >>> do I tell it to accept large files (100mb+) ? >> >> Others have given good advice on how to do it, but wanted to say you don''t >> need to do anything to get it to accept 100mb files. We do that quite a >> bit and while it takes awhile works just fine. It does tie up a mongrel >> process, but we''re okay with that. > > Quick clarification to what Philip said: The actual upload doesn''t > block mongrel since it''s done in a thread before rails gets the uploaded > body. Now, once Rails gets it the cgi.rb multipart mime processing is > run on the resulting file which can eat your CPU for about 20 > seconds/100mb of file (based on my observations).You lost me a bit... are you saying the actual upload doesn''t block mongrel with or without the plugin? And when you say "mongrel" do you really mean "rails within mongrel" or just mongrel? I''m thinking you mean that uploads don''t block *mongrel* ever, but without the plugin uploads block rails within mongrel which affectively blocks any other rails requests. Yes?
> On Tue, 28 Nov 2006 14:20:00 -0600 (CST) > Philip Hallstrom <mongrel at philip.pjkh.com> wrote: > >>> If you use something like mongrel_upload_progress, the single-threadedness >>> of Rails is no problem because Mongrel will intercept the upload and not >>> hand it off to Rails until it''s complete. That''s what I''m doing at the >>> moment, and my app runs just fine on one Mongrel. >> >> I haven''t used this plugin, but just gave it a quick read through... one >> thing I didn''t see mentioned... can I skip the upload progress stuff and >> just get the benefits of it not blocking mongrel/rails? > > Yep, actually that''s an incredibly trivial thing to do. You''d probably > be better off just starting with a small handler that moved the files > where you wanted after the upload, and then expand it to your needs.Yeah, you''re right. I grabbed the Rick''s svn repo and ran his example in about 10 minutes... That''s pretty slick :) Slick enough I might keep that progress bar :) -philip
Holy crap! I leave for an hour and find 24 messages on this thread! Take about a nudge on the right direction... more like a shove! Thanks amigos! Zed... et al! Great job! I''ll keep you guys posted. Also once I figure all this out... I''ll do a write-up. I''m also investigating how all this will work with nginx (which I assume will be more efficient!) cheers, -rjs- On 11/28/06, Philip Hallstrom <mongrel at philip.pjkh.com> wrote:> > >>> I have an Apache 2.2.3 (mod_proxy_balancer) frontend server that does > >>> not have mongrel installed. It does proxy requests to several other > >>> mongrel-only servers (each running 2 mongrel processes). Each mongrel > >>> node has the same rails code-base and it''s working perfectly. > >>> > >>> However, my question is when I add an upload file form... where is it > >>> going to physically put that file? I mean since it''s hitting either > one > >>> node or the other, so how does mongrel deal with that? and how or > where > >>> do I tell it to accept large files (100mb+) ? > >> > >> Others have given good advice on how to do it, but wanted to say you > don''t > >> need to do anything to get it to accept 100mb files. We do that quite > a > >> bit and while it takes awhile works just fine. It does tie up a > mongrel > >> process, but we''re okay with that. > > > > Quick clarification to what Philip said: The actual upload doesn''t > > block mongrel since it''s done in a thread before rails gets the uploaded > > body. Now, once Rails gets it the cgi.rb multipart mime processing is > > run on the resulting file which can eat your CPU for about 20 > > seconds/100mb of file (based on my observations). > > You lost me a bit... are you saying the actual upload doesn''t block > mongrel with or without the plugin? And when you say "mongrel" do you > really mean "rails within mongrel" or just mongrel? > > I''m thinking you mean that uploads don''t block *mongrel* ever, but without > the plugin uploads block rails within mongrel which affectively blocks any > other rails requests. > > Yes? > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061128/32ad2c12/attachment.html
There is an example merb controller for use with the mongrel_upload_progress if anyone is interested in using a small merb app for their upload app server. We have a number of customers doing this and some of them are handling a ton of file uploads all the time. Merb can also be a nice secure download manager. The Files controller in the example app in the merb svn shows how easy it is to do: http://svn.devjavu.com/merb/examples/sample_app/dist/app/controllers/ files.rb You can do a svn checkout of the merb svn here: $ svn co http://svn.devjavu.com/merb And build it and play with the example app like this: $ cd merb $ sudo rake install $ cd examples/sample_app $ merb -p 4000 Then go to http://localhost:4000/files/start in your browser to see an example of the upload progress. Sice it is running locally you shoudl try to upload a file at least 100Mb or bigger or else it will happen too fast and you won''t see the progress bar. The nice thing about the way merb works is that it doesn''t use cgi.rb for mime parsing and it is written in a more thread safe way then actionpack. So multiple concurrent requests can be processed and have their mime parsing done all at the same time. Merb only does a mutex around the smallest possible place where you might be calling ActiveRecord code. With the rails dispatcher, the mutex surrounds the entire routing code and mime parsing and your controler code. Merb only locks around the action in your controller code. It does route recognition and mime parsing outside the mutex in a multi threaded way. This allows us to use ActiveRecord in single threaded mode which is better then AR''s multi threaded mode for performance. So we use the smallest sized lock we can in order to still support this, but we do all the heavyweight routing and mime parsing concurrently. I promised Zed I would do a write up about using a merb app for a dedicated upload server in conjunction with a rails app. I will try to get something written this week and posted. Until then I am open to and questions about merb and you can file tickets for support or feature requests here: http://merb.devjavu.com/ http://merb.devjavu.com/projects/merb/register Zed and I hope to come up with a solution so that we can use the upload progress without reparsing the mime twice per file. Right now mongrel streams the upload to a tmp file and increments the progress bar all the way to 100% before it hands control to your rails or merb controller to deal with the file. But then rails or merb has to parse the mime boundaries of the entire file again. So you may notice with big files that right when the progress bar makes it to 95-100% it will pause while rails or meb parses the mime again into another tmpfile. This is obviously not an optimal way to do this. Zed has started an awesome fast C mime carver and I will be working with him to integrate this so uploads can be parsed once with the fast C mimer and then just handed off to the next handler with no need to reparse. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)
Rogelio, Nginx will buffer the entire request before sending it to any mongrels... This probably isn''t what you want when dealing with large uploads ;). Check out: http://wiki.codemongers.com/NginxHttpProxyModule Ezra mentioned a while back that the author of Nginx may fix this issue in the future, but for now your requests won''t make it through Nginx to Rails (or a dedicated upload mongrel) until the upload is complete... Matt On 11/29/06, Philip Hallstrom <mongrel at philip.pjkh.com> wrote:> > > On Tue, 28 Nov 2006 14:20:00 -0600 (CST) > > Philip Hallstrom <mongrel at philip.pjkh.com> wrote: > > > >>> If you use something like mongrel_upload_progress, the > single-threadedness > >>> of Rails is no problem because Mongrel will intercept the upload and > not > >>> hand it off to Rails until it''s complete. That''s what I''m doing at the > >>> moment, and my app runs just fine on one Mongrel. > >> > >> I haven''t used this plugin, but just gave it a quick read through... > one > >> thing I didn''t see mentioned... can I skip the upload progress stuff > and > >> just get the benefits of it not blocking mongrel/rails? > > > > Yep, actually that''s an incredibly trivial thing to do. You''d probably > > be better off just starting with a small handler that moved the files > > where you wanted after the upload, and then expand it to your needs. > > Yeah, you''re right. I grabbed the Rick''s svn repo and ran his example in > about 10 minutes... > > That''s pretty slick :) Slick enough I might keep that progress bar :) > > -philip > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-- Matt White ----------------------- Thermal Creative http://blog.thermalcreative.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/12603ffa/attachment.html
On Tue, 28 Nov 2006 15:26:04 -0600 (CST) Philip Hallstrom <mongrel at philip.pjkh.com> wrote:> > Quick clarification to what Philip said: The actual upload doesn''t > > block mongrel since it''s done in a thread before rails gets the uploaded > > body. Now, once Rails gets it the cgi.rb multipart mime processing is > > run on the resulting file which can eat your CPU for about 20 > > seconds/100mb of file (based on my observations). > > You lost me a bit... are you saying the actual upload doesn''t block > mongrel with or without the plugin? And when you say "mongrel" do you > really mean "rails within mongrel" or just mongrel? > > I''m thinking you mean that uploads don''t block *mongrel* ever, but without > the plugin uploads block rails within mongrel which affectively blocks any > other rails requests.Here''s the process: 1) Mongrel accepts the client, creates a thread, and processes the headers into an HttpRequest object. 2) Then it hits the body, it looks at the content-length and sets up to read that much. 2a) If the body is over 16k then it is streamed to a Tempfile. Otherwise into a StringIO. 3) Mongrel then reads 16k chunks and writes the results into the IO (Tempfile or StringIO). 4) Once the body is complete, and the content-length checks out, then the HttpRequest is complete. 5) The HttpRequest and an HttpResponse is passed to the RailsHandler for processing. 6) LOCK! 7) Rails runs cgi.rb and a bunch of other stuff on the uploaded file. At this point you aren''t blocked by the remote client''s IO since it''s already saved by Mongrel, but cgi.rb is really slow so it''ll take a while depending on the size of the file to parse out the multi-part mime. 9) Rails processes the request and sends back the response (which is also stuffed into a StringIO). 10) UNLOCK! 11) Mongrel now takes the response headers and response body and sends those to the remote client fully threaded. So, there''s only a short part of the process where Rails is actually blocking the upload. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Tue, 28 Nov 2006 23:08:56 +0200 Jarkko Laine <jarkko at jlaine.net> wrote:> On 28.11.2006, at 22.41, Zed A. Shaw wrote: > > > On Tue, 28 Nov 2006 21:46:06 +0200 > > Jarkko Laine <jarkko at jlaine.net> wrote: > >> Out of curiosity, has anyone managed to pair mongrel_upload_progress > >> (or any of the other options) with MogileFS? > > > > That''s been mentioned in the past, but I believe it wouldn''t work > > since the file has to be streamed to the local disk and then pushed > > up to MogileFS. Since disks are so cheap it turns out to be > > cheaper to just upload the file to a big disk and then use a fast > > web server to serve the files. > > The problem with this, I think, is that you won''t get the benefits of > no single point of failure (unless you use an expensive RAID array or > SAN setup).So you''re saying a couple of web servers running apache or nginx to serve files right off a server''s disk is LESS reliable than a nearly unproven MogileFS setup that uses mysql, trackers, storage nodes, and then eventually just transmits off HTTP anyway? Got any metrics to back this claim? -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
>>>> Out of curiosity, has anyone managed to pair mongrel_upload_progress >>>> (or any of the other options) with MogileFS? >>> >>> That''s been mentioned in the past, but I believe it wouldn''t work >>> since the file has to be streamed to the local disk and then pushed >>> up to MogileFS. Since disks are so cheap it turns out to be >>> cheaper to just upload the file to a big disk and then use a fast >>> web server to serve the files. >> >> The problem with this, I think, is that you won''t get the benefits of >> no single point of failure (unless you use an expensive RAID array or >> SAN setup). > > So you''re saying a couple of web servers running apache or nginx to > serve files right off a server''s disk is LESS reliable than a nearly > unproven MogileFS setup that uses mysql, trackers, storage nodes, and > then eventually just transmits off HTTP anyway?I think he was referring to "big disk... web server", and the obvious lack of plurals... one disk on one web server is giong to give you a single point of failure... What we did was to have a "master media" server that everything gets uploaded to. Then we have several slaves that are configured such that if they don''t have the file requested they get it from the master. We also have things setup so that the master will request each file from the slave upon it receiving a new file so within about no time we''ve replicated the file to several slaves and on the off chance one gets uploaded manually it will get picked up... Works for us...
Ezra,
Great job on merb! And the myriad of other projects you''re involved
in!
Quick question:
- I followed your instructions checked out the latest merb trunk (rev 99)
- rake installed (after installing all it''s dependencies... do you mind
listing those on the README somewhere? :)
- cd''d to the sample_app, and proceeded to upload a 500mb iso image
- the problem: it spins its wheels until it''s finished and then it
shows:
File uploaded successfully "UPLOADED:
{"commit"=>"Upload",
"action"=>"upload", "id"=>nil,
"controller"=>"files",
"upload_id"=>"1164816252",
"data"=>{"name"=>"data",
"type"=>"application/x-cd-image",
"tempfile"=>#,
"filename"=>"debian-31r2-i386-netinst.iso"}}
- so it''s uploading fine but it''s not showing the progress of
the upload.
- what am I doing wrong?
thanks again!
cheers,
-rjs-
here''s what my terminal shows:
{"action"=>"progress", "id"=>nil,
"controller"=>"files",
"upload_id"=>"1164816252"}(this one several times)
skip mutex (this one several times)
{"action"=>"progress", "id"=>nil,
"controller"=>"files",
"upload_id"=>"1164816252"}(this one several times)
=> fh[type] = application/x-cd-image
skip mutex (this one several times)
{"action"=>"progress", "id"=>nil,
"controller"=>"files",
"upload_id"=>"1164816252"}(this one several times)
{"commit"=>"Upload",
"action"=>"upload", "id"=>nil,
"controller"=>"files",
"upload_id"=>"1164816252",
"data"=>{"name"=>"data",
"type"=>"application/x-cd-image",
"tempfile"=>#<File:/tmp/Merb7817.0>,
"filename"=>"debian-31r2-i386-netinst.iso"}}
skip mutex
On 11/28/06, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:>
>
> There is an example merb controller for use with the
> mongrel_upload_progress if anyone is interested in using a small
> merb app for their upload app server. We have a number of customers
> doing this and some of them are handling a ton of file uploads all
> the time. Merb can also be a nice secure download manager. The Files
> controller in the example app in the merb svn shows how easy it is to
> do:
>
> http://svn.devjavu.com/merb/examples/sample_app/dist/app/controllers/
> files.rb
>
> You can do a svn checkout of the merb svn here:
>
> $ svn co http://svn.devjavu.com/merb
>
> And build it and play with the example app like this:
>
> $ cd merb
> $ sudo rake install
> $ cd examples/sample_app
> $ merb -p 4000
>
> Then go to http://localhost:4000/files/start in your browser to
> see
> an example of the upload progress. Sice it is running locally you
> shoudl try to upload a file at least 100Mb or bigger or else it will
> happen too fast and you won''t see the progress bar.
>
> The nice thing about the way merb works is that it doesn''t
use
> cgi.rb for mime parsing and it is written in a more thread safe way
> then actionpack. So multiple concurrent requests can be processed and
> have their mime parsing done all at the same time. Merb only does a
> mutex around the smallest possible place where you might be calling
> ActiveRecord code. With the rails dispatcher, the mutex surrounds the
> entire routing code and mime parsing and your controler code. Merb
> only locks around the action in your controller code. It does route
> recognition and mime parsing outside the mutex in a multi threaded
> way. This allows us to use ActiveRecord in single threaded mode which
> is better then AR''s multi threaded mode for performance. So we use
> the smallest sized lock we can in order to still support this, but we
> do all the heavyweight routing and mime parsing concurrently.
>
> I promised Zed I would do a write up about using a merb app for a
> dedicated upload server in conjunction with a rails app. I will try
> to get something written this week and posted. Until then I am open
> to and questions about merb and you can file tickets for support or
> feature requests here:
>
> http://merb.devjavu.com/
> http://merb.devjavu.com/projects/merb/register
>
> Zed and I hope to come up with a solution so that we can use the
> upload progress without reparsing the mime twice per file. Right now
> mongrel streams the upload to a tmp file and increments the progress
> bar all the way to 100% before it hands control to your rails or merb
> controller to deal with the file. But then rails or merb has to parse
> the mime boundaries of the entire file again. So you may notice with
> big files that right when the progress bar makes it to 95-100% it
> will pause while rails or meb parses the mime again into another
> tmpfile. This is obviously not an optimal way to do this. Zed has
> started an awesome fast C mime carver and I will be working with him
> to integrate this so uploads can be parsed once with the fast C mimer
> and then just handed off to the next handler with no need to reparse.
>
> Cheers-
>
> -- Ezra Zygmuntowicz
> -- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/65a54e43/attachment-0001.html
Ezra:
Doh! The @upstatus var is never set because of the following:
Params: {"action"=>"progress", "id"=>nil,
"controller"=>"files",
"upload_id"=>"1164816252"}
Routing to controller: Files action: progress
Parsing HTTP Input took: 0.038226 seconds
uninitialized constant Mongrel::Uploads - (NameError)
/usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:100:in
`const_missing''
/home/rsamour/public_html/merb/examples/sample_app/dist/app/controllers/files.rb:22:in
`progress''
The problem stems from the files.rb file (line 22 and 23). I know this is
straight from the mup example the
docs<http://mongrel.rubyforge.org/docs/upload_progress.html>
I''m missing a require somewhere... or the example missing it? hmmm
-rjs-
p.s. I do have the mup gem installed.
On 11/29/06, Rogelio Samour <rogelio.samour at gmail.com>
wrote:>
> Ezra,
>
> Great job on merb! And the myriad of other projects you''re
involved in!
>
> Quick question:
> - I followed your instructions checked out the latest merb trunk (rev 99)
> - rake installed (after installing all it''s dependencies... do you
mind
> listing those on the README somewhere? :)
> - cd''d to the sample_app, and proceeded to upload a 500mb iso
image
> - the problem: it spins its wheels until it''s finished and then it
> shows:
>
> File uploaded successfully "UPLOADED:
{"commit"=>"Upload",
> "action"=>"upload", "id"=>nil,
>
> "controller"=>"files",
"upload_id"=>"1164816252",
>
> "data"=>{"name"=>"data",
>
> "type"=>"application/x-cd-image",
>
> "tempfile"=>#,
>
> "filename"=>" debian-31r2-i386-netinst.iso"}}
> - so it''s uploading fine but it''s not showing the
progress of the upload.
> - what am I doing wrong?
>
> thanks again!
>
> cheers,
>
> -rjs-
>
> here''s what my terminal shows:
> {"action"=>"progress", "id"=>nil,
"controller"=>"files",
> "upload_id"=>"1164816252"}(this one several times)
> skip mutex (this one several times)
> {"action"=>"progress", "id"=>nil,
"controller"=>"files",
> "upload_id"=>"1164816252"} (this one several times)
> => fh[type] = application/x-cd-image
> skip mutex (this one several times)
> {"action"=>"progress", "id"=>nil,
"controller"=>"files",
> "upload_id"=>"1164816252"} (this one several times)
> {"commit"=>"Upload",
"action"=>"upload", "id"=>nil,
"controller"=>"files",
> "upload_id"=>"1164816252",
"data"=>{"name"=>"data",
> "type"=>"application/x-cd-image",
"tempfile"=>#<File:/tmp/Merb7817.0>,
> "filename"=>" debian-31r2-i386-netinst.iso"}}
> skip mutex
>
>
>
> On 11/28/06, Ezra Zygmuntowicz < ezmobius at gmail.com > wrote:
> >
> >
> > There is an example merb controller for use with the
> > mongrel_upload_progress if anyone is interested in using a small
> > merb app for their upload app server. We have a number of customers
> > doing this and some of them are handling a ton of file uploads all
> > the time. Merb can also be a nice secure download manager. The Files
> > controller in the example app in the merb svn shows how easy it is to
> > do:
> >
> > http://svn.devjavu.com/merb/examples/sample_app/dist/app/controllers/
> > files.rb
> >
> > You can do a svn checkout of the merb svn here:
> >
> > $ svn co http://svn.devjavu.com/merb
> >
> > And build it and play with the example app like this:
> >
> > $ cd merb
> > $ sudo rake install
> > $ cd examples/sample_app
> > $ merb -p 4000
> >
> > Then go to http://localhost:4000/files/start in your browser
to
> > see
> > an example of the upload progress. Sice it is running locally you
> > shoudl try to upload a file at least 100Mb or bigger or else it will
> > happen too fast and you won''t see the progress bar.
> >
> > The nice thing about the way merb works is that it
doesn''t use
> > cgi.rb for mime parsing and it is written in a more thread safe way
> > then actionpack. So multiple concurrent requests can be processed and
> > have their mime parsing done all at the same time. Merb only does a
> > mutex around the smallest possible place where you might be calling
> > ActiveRecord code. With the rails dispatcher, the mutex surrounds the
> > entire routing code and mime parsing and your controler code. Merb
> > only locks around the action in your controller code. It does route
> > recognition and mime parsing outside the mutex in a multi threaded
> > way. This allows us to use ActiveRecord in single threaded mode which
> > is better then AR''s multi threaded mode for performance. So
we use
> > the smallest sized lock we can in order to still support this, but we
> > do all the heavyweight routing and mime parsing concurrently.
> >
> > I promised Zed I would do a write up about using a merb app
for
> > a
> > dedicated upload server in conjunction with a rails app. I will try
> > to get something written this week and posted. Until then I am open
> > to and questions about merb and you can file tickets for support or
> > feature requests here:
> >
> > http://merb.devjavu.com/
> > http://merb.devjavu.com/projects/merb/register
> >
> > Zed and I hope to come up with a solution so that we can use
the
> >
> > upload progress without reparsing the mime twice per file. Right now
> > mongrel streams the upload to a tmp file and increments the progress
> > bar all the way to 100% before it hands control to your rails or merb
> > controller to deal with the file. But then rails or merb has to parse
> > the mime boundaries of the entire file again. So you may notice with
> > big files that right when the progress bar makes it to 95-100% it
> > will pause while rails or meb parses the mime again into another
> > tmpfile. This is obviously not an optimal way to do this. Zed has
> > started an awesome fast C mime carver and I will be working with him
> > to integrate this so uploads can be parsed once with the fast C mimer
> > and then just handed off to the next handler with no need to reparse.
> >
> > Cheers-
> >
> > -- Ezra Zygmuntowicz
> > -- Lead Rails Evangelist
> > -- ez at engineyard.com
> > -- Engine Yard, Serious Rails Hosting
> > -- (866) 518-YARD (9273)
> >
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/4892344c/attachment.html
On Nov 29, 2006, at 9:19 AM, Rogelio Samour wrote:> Ezra: > <snip>Hey Rogelio- Sorry I still need to write more documentation for merb. It has a lot of features that aren''t documented well yet. The secret magic to get the example to work is as follows: You need to tell merb there is a config file to be read. And in the sample app there is a mup.conf which is the config file for mongrel_upload-Progress. So you need to start the sample app like this: $ cd examples/sample_app $ merb -f dist/conf/mup.conf Right now the example will only wotk with one merb running. I am still working on the integrated drb server for when you run a merb cluster. I am happy to help with any merb issues you have. I should have the merb mailing list up shortly. Also any problems or feature requests you have please add tickets to the merb trac. I am trying to centralize where I trac issues. -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)
Coolio! That did it! I was adding this to the conf/merb.yml in hopes of getting it to work: :config_script: conf/mup.conf: Of course that didn''t work. But the -f did the job! The question now is how do you get all this to work nicely with my rails app? thanks again for merb (my new best friend) :-p -rjs- On 11/29/06, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:> > > On Nov 29, 2006, at 9:19 AM, Rogelio Samour wrote: > > > Ezra: > > <snip> > > > Hey Rogelio- > > Sorry I still need to write more documentation for merb. It has a > lot of features that aren''t documented well yet. The secret magic to > get the example to work is as follows: > > You need to tell merb there is a config file to be read. And in the > sample app there is a mup.conf which is the config file for > mongrel_upload-Progress. So you need to start the sample app like this: > > $ cd examples/sample_app > $ merb -f dist/conf/mup.conf > > Right now the example will only wotk with one merb running. I am > still working on the integrated drb server for when you run a merb > cluster. I am happy to help with any merb issues you have. I should > have the merb mailing list up shortly. Also any problems or feature > requests you have please add tickets to the merb trac. I am trying to > centralize where I trac issues. > > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20061129/413fb45f/attachment.html