Rails - Mar 2008 - offloading big uploads to front-end webserver

I''m doing a website which has quite a few big uploads / downloads to
manage.

Since rails isn''t multithreaded (yet), this means every big upload/
download will keep a mongrel/thin/whatever thread for itself doing
nothing but waiting for the transfer to finish.

Now, for public files, this of course gets handled by my frontend
nginx server.

For private files, I use the X-Accel-Redirect header, so the transfer
gets offloaded to nginx as well.

Only for uploads, I couldn''t find a nice way to handle them.
Is there a way to somehow tell nginx to first receive the whole file,
then transfer it to mongrel/thin so it doesn''t lock up a full
instance?
Any way to do this, while first consulting rails if this should be
allowed?

Any help is greatly appreciated.

Greetings,
Mathijs
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

On 25 Mar 2008, at 22:45, bluescreen303-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
wrote:
> I''m doing a website which has quite a few big uploads / downloads
to
> manage.
>
> Since rails isn''t multithreaded (yet), this means every big
upload/
> download will keep a mongrel/thin/whatever thread for itself doing
> nothing but waiting for the transfer to finish.
>
> Now, for public files, this of course gets handled by my frontend
> nginx server.
>
> For private files, I use the X-Accel-Redirect header, so the transfer
> gets offloaded to nginx as well.
>
> Only for uploads, I couldn''t find a nice way to handle them.
> Is there a way to somehow tell nginx to first receive the whole file,
> then transfer it to mongrel/thin so it doesn''t lock up a full
> instance?
> Any way to do this, while first consulting rails if this should be
> allowed?
First of all, Mongrel is multithreaded and Mongrel doesn''t hand off  
the uploaded file to Rails until after the upload itself (i.e. the  
upload itself doesn''t block the mongrel at all). That said, after the  
upload the Mongrel will be locked for the duration of processing in  
Rails.

Depending on what you are actually doing in Rails, you have several  
possibilities:
1) Just saving the file to disk: just let Rails handle it using a  
Tempfile, see the attachment_fu source code. It''s pretty fast and  
won''t lock your Mongrel for long since the Rails request, which IS  
single threaded, won''t take long
2) Extensive processing: offload the upload to a background process,  
and here you have another choice:
      - BackgroundRB (and other background processors): has access to  
the Rails stack and runs behind Rails, i.e. Rails will hand over the  
file to the background worker
      - Merb: a Rails like framework that contrary to Rails is  
multithreaded, it can sit behind a single mongrel and process heaps of  
uploads without locking the mongrel. You can plugin ActiveRecord into  
merb for model access. You''ll also need some rewrite magic to direct  
uploads to the Merb application if you want the rest served through  
Rails.

Best regards

Peter De Berdt


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---