Backgroundrb devel - Mar 2007 - example code for file upload

I have an archiving application that handles large file uploads.  I  
already use backgroundrb to run a behind the scenes bulk import of  
records.  All good.  Thank you Ezra.

Now I would like to add long running file uploads to the things that  
backgroundrb does for me.  That''s right that ole chestnut. 
I''ve seen
a lot of mention of this practice around the web, but have yet come  
across good example code about how this is done.  At least not beyond  
the ajax progress bar aspect.

I guess I''m looking for answer to how one gets the stream of the  
uploading file submitted by the user from mongrel to a backgroundrb  
before the upload actually completes?

Cheers,
Walter

On Mar 29, 2007, at 8:18 PM, Walter McGinnis wrote:
> I have an archiving application that handles large file uploads.  I
> already use backgroundrb to run a behind the scenes bulk import of
> records.  All good.  Thank you Ezra.
>
> Now I would like to add long running file uploads to the things that
> backgroundrb does for me.  That''s right that ole chestnut. 
I''ve seen
> a lot of mention of this practice around the web, but have yet come
> across good example code about how this is done.  At least not beyond
> the ajax progress bar aspect.
>
> I guess I''m looking for answer to how one gets the stream of the
> uploading file submitted by the user from mongrel to a backgroundrb
> before the upload actually completes?
>
> Cheers,
> Walter
>
Hey Walter-

	You don''t. BackgrounDrb cannot handle file uploads. You probably  
want to look at the mongrel_upload_progress gem. Bdrb can do progress  
bars for tasks it runs but the cotnrol flow of http requests and  
rails does not allow you to do anything with the file stream until  
its fully uploaded already so bdrb never has a chance to get involved.

Cheers-

-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

On Mar 31, 2007, at 4:00 AM, Ezra Zygmuntowicz wrote:
>
> On Mar 29, 2007, at 8:18 PM, Walter McGinnis wrote:
>
>>
>> I guess I''m looking for answer to how one gets the stream of
the
>> uploading file submitted by the user from mongrel to a backgroundrb
>> before the upload actually completes?
>>
>
> Hey Walter-
>
> 	You don''t. BackgrounDrb cannot handle file uploads. You probably
> want to look at the mongrel_upload_progress gem. Bdrb can do  
> progress bars for tasks it runs but the cotnrol flow of http  
> requests and rails does not allow you to do anything with the file  
> stream until its fully uploaded already so bdrb never has a chance  
> to get involved.
>
That makes sense.  Thanks for clearing that up.  I had already looked  
at the doc on the mongrel progress bar gem.  I''ll return to it.

My real problem is uploads over crap network connections turning into  
CPU eating zombie mongrels.  Man, I love it when geek speak sounds  
like a horror movie premise, but in this case it''s a real pain.  I  
suspect that the solution for that is going to be mongrel based  
though.  I''ll RTFM over in mongrel doc land, but any hints are  
appreciated.

Cheers,
Walter

On Mar 30, 2007, at 4:36 PM, Walter McGinnis wrote:
> My real problem is uploads over crap network connections turning into
> CPU eating zombie mongrels.  Man, I love it when geek speak sounds
> like a horror movie premise, but in this case it''s a real pain.  I
> suspect that the solution for that is going to be mongrel based
> though.  I''ll RTFM over in mongrel doc land, but any hints are
> appreciated.
Walter,

for very large files (200+ MB) I recommend using Apache and the  
mod_put extension [1]. I added an extension to mod_put so it would  
support resumable PUT operations which is very handy for "crap  
network connections." :-)

I also created a patch for Mongrel to allow it to better handle large  
file uploads, but it hasn''t been accepted by Zed. I can provide a  
copy to you if you are interested.

cr

[1] http://www.gknw.at/development/apache/httpd-2.0/unix/modules/ 
mod_put-2.0.8.tar.gz