Rails - May 2006 - Stop browser timeout with large file uploads + processing?

Hi

Im uploading a file to my rails app, i''s usually a large file, 5-50mb. 
Once the file has been uploaded it must be parsed and processed, this 
can take a very long time, 1-20mins.  The problem is that most of the 
time  my browser timeouts.

Im not really sure how to solve it.  One idea is that i have a script 
run constantly on the server that checks for uploaded files in a 
specified folder.  This would prevent timeouts.  The user would simply 
upload the file and be shown a screen telling them that the data will be 
in the system in 1-20 mins.

Im not sure if this is an ideal solution though.  What are your 
thoughts? Does anyone have experience with this or have  better 
solution?

-- 
Posted via http://www.ruby-forum.com/.

Chris Richards wrote:
> Hi
> 
> Im uploading a file to my rails app, i''s usually a large file,
5-50mb.
> Once the file has been uploaded it must be parsed and processed, this 
> can take a very long time, 1-20mins.  The problem is that most of the 
> time  my browser timeouts.
> 
> Im not really sure how to solve it.  One idea is that i have a script 
> run constantly on the server that checks for uploaded files in a 
> specified folder.  This would prevent timeouts.  The user would simply 
> upload the file and be shown a screen telling them that the data will be 
> in the system in 1-20 mins.
> 
> Im not sure if this is an ideal solution though.  What are your 
> thoughts? Does anyone have experience with this or have  better 
> solution?
After the browser uploads the file trigger a background process with an 
ajax call. This then at various points in this long running process 
update a session variable with the progress on your long action:

  def long_action
    #do stuff
    session[:progress] = 0.1
    session.update

    #do stuff
    session[:progress] = 0.2
    session.update
  end

Then have a periodic AJAX call to retrieve a progress bar partial based 
on the session[:progress]

  <p>Progress: <%= (session[:progress] * 100).to_i %>%</p>

I have used this technique with an action that uploads some large chunks 
of data to external ftp servers, runs about 10-30 minutes.  The ajax 
keeps the browser from getting bored.  Note this will not work in 
webrick.  It require a minimum of 2 server processes and Webrick only 
uses 1.  You need one proc to do the long running action, and one proc 
to respond to the ajax queries.

You might even be able to do the same thing with the uploading of the 
file itself as well, but I haven''t actually pulled that one off.  Check
out the file_upload_progress plugin (i think thats whats its called) for 
more info on ajaxy uploads.

-- 
Posted via http://www.ruby-forum.com/.

Hi~

On May 5, 2006, at 10:01 AM, Alex Wayne wrote:
> Chris Richards wrote:
>> Hi
>>
>> Im uploading a file to my rails app, i''s usually a large file,
>> 5-50mb.
>> Once the file has been uploaded it must be parsed and processed, this
>> can take a very long time, 1-20mins.  The problem is that most of the
>> time  my browser timeouts.
>>
>> Im not really sure how to solve it.  One idea is that i have a script
>> run constantly on the server that checks for uploaded files in a
>> specified folder.  This would prevent timeouts.  The user would  
>> simply
>> upload the file and be shown a screen telling them that the data  
>> will be
>> in the system in 1-20 mins.
>>
>> Im not sure if this is an ideal solution though.  What are your
>> thoughts? Does anyone have experience with this or have  better
>> solution?
>
> After the browser uploads the file trigger a background process  
> with an
> ajax call. This then at various points in this long running process
> update a session variable with the progress on your long action:
>
>   def long_action
>     #do stuff
>     session[:progress] = 0.1
>     session.update
>
>     #do stuff
>     session[:progress] = 0.2
>     session.update
>   end
>
> Then have a periodic AJAX call to retrieve a progress bar partial  
> based
> on the session[:progress]
>
>   <p>Progress: <%= (session[:progress] * 100).to_i
%>%</p>
>
> I have used this technique with an action that uploads some large  
> chunks
> of data to external ftp servers, runs about 10-30 minutes.  The ajax
> keeps the browser from getting bored.  Note this will not work in
> webrick.  It require a minimum of 2 server processes and Webrick only
> uses 1.  You need one proc to do the long running action, and one proc
> to respond to the ajax queries.
>
> You might even be able to do the same thing with the uploading of the
> file itself as well, but I haven''t actually pulled that one off.
> Check
> out the file_upload_progress plugin (i think thats whats its  
> called) for
> more info on ajaxy uploads.
	I have a plugin that will be released in the next few days that is  
precisely written to accomodate this kind of long running background  
process. It will even work with just one webrick or fcgi or whatever.  
You can see some screencasts and the thoughts behind it here on my  
blog. Stay tuned for the released code which is forthcoming RSN.

http://brainspl.at/articles/2006/05/04/preview-of-drbworker-plugin- 
with-ajax-progress-bars

http://brainspl.at/drb_progress.mov

http://brainspl.at/drb_ajax_tail.mov


Cheers-
-Ezra

Ezra~

That log tailer looks amazing! I can''t wait!

~ Ben

On 5/5/06, Ezra Zygmuntowicz <ezmobius@gmail.com>
wrote:
> Hi~
>
> On May 5, 2006, at 10:01 AM, Alex Wayne wrote:
>
> > Chris Richards wrote:
> >> Hi
> >>
> >> Im uploading a file to my rails app, i''s usually a large
file,
> >> 5-50mb.
> >> Once the file has been uploaded it must be parsed and processed,
this
> >> can take a very long time, 1-20mins.  The problem is that most of
the
> >> time  my browser timeouts.
> >>
> >> Im not really sure how to solve it.  One idea is that i have a
script
> >> run constantly on the server that checks for uploaded files in a
> >> specified folder.  This would prevent timeouts.  The user would
> >> simply
> >> upload the file and be shown a screen telling them that the data
> >> will be
> >> in the system in 1-20 mins.
> >>
> >> Im not sure if this is an ideal solution though.  What are your
> >> thoughts? Does anyone have experience with this or have  better
> >> solution?
> >
> > After the browser uploads the file trigger a background process
> > with an
> > ajax call. This then at various points in this long running process
> > update a session variable with the progress on your long action:
> >
> >   def long_action
> >     #do stuff
> >     session[:progress] = 0.1
> >     session.update
> >
> >     #do stuff
> >     session[:progress] = 0.2
> >     session.update
> >   end
> >
> > Then have a periodic AJAX call to retrieve a progress bar partial
> > based
> > on the session[:progress]
> >
> >   <p>Progress: <%= (session[:progress] * 100).to_i
%>%</p>
> >
> > I have used this technique with an action that uploads some large
> > chunks
> > of data to external ftp servers, runs about 10-30 minutes.  The ajax
> > keeps the browser from getting bored.  Note this will not work in
> > webrick.  It require a minimum of 2 server processes and Webrick only
> > uses 1.  You need one proc to do the long running action, and one proc
> > to respond to the ajax queries.
> >
> > You might even be able to do the same thing with the uploading of the
> > file itself as well, but I haven''t actually pulled that one
off.
> > Check
> > out the file_upload_progress plugin (i think thats whats its
> > called) for
> > more info on ajaxy uploads.
>
>         I have a plugin that will be released in the next few days that is
> precisely written to accomodate this kind of long running background
> process. It will even work with just one webrick or fcgi or whatever.
> You can see some screencasts and the thoughts behind it here on my
> blog. Stay tuned for the released code which is forthcoming RSN.
>
> http://brainspl.at/articles/2006/05/04/preview-of-drbworker-plugin-
> with-ajax-progress-bars
>
> http://brainspl.at/drb_progress.mov
>
> http://brainspl.at/drb_ajax_tail.mov
>
>
> Cheers-
> -Ezra
>
>
>
>
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>

--
Ben Reubenstein
303-947-0446
http://www.benr75.com