Mongrel users - Feb 2007 - Mongrel upload progress not showing progress on production server

Hi all,

Tried out the mongrel upload progress plugin with Drb and it works
great on my OSX development box, but when putting it into production
(Ubuntu Dapper), uploads complete but the app isn''t returning any
values for upload progress, and uploads are not showing up in the
queue when running upload_client.rb. Before anyone asks, yes, I''m
running both the mongrel instances and the upload.rb script.

Below are the relevant log entries and code snippets. Thanks!

data_file.rb:

?

  def progress
    warn ''progress''
    render :update do |page|
      warn ''update''
      @status = Mongrel::Uploads.check(params[:upload_id])
      warn @status.inspect
      page.upload_progress.update(@status[:size], @status[:received]) if @status
    end
  end

?

mongrel.log:

...
progress
update
nil
progress
update
nil
progress
update
nil
?

production.log:

Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
22:16:29) [POST]
  Parameters: {"action"=>"progress",
"controller"=>"data_files",
"upload_id"=>"1172268962"}
Completed in 0.00706 (141 reqs/sec) | Rendering: 0.00667 (94%) | 200
OK [http://central.incomumdesign.com/files/progress?upload_id=1172268962]


Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
22:16:29) [POST]
  Parameters: {"action"=>"progress",
"controller"=>"data_files",
"upload_id"=>"1172268962"}
Completed in 0.00854 (117 reqs/sec) | Rendering: 0.00813 (95%) | DB:
0.00000 (0%) | 200 OK
[http://central.incomumdesign.com/files/progress?upload_id=1172268962]


Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
22:16:32) [POST]
  Parameters: {"action"=>"progress",
"controller"=>"data_files",
"upload_id"=>"1172268962"}
Completed in 0.00670 (149 reqs/sec) | Rendering: 0.00635 (94%) | 200
OK [http://central.incomumdesign.com/files/progress?upload_id=1172268962]


Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
22:16:35) [POST]
  Parameters: {"action"=>"progress",
"controller"=>"data_files",
"upload_id"=>"1172268962"}
Completed in 0.00759 (131 reqs/sec) | Rendering: 0.00719 (94%) | DB:
0.00000 (0%) | 200 OK
[http://central.incomumdesign.com/files/progress?upload_id=1172268962]


Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
22:16:38) [POST]
  Parameters: {"action"=>"progress",
"controller"=>"data_files",
"upload_id"=>"1172268962"}
Completed in 0.00802 (124 reqs/sec) | Rendering: 0.00757 (94%) | 200
OK [http://central.incomumdesign.com/files/progress?upload_id=1172268962]


Processing DataFilesController#upload (for 201.17.52.1 at 2007-02-23
22:16:40) [POST]
  Session ID: 23b0c52d05a16439015cb313a05a7660
  Parameters: {"commit"=>"Upload",
"xhr"=>"true", "action"=>"upload",
"controller"=>"data_files",
"upload_id"=>"1172268962",
"data_file"=>{"description"=>""},
"data"=>#<File:/tmp/CGI12410.0>}
Completed in 0.26787 (3 reqs/sec) | Rendering: 0.00011 (0%) | DB:
0.25820 (96%) | 200 OK
[http://central.incomumdesign.com/files/upload?upload_id=1172268962&xhr=true]

Ben,

Check a few things first:
  - Make sure the backgroundDRb server and worker processes are  
running on your production box
  - Make sure that both slave and daemons gems (required by  
backgroundDRb) are installed in the production environment.

   ~Wayne

On Feb 27, 2007, at 09:40 , Benjamin Jackson wrote:
> Hi all,
>
> Tried out the mongrel upload progress plugin with Drb and it works
> great on my OSX development box, but when putting it into production
> (Ubuntu Dapper), uploads complete but the app isn''t returning any
> values for upload progress, and uploads are not showing up in the
> queue when running upload_client.rb. Before anyone asks, yes, I''m
> running both the mongrel instances and the upload.rb script.
>
> Below are the relevant log entries and code snippets. Thanks!
>
> data_file.rb:
>
> ?
>
>   def progress
>     warn ''progress''
>     render :update do |page|
>       warn ''update''
>       @status = Mongrel::Uploads.check(params[:upload_id])
>       warn @status.inspect
>       page.upload_progress.update(@status[:size], @status 
> [:received]) if @status
>     end
>   end
>
> ?
>
> mongrel.log:
>
> ...
> progress
> update
> nil
> progress
> update
> nil
> progress
> update
> nil
> ?
>
> production.log:
>
> Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
> 22:16:29) [POST]
>   Parameters: {"action"=>"progress",
"controller"=>"data_files",
> "upload_id"=>"1172268962"}
> Completed in 0.00706 (141 reqs/sec) | Rendering: 0.00667 (94%) | 200
> OK [http://central.incomumdesign.com/files/progress? 
> upload_id=1172268962]
>
>
> Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
> 22:16:29) [POST]
>   Parameters: {"action"=>"progress",
"controller"=>"data_files",
> "upload_id"=>"1172268962"}
> Completed in 0.00854 (117 reqs/sec) | Rendering: 0.00813 (95%) | DB:
> 0.00000 (0%) | 200 OK
> [http://central.incomumdesign.com/files/progress?upload_id=1172268962]
>
>
> Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
> 22:16:32) [POST]
>   Parameters: {"action"=>"progress",
"controller"=>"data_files",
> "upload_id"=>"1172268962"}
> Completed in 0.00670 (149 reqs/sec) | Rendering: 0.00635 (94%) | 200
> OK [http://central.incomumdesign.com/files/progress? 
> upload_id=1172268962]
>
>
> Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
> 22:16:35) [POST]
>   Parameters: {"action"=>"progress",
"controller"=>"data_files",
> "upload_id"=>"1172268962"}
> Completed in 0.00759 (131 reqs/sec) | Rendering: 0.00719 (94%) | DB:
> 0.00000 (0%) | 200 OK
> [http://central.incomumdesign.com/files/progress?upload_id=1172268962]
>
>
> Processing DataFilesController#progress (for 201.17.52.1 at 2007-02-23
> 22:16:38) [POST]
>   Parameters: {"action"=>"progress",
"controller"=>"data_files",
> "upload_id"=>"1172268962"}
> Completed in 0.00802 (124 reqs/sec) | Rendering: 0.00757 (94%) | 200
> OK [http://central.incomumdesign.com/files/progress? 
> upload_id=1172268962]
>
>
> Processing DataFilesController#upload (for 201.17.52.1 at 2007-02-23
> 22:16:40) [POST]
>   Session ID: 23b0c52d05a16439015cb313a05a7660
>   Parameters: {"commit"=>"Upload",
"xhr"=>"true", "action"=>"upload",
> "controller"=>"data_files",
"upload_id"=>"1172268962",
> "data_file"=>{"description"=>""},
"data"=>#<File:/tmp/CGI12410.0>}
> Completed in 0.26787 (3 reqs/sec) | Rendering: 0.00011 (0%) | DB:
> 0.25820 (96%) | 200 OK
> [http://central.incomumdesign.com/files/upload? 
> upload_id=1172268962&xhr=true]
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

On Tue, 27 Feb 2007 11:40:23 -0300
"Benjamin Jackson" <bhjackson at gmail.com> wrote:
> Hi all,
> 
> Tried out the mongrel upload progress plugin with Drb and it works
> great on my OSX development box, but when putting it into production
> (Ubuntu Dapper), uploads complete but the app isn''t returning any
> values for upload progress, and uploads are not showing up in the
> queue when running upload_client.rb. Before anyone asks, yes, I''m
> running both the mongrel instances and the upload.rb script.
If it''s behind nginx or other servers (apache?) sometimes they buffer
the request before sending the status back.  Otherwise, grab firebug
and use it to see what is going on.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

> If it''s behind nginx or other servers (apache?) sometimes they
buffer
> the request before sending the status back.  Otherwise, grab firebug
> and use it to see what is going on.
Thanks Zed. It is in fact behind nginx with gzip compression. How can
I work around this? By increasing/decreasing the buffer size? Here''s
the relevant (I think) section of my nginx config:

    http {
      include       conf/mime.types;
      default_type  application/octet-stream;
      access_log  logs/access.log;
      sendfile     on;
      tcp_nopush     on;
      keepalive_timeout  65;
      tcp_nodelay        on;
      gzip  on;
      gzip_min_length  1100;
      gzip_buffers     4 8k;
      gzip_types       text/plain;
      ...

Is there a good tutorial for mongrel upload progress, which includes a
setup like the one you''re describing? I looked at the one on the
mongrel site, and it got me only so far before I needed to abandon the
idea, since it was becoming a lot of work for little return. My main
issue was trying to use it for something other than strictly uploading
files with no other information.

Thanks

On 2/28/07, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> On Tue, 27 Feb 2007 23:32:13 -0300
> "Benjamin Jackson" <bhjackson at gmail.com> wrote:
>
> > > If it''s behind nginx or other servers (apache?)
sometimes they buffer
> > > the request before sending the status back.  Otherwise, grab
firebug
> > > and use it to see what is going on.
> >
> > Thanks Zed. It is in fact behind nginx with gzip compression. How can
> > I work around this? By increasing/decreasing the buffer size?
Here''s
> > the relevant (I think) section of my nginx config:
>
> You can''t.  Nginx''s author is coming out with a version
that fixes
> this, but otherwise it''s just inherent in the design.
>
> Your best bet actually is to just do these uploads to a completely
> offline standalone Mongrel server so you don''t tie up rails. 
It''s about
> the only compromise there is other than just not doing it in the first
> place.
>
> Even if nginx changes it''s request dispatching method,
you''ll still
> have a problem with locking up rails for each of the polling requests.
>
> --
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

On Tue, 27 Feb 2007 23:32:13 -0300
"Benjamin Jackson" <bhjackson at gmail.com> wrote:
> > If it''s behind nginx or other servers (apache?) sometimes
they buffer
> > the request before sending the status back.  Otherwise, grab firebug
> > and use it to see what is going on.
> 
> Thanks Zed. It is in fact behind nginx with gzip compression. How can
> I work around this? By increasing/decreasing the buffer size?
Here''s
> the relevant (I think) section of my nginx config:
You can''t.  Nginx''s author is coming out with a version that
fixes
this, but otherwise it''s just inherent in the design.

Your best bet actually is to just do these uploads to a completely
offline standalone Mongrel server so you don''t tie up rails. 
It''s about
the only compromise there is other than just not doing it in the first
place.

Even if nginx changes it''s request dispatching method, you''ll
still
have a problem with locking up rails for each of the polling requests.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Zed,
> > Thanks Zed. It is in fact behind nginx with gzip compression. How can
> > I work around this? By increasing/decreasing the buffer size?
Here''s
> > the relevant (I think) section of my nginx config:
>
> You can''t.  Nginx''s author is coming out with a version
that fixes
> this, but otherwise it''s just inherent in the design.
>
> Your best bet actually is to just do these uploads to a completely
> offline standalone Mongrel server so you don''t tie up rails. 
It''s about
> the only compromise there is other than just not doing it in the first
> place.
By "offline" you mean not serving rails, right? This would require the
first of the two options presented on the site:
> This is fine and dandy, but not too many sites run on a single Mongrel.
You''ll quickly
> run into problems with multiple mongrels since only one Mongrel process
knows about
> the upload. You''ll either have to specify a specific mongrel port
to communicate with,
> or set up a dedicated mongrel upload process. The third option, is use DRb.
Let me know if I''m on track, or if not please clarify. Thanks :)

Hey Zed,
> Honestly, the idea of upload progress done this way (with polling http
> requests) is a super bad idea in my book.  It''s currently a hack
to
> work around a broken UI in the browser, so I try not to steer people in
> this direction.
>
> What you should do, if you''re interested, is contact Ezra at
> ez at engineyard.com as he does custom setups of this using Merb.  If you
> want to do it yourself, http://merb.devjavu.com/ has example code in it
> that you can use.
What would you suggest to someone who needs to allow clients to upload
files of more than a meg or two if not an upload progress script? It''s
difficult to tell someone that they''ll need to stare at their monitor
for a half hour to an hour while the upload churns on with no progress
:P

On Wed, 28 Feb 2007 09:59:43 -0600
"Joey Geiger" <jgeiger at gmail.com> wrote:
> Is there a good tutorial for mongrel upload progress, which includes a
> setup like the one you''re describing? I looked at the one on the
> mongrel site, and it got me only so far before I needed to abandon the
> idea, since it was becoming a lot of work for little return. My main
> issue was trying to use it for something other than strictly uploading
> files with no other information.
Honestly, the idea of upload progress done this way (with polling http
requests) is a super bad idea in my book.  It''s currently a hack to
work around a broken UI in the browser, so I try not to steer people in
this direction.

What you should do, if you''re interested, is contact Ezra at
ez at engineyard.com as he does custom setups of this using Merb.  If you
want to do it yourself, http://merb.devjavu.com/ has example code in it
that you can use.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On Feb 28, 2007, at 2:04 PM, Benjamin Jackson wrote:
> Hey Zed,
>
>> Honestly, the idea of upload progress done this way (with polling  
>> http
>> requests) is a super bad idea in my book.  It''s currently a
hack to
>> work around a broken UI in the browser, so I try not to steer  
>> people in
>> this direction.
>>
>> What you should do, if you''re interested, is contact Ezra at
>> ez at engineyard.com as he does custom setups of this using Merb.  If  
>> you
>> want to do it yourself, http://merb.devjavu.com/ has example code  
>> in it
>> that you can use.
>
> What would you suggest to someone who needs to allow clients to upload
> files of more than a meg or two if not an upload progress script?
It''s
> difficult to tell someone that they''ll need to stare at their
monitor
> for a half hour to an hour while the upload churns on with no progress
> :P

	Using a flash uploader widget is the best way you can do this  
currently. You can have a nice multiple uploads at a time flash  
widget  that does the progress bar on the client side like it should  
be. This means no polling *and* you get nice progress bars and can  
select multiple files at once.


Cheers-
-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

On Wed, 28 Feb 2007 17:41:56 -0300
"Benjamin Jackson" <bhjackson at gmail.com> wrote:
> Zed,
> 
> > > Thanks Zed. It is in fact behind nginx with gzip compression. How
can
> > > I work around this? By increasing/decreasing the buffer size?
Here''s
> > > the relevant (I think) section of my nginx config:
> >
> > You can''t.  Nginx''s author is coming out with a
version that fixes
> > this, but otherwise it''s just inherent in the design.
> >
> > Your best bet actually is to just do these uploads to a completely
> > offline standalone Mongrel server so you don''t tie up rails. 
It''s about
> > the only compromise there is other than just not doing it in the first
> > place.
> 
> By "offline" you mean not serving rails, right? This would
require the
> first of the two options presented on the site:
No, by "offline" I mean a separate server that uses only Mongrel (not
rails) to handle all uploads and just the upload/status part.  Using
iframes, redirects, and the session cookies with data stored in the db
it''s possible to pull this off and nobody will know.  I''ve
done it a
few times, and Ezra''s got quite a few under his belt.
> > This is fine and dandy, but not too many sites run on a single
Mongrel. You''ll quickly
> > run into problems with multiple mongrels since only one Mongrel
process knows about
> > the upload. You''ll either have to specify a specific mongrel
port to communicate with,
> > or set up a dedicated mongrel upload process. The third option, is use
DRb.
Many sites however run single Mongrel''s for single purposes.  This is
the great thing about a small fast web server.  You don''t *have* to use
rails for everything, and can distribute the work to specialized
servers that do stuff like this in a lightweight fashion.  Basically,
it''s not an either/or proposition.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On Wed, 28 Feb 2007 19:04:25 -0300
"Benjamin Jackson" <bhjackson at gmail.com> wrote:
> Hey Zed,
> 
> > Honestly, the idea of upload progress done this way (with polling http
> > requests) is a super bad idea in my book.  It''s currently a
hack to
> > work around a broken UI in the browser, so I try not to steer people
in
> > this direction.
> >
> > What you should do, if you''re interested, is contact Ezra at
> > ez at engineyard.com as he does custom setups of this using Merb.  If
you
> > want to do it yourself, http://merb.devjavu.com/ has example code in
it
> > that you can use.
> 
> What would you suggest to someone who needs to allow clients to upload
> files of more than a meg or two if not an upload progress script?
It''s
> difficult to tell someone that they''ll need to stare at their
monitor
> for a half hour to an hour while the upload churns on with no progress
> :P
First off, an upload of 2-3 mb won''t take so long that you have to
provide a second-by-second byte accurate feedback mechanism to them.  A
simple spinner graphic saying "just a moment" is most more than
enough.

If your application is such that you need absolute accuracy on
progress, resumable uploads, validation of the upload, etc. then don''t
use HTTP.  It doesn''t do any of this and your app will have a mountain
to climb over.

If you need to simply provide a nice way to show progress of really
large files (and again, HTTP wasn''t designed for this so
you''re
basically fixing a browser usability issue) then you should use flash.
There''s a few rather nice upload progress meters that don''t
use this
stupid 2-second HTTP poll mechanism and just do what the rest of the
world does:  keep track of what''s been sent over the socket.

In other words, using repeated HTTP requests to ask the server what the
browser has sent in a blow-by-blow accurate way so that you can show
someone what''s effectively a spinner for a few seconds or minutes is a
dumb way to use HTTP.  There''s other much better tools, especially if
you need to have the upload be reliable in any way.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.