My application accepts uploads from users, which can be quite huge in some cases. This of course requires setting the unicorn timeout to something much higher than 60 seconds, more like 10 minutes. Are there any drawbacks to doing this, other than the obvious drawback of not killing off long-running requests that are illegitimate? I''ve googled quite a bit about this and have found surprisingly little -- I guess people who have apps that receive uploads just generally don''t use unicorn? Thanks, John
John Joseph Bachir <j at jjb.cc> wrote:> My application accepts uploads from users, which can be quite huge in > some cases. This of course requires setting the unicorn timeout to > something much higher than 60 seconds, more like 10 minutes. > > Are there any drawbacks to doing this, other than the obvious drawback > of not killing off long-running requests that are illegitimate?Are your users remote? (outside of your immediate LAN). The upload speed (and thus timeout) for unicorn should be based on the nginx <-> unicorn transfer rate. Unicorn should never talk to users directly and users can upload slowly to nginx which buffers requests to the filesystem, first.> I''ve googled quite a bit about this and have found surprisingly little > -- I guess people who have apps that receive uploads just generally > don''t use unicorn?I have a LAN-only application that regularly processes uploads several hundreds of megabytes (via PUT) directly to Unicorn. The local disk I/O is often the limiting factor since the parallel requests fill up the kernel buffers and wait on disk I/O. The Rack application itself unfortunately needs to seek/rewind so it must be in the filesystem. Rainbows! with ThreadSpawn/ThreadPool can process uploads without buffering them to disk first (but the Rack multipart parser may not). I often stream several hundred megabytes of data directly to apps on Rainbows! via PUT requests (curl -T), too.
You need to be buffering the upload before sending it to Unicorn. Are you using nginx? We''re stuck on Apache, so I just did all my uploading / image-manipulation with nodejs instead. Clifton
On Thu, Sep 29, 2011 at 2:47 PM, Eric Wong <normalperson at yhbt.net> wrote:> The upload speed (and thus timeout) for unicorn should be based on > the nginx <-> unicorn transfer rate. ?Unicorn should never talk to users > directly and users can upload slowly to nginx which buffers requests to the > filesystem, first.On Thu, Sep 29, 2011 at 2:48 PM, <cliftonk at gmail.com> wrote:> You need to be buffering the upload before sending it to Unicorn. Are > you using nginx?Thanks for the responses! So, I''ve see this module: https://github.com/vkholodkov/nginx-upload-module Is that what you are referring to? Are there other solutions? Does NGINX have inherent behavior which buffers requests? (Eric''s email seemed to suggest that). Thanks, john
John Joseph Bachir <j at jjb.cc> wrote:> On Thu, Sep 29, 2011 at 2:47 PM, Eric Wong <normalperson at yhbt.net> wrote: > > The upload speed (and thus timeout) for unicorn should be based on > > the nginx <-> unicorn transfer rate. ?Unicorn should never talk to users > > directly and users can upload slowly to nginx which buffers requests to the > > filesystem, first. > > On Thu, Sep 29, 2011 at 2:48 PM, <cliftonk at gmail.com> wrote: > > You need to be buffering the upload before sending it to Unicorn. Are > > you using nginx? > > Thanks for the responses! So, I''ve see this module: > https://github.com/vkholodkov/nginx-upload-moduleI think that''s unrelated to what you''re doing (it appears to be for nginx-only setups).> Is that what you are referring to? Are there other solutions? Does > NGINX have inherent behavior which buffers requests? (Eric''s email > seemed to suggest that).Yes, nginx buffers entire requests by default before sending them to unicorn.
On Thu, Sep 29, 2011 at 4:05 PM, John Joseph Bachir <j at jjb.cc> wrote:> Does NGINX have inherent behavior which buffers requests? (Eric''s email > seemed to suggest that).Nginx always buffers the entire body of requests before calling the upstream application. If the body exceeds client_body_buffer_size Nginx will store it in a tempfile.> Thanks for the responses! So, I''ve see this module: > https://github.com/vkholodkov/nginx-upload-moduleThe upload module works a little differently. It allows you to configure specific upload endpoints that Nginx will preprocess. Rather than forwarding the entire request body to the upstream application, Nginx will parse the multipart/form-data encoded files, store them in tempfiles, and forward only the paths of the tempfiles to the upstream application. Justin
So buffering the entire request and then sending it to the backend is the default behavior of HttpUpstreamModule? That''s fascinating. It almost seems like that''s a bad design choice, because it inhibits the possibility of parallel work being done as the request comes in (like, it''s the opposite of how unix pipes work).
John Joseph Bachir <j at jjb.cc> wrote:> So buffering the entire request and then sending it to the backend is > the default behavior of HttpUpstreamModule? That''s fascinating. It > almost seems like that''s a bad design choice, because it inhibits the > possibility of parallel work being done as the request comes in (like, > it''s the opposite of how unix pipes work).Yes, exactly. It''s a design trade-off; if a client is uploading slowly (whether intentionally or maliciously) the process-per-client model of Unicorn is /extremely/ inefficient and will easily be shut down by slow clients. If you want to be able to process uploads in a Rack application while the client is uploading, you''d have to ensure your app is thread-safe and use Rainbows! + ThreadSpawn/ThreadPool.