Hi. Unicorn 3.3.1 fails with "Read error: #<IOError: closed stream>" on request with attachments bigger than ~150kb. Ruby: 1.9.2-p0 Rails: 3.0.1 Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb. Any ides why this happens?
Alexey Bondar <alexey.bondar at gmail.com> wrote:> Hi. > > Unicorn 3.3.1 fails with "Read error: #<IOError: closed stream>" on request with attachments bigger than ~150kb. > > Ruby: 1.9.2-p0 > Rails: 3.0.1 > > Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa > Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg > > I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb.Anything in the nginx error log? Did this happen with Unicorn 1.1.x? The input code changed in the 3.x series drastically to support non-rewindable input, so some bugs could''ve crept in. -- Eric Wong
Alexey Bondar <alexey.bondar at gmail.com> wrote: (back to mailing list)> On Jan 24, 2011, at 21:19 , Eric Wong wrote: > > Eric Wong <normalperson at yhbt.net> wrote: > >> Alexey Bondar <alexey.bondar at gmail.com> wrote: > >>> Hi. > >>> > >>> Unicorn 3.3.1 fails with "Read error: #<IOError: closed stream>" on request with attachments bigger than ~150kb. > >>> > >>> Ruby: 1.9.2-p0 > >>> Rails: 3.0.1 > >>> > >>> Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa > >>> Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg > >>> > >>> I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb. > >> > >> Anything in the nginx error log? Did this happen with Unicorn 1.1.x? > >> The input code changed in the 3.x series drastically to support > >> non-rewindable input, so some bugs could''ve crept in. > > In nginx log only two messages: body buffered to disk and backend timeout.Wait, nginx buffered the body to disk when client_body_buffer_size is 128mb and you have trouble with a 150kb request?> Same issue with 1.1.xOK. Which timeout exactly are you hitting in nginx? How slow is your client? Is your request queue very huge and you don''t have enough Unicorn workers to handle the load? -- Eric Wong
Alexey Bondar <alexey.bondar at gmail.com> wrote:> On Jan 25, 2011, at 03:51 , Eric Wong wrote: > > Wait, nginx buffered the body to disk when client_body_buffer_size is > > 128mb and you have trouble with a 150kb request? > > Oops, my fault: not body_buffer_size, but client_max_body_size. client_body_buffer_size is 128k. > > As I understand, in this case nginx should buffer whole request body, before sending to backend?Yes, nginx always buffers the whole request body> >> Same issue with 1.1.x > > > > OK. Which timeout exactly are you hitting in nginx? > > Logs: > 2011/01/23 21:09:05 [warn] 79095#0: *261 a client request body is buffered to a temporary file /var/spool/nginx-client-body/3/00/0000000003<snip>> 2011/01/23 21:09:35 [error] 79095#0: *261 upstream timed out (60: Operation timed out) while sending request to upstream<snip> Very strange. Do you process env["rack.input"] directly from your app or do you let Rails/Rack handle everything? This probably won''t help, but can you try increasing the proxy_send_timeout in nginx? What''s your Unicorn timeout?
Alexey Bondar <alexey.bondar at gmail.com> wrote:> Unicorn timeout is 75s. I just tried with 75s proxy_send_timeout: same > issue.What client are you using to send? I wonder if Content-Length is somehow screwed up because a client is trying to compress the data. What happens when you try hitting Unicorn directly without nginx? -- Eric Wong
Alexey Bondar <alexey.bondar at gmail.com> wrote:> On Jan 25, 2011, at 06:53 , Eric Wong wrote: > > What client are you using to send? I wonder if Content-Length is > > somehow screwed up because a client is trying to compress the data. > > Different: Webkit based browsers, FF, Opera.OK. I''m fairly certain those clients don''t make compressed requests, but I''m out of ideas as to what could be wrong...> > What happens when you try hitting Unicorn directly without nginx? > > Hm... How I can directly hit unicorn via unix socket?Just have it listen on a TCP port, too (it can handle multiple listen statements in the config). Maybe we can narrow it down to something in the nginx config... -- Eric Wong
Alexey Bondar <alexey.bondar at gmail.com> wrote:> On Jan 25, 2011, at 07:10 , Eric Wong wrote: > > OK. I''m fairly certain those clients don''t make compressed requests, > > but I''m out of ideas as to what could be wrong... > > > > Just have it listen on a TCP port, too (it can handle multiple > > listen statements in the config). Maybe we can narrow it down > > to something in the nginx config... > > Hm. Yeah, direct access works well. > Uh, what can be wrong with nginx config in this case? :(I would trim your config down the the bare minimum and use the defaults wherever you can. Then start adding things back in and see what broke it. -- Eric Wong
Alexey Bondar <alexey.bondar at gmail.com> wrote:> On Jan 25, 2011, at 07:54 , Eric Wong wrote: > > Alexey Bondar <alexey.bondar at gmail.com> wrote: > >> Hm. Yeah, direct access works well. > >> Uh, what can be wrong with nginx config in this case? :( > > > > I would trim your config down the the bare minimum and use the defaults > > wherever you can. Then start adding things back in and see what broke > > it. > > Maxim Dounin from nginx mail list says that this issue caused by some issues with sendfile in > FreeBSD 8. > > After setting sendfile to off uploads works well. > > You can add this info to known 3rd party issues. :)Thanks Alexey!>From 8ac0ae45a04f5f121f323c182403ef6eb0d8aa18 Mon Sep 17 00:00:00 2001From: Eric Wong <normalperson at yhbt.net> Date: Tue, 25 Jan 2011 13:30:21 -0800 Subject: [PATCH] KNOWN_ISSUES: FreeBSD 8 and sendfile can be buggy Reported by Alexey Bondar. --- KNOWN_ISSUES | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES index bc3dac5..95f4e43 100644 --- a/KNOWN_ISSUES +++ b/KNOWN_ISSUES @@ -6,6 +6,10 @@ acceptable solution. Those issues are documented here. * For notes on sandboxing tools such as Bundler or Isolate, see the {Sandbox}[link:Sandbox.html] page. +* nginx with "sendfile on" under FreeBSD 8 is broken when + uploads are buffered to disk. Disabling sendfile is required to + work around this bug which should be fixed in newer versions of FreeBSD. + * Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will segfault if called after forking. This is fixed in trunk (r26936) and should be backported to the next 1.9.1 stable release (after p378). -- Eric Wong