Hi,
I''m trying to use mongrel 0.3.13 with Rail 1.0
Believe it or not, on OS/X it works well for weeks at a time. On
linux I''m getting Input/output errors after anything from a few hours
to days, never more than a week. Once they start they don''t stop.
The only pattern I''ve seen is that it always (seems) to happen while
rendering an rhtml file. I''ve never seen it happen with a static
file, for instance.
Until yesterday, once the errors started they happened on every page
rendered. Yesterday, some pages would be rendered okay, others fail.
I mean that some rhtml files would work and other wouldn''t -- the
rhtml files all follow a similar pattern so there''s nothing
particularly special about any of them, working or failing.
Has anyone seen this before? Any suggestions? Is there something
useful to check in Rails or Mongrel?
I''ve appended a recent stack trace. Don''t read anything into
the
rhtml where it failed, it happens with the simplest <%= @var -%> you
can imagine. This trace might not be the best example, but it is the
one I''ve got.
Cheers,
Bob
----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
Processing StoryController#index (for 216.138.231.102 at 2006-06-12
07:34:38) [GET]
Parameters: {"story"=>"1144782860_45",
"action"=>"index",
"id"=>"cdmb-migrated",
"controller"=>"raconteur/story",
"pub"=>"1145371264_1"}
raconteur_user: super
Rendering within layouts/raconteur
Rendering raconteur/story/index
ActionView::TemplateError (Input/output error) on line #15 of app/
views/raconteur/story/index.rhtml:
12: <%
13: if @the_story.the_description and 0 <
@the_story.the_description.size then
14: %>
15: <div class=''pageDescription''><%=
in_place_editor_field :the_story, :the_description, {}, {:url =>
url_for(:id => @installation_id, :pub => @publication_id, :story =>
@story_id, :action => ''set_the_story_description''), :rows
=>
15, :save_text => _("save"), :cancel_text => _
("cancel"), :saving_text => _("Saving...")}
%></div>
16: <%
17: else
18: %>
/home/raconteur/gems/gems/actionpack-1.11.2/lib/action_view/
helpers/java_script_macros_helper.rb:62:in `write''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/action_view/
helpers/java_script_macros_helper.rb:62:in `in_place_editor_field''
#{RAILS_ROOT}/app/views/raconteur/story/index.rhtml:15
/home/raconteur/gems/gems/actionpack-1.11.2/lib/action_view/
base.rb:268:in `compile_and_render_template''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/action_view/
base.rb:244:in `render_template''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/action_view/
base.rb:205:in `render_file''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/base.rb:655:in `render_file''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/base.rb:595:in `render_with_no_layout''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/layout.rb:220:in `render_without_benchmark''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/benchmarking.rb:53:in `render''
/home/raconteur/lib/ruby/1.8/benchmark.rb:293:in `measure''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/benchmarking.rb:53:in `render''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/base.rb:854:in `perform_action_without_filters''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/filters.rb:332:in `perform_action_without_benchmark''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/benchmarking.rb:69:in `perform_action_without_rescue''
/home/raconteur/lib/ruby/1.8/benchmark.rb:293:in `measure''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/benchmarking.rb:69:in `perform_action_without_rescue''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/rescue.rb:82:in `perform_action''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/base.rb:369:in
`process_without_session_management_support''
/home/raconteur/gems/gems/actionpack-1.11.2/lib/
action_controller/session_management.rb:116:in `process''
/home/raconteur/gems/gems/rails-1.0.0/lib/dispatcher.rb:38:in
`dispatch''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel/rails.rb:
85:in `process''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel.rb:532:in
`process_client''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel.rb:531:in
`process_client''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel.rb:600:in
`run''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel.rb:599:in
`run''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel.rb:588:in
`run''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel.rb:918:in
`run''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel.rb:917:in
`run''
/home/raconteur/gems/gems/mongrel-0.3.13/bin/mongrel_rails:
122:in `run''
/home/raconteur/gems/gems/mongrel-0.3.13/lib/mongrel/command.rb:
208:in `run''
/home/raconteur/gems/gems/mongrel-0.3.13/bin/mongrel_rails:230
/home/raconteur/gems/bin/mongrel_rails:18
On Mon, 2006-06-12 at 08:37 -0400, Bob Hutchison wrote:> Hi, > > I''m trying to use mongrel 0.3.13 with Rail 1.0 > > Believe it or not, on OS/X it works well for weeks at a time. On > linux I''m getting Input/output errors after anything from a few hours > to days, never more than a week. Once they start they don''t stop. > > The only pattern I''ve seen is that it always (seems) to happen while > rendering an rhtml file. I''ve never seen it happen with a static > file, for instance. > > Until yesterday, once the errors started they happened on every page > rendered. Yesterday, some pages would be rendered okay, others fail. > I mean that some rhtml files would work and other wouldn''t -- the > rhtml files all follow a similar pattern so there''s nothing > particularly special about any of them, working or failing. > > Has anyone seen this before? Any suggestions? Is there something > useful to check in Rails or Mongrel?Are you seeing these errors usually after midnight and do you have log rotation enabled for any of the log files? I''ve found log rotation causes many problems. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/
Hi Zed, On Jun 12, 2006, at 9:03 AM, Zed Shaw wrote:> On Mon, 2006-06-12 at 08:37 -0400, Bob Hutchison wrote: >> Hi, >> >> I''m trying to use mongrel 0.3.13 with Rail 1.0 >> >> Believe it or not, on OS/X it works well for weeks at a time. On >> linux I''m getting Input/output errors after anything from a few hours >> to days, never more than a week. Once they start they don''t stop. >> >> The only pattern I''ve seen is that it always (seems) to happen while >> rendering an rhtml file. I''ve never seen it happen with a static >> file, for instance. >> >> Until yesterday, once the errors started they happened on every page >> rendered. Yesterday, some pages would be rendered okay, others fail. >> I mean that some rhtml files would work and other wouldn''t -- the >> rhtml files all follow a similar pattern so there''s nothing >> particularly special about any of them, working or failing. >> >> Has anyone seen this before? Any suggestions? Is there something >> useful to check in Rails or Mongrel? > > Are you seeing these errors usually after midnight and do you have log > rotation enabled for any of the log files? I''ve found log rotation > causes many problems.I don''t have log rotation enabled. And there doesn''t appear to be any time of day involvement either. Cheers, Bob> > > > -- > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
On Mon, 2006-06-12 at 09:14 -0400, Bob Hutchison wrote:> >> Has anyone seen this before? Any suggestions? Is there something > >> useful to check in Rails or Mongrel? > > > > Are you seeing these errors usually after midnight and do you have log > > rotation enabled for any of the log files? I''ve found log rotation > > causes many problems. > > I don''t have log rotation enabled. And there doesn''t appear to be any > time of day involvement either.You know there''s no "write" in that help file anywhere, and even if there were it''s being written to a StringIO internally so you wouldn''t get a write error. The only possible places to get write errors are: 1) MySQL -- See the e-mail today about mysql timeouts. 2) $stderr,$stdout -- rhtml *LOVES* to write crap to these, but Mongrel catches these and redirects to StringIO as well. 3) Files -- Make sure you''re not dumping to a file that''s then removed on accident. Although this is weird. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/
On Jun 12, 2006, at 9:31 AM, Zed Shaw wrote:> On Mon, 2006-06-12 at 09:14 -0400, Bob Hutchison wrote: > >>>> Has anyone seen this before? Any suggestions? Is there something >>>> useful to check in Rails or Mongrel? >>> >>> Are you seeing these errors usually after midnight and do you >>> have log >>> rotation enabled for any of the log files? I''ve found log rotation >>> causes many problems. >> >> I don''t have log rotation enabled. And there doesn''t appear to be any >> time of day involvement either. > > You know there''s no "write" in that help file anywhere,Yeah I know :-) I don''t know what to think of that. Maybe something is trapping and then re-throwing the exception? In other cases, when I''ve checked there had been a write.> and even if > there were it''s being written to a StringIO internally so you wouldn''t > get a write error.StringIO throws IOErrors for a number of reasons (I just looked at the C code). So, your comments below are looking very interesting.> The only possible places to get write errors are: > > 1) MySQL -- See the e-mail today about mysql timeouts.I don''t use a conventional database (and if any I/O were to happen with the DB I use I would know about it since it log the fact if legal, or throw its own exception otherwise). The log it uses is the standard log (where I got the stack trace from so that should be okay).> 2) $stderr,$stdout -- rhtml *LOVES* to write crap to these, but > Mongrel > catches these and redirects to StringIO as well.This is interesting. stderr and stdout have probably been detached. I''ll look into this for sure.> 3) Files -- Make sure you''re not dumping to a file that''s then > removed > on accident. Although this is weird.I don''t think this is happening. Everything is pretty uniform and so if even one page is working this probably isn''t happening. Thanks, Bob> > > -- > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
Hi,
More on this problem. Maybe a even a clue??
I restarted the server just before I sent the last message (Jun 12,
2006, at 9:48 AM). The difference was that I redirected stderr to a
file (you had mentioned stdout and stderr issues). Things were
running fine until between 17:04 and 17:12 EDT yesterday.
The error this time was really different, so maybe not related at
all. It was getting name errors like:
NameError (undefined method `options_of'' for class
`Module'')
but for all kinds of different methods and classes.
Looking in the stderr file it looks as though there was a HUP signal
caught. But no timestamp. I''m guessing it happened sometime between
17:04 and 17:12.
Now, the specific errors I''ve been getting are different than those
I''ve been complaining about. Is it possible that previously the HUP
signal handling actually worked (mostly) but left the IO system in
trouble?
Thanks,
Bob
On Jun 12, 2006, at 9:48 AM, Bob Hutchison wrote:
>
> On Jun 12, 2006, at 9:31 AM, Zed Shaw wrote:
>
>> On Mon, 2006-06-12 at 09:14 -0400, Bob Hutchison wrote:
>>
>>>>> Has anyone seen this before? Any suggestions? Is there
something
>>>>> useful to check in Rails or Mongrel?
>>>>
>>>> Are you seeing these errors usually after midnight and do you
>>>> have log
>>>> rotation enabled for any of the log files? I''ve found
log rotation
>>>> causes many problems.
>>>
>>> I don''t have log rotation enabled. And there
doesn''t appear to be
>>> any
>>> time of day involvement either.
>>
>> You know there''s no "write" in that help file
anywhere,
>
> Yeah I know :-) I don''t know what to think of that. Maybe
something
> is trapping and then re-throwing the exception?
>
> In other cases, when I''ve checked there had been a write.
>
>> and even if
>> there were it''s being written to a StringIO internally so you
>> wouldn''t
>> get a write error.
>
> StringIO throws IOErrors for a number of reasons (I just looked at
> the C code). So, your comments below are looking very interesting.
>
>> The only possible places to get write errors are:
>>
>> 1) MySQL -- See the e-mail today about mysql timeouts.
>
> I don''t use a conventional database (and if any I/O were to happen
> with the DB I use I would know about it since it log the fact if
> legal, or throw its own exception otherwise). The log it uses is
> the standard log (where I got the stack trace from so that should
> be okay).
>
>> 2) $stderr,$stdout -- rhtml *LOVES* to write crap to these, but
>> Mongrel
>> catches these and redirects to StringIO as well.
>
> This is interesting. stderr and stdout have probably been detached.
> I''ll look into this for sure.
>
>> 3) Files -- Make sure you''re not dumping to a file
that''s then
>> removed
>> on accident. Although this is weird.
>
> I don''t think this is happening. Everything is pretty uniform and
> so if even one page is working this probably isn''t happening.
>
>
> Thanks,
> Bob
>
>>
>>
>> --
>> Zed A. Shaw
>> http://www.zedshaw.com/
>> http://mongrel.rubyforge.org/
>>
>>
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>
> ----
> Bob Hutchison -- blogs at <http://www.recursive.ca/
> hutch/>
> Recursive Design Inc. -- <http://www.recursive.ca/>
> Raconteur -- <http://www.raconteur.info/>
> xampl for Ruby -- <http://rubyforge.org/projects/
> xampl/>
>
>
>
----
Bob Hutchison -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc. -- <http://www.recursive.ca/>
Raconteur -- <http://www.raconteur.info/>
xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
On Thu, 2006-06-15 at 07:18 -0400, Bob Hutchison wrote:> Hi, > > More on this problem. Maybe a even a clue?? > > I restarted the server just before I sent the last message (Jun 12, > 2006, at 9:48 AM). The difference was that I redirected stderr to a > file (you had mentioned stdout and stderr issues). Things were > running fine until between 17:04 and 17:12 EDT yesterday. > > The error this time was really different, so maybe not related at > all. It was getting name errors like: > > NameError (undefined method `options_of'' for class `Module'') > > but for all kinds of different methods and classes. > > Looking in the stderr file it looks as though there was a HUP signal > caught. But no timestamp. I''m guessing it happened sometime between > 17:04 and 17:12. > > Now, the specific errors I''ve been getting are different than those > I''ve been complaining about. Is it possible that previously the HUP > signal handling actually worked (mostly) but left the IO system in > trouble?So you''re seeing a HUP signal for some reason, so that''s something you have to track down. But, HUP does a "reload" of Rails, which tends to not work so well. If you get a HUP and the server stops working after that, then the only thing you can do is not send the process HUP signals. Is someone doing a capistrano deployment that is doing a reload rather than a restart? -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/
On Jun 15, 2006, at 11:44 AM, Zed Shaw wrote:> On Thu, 2006-06-15 at 07:18 -0400, Bob Hutchison wrote: >> Hi, >> >> More on this problem. Maybe a even a clue?? >> >> I restarted the server just before I sent the last message (Jun 12, >> 2006, at 9:48 AM). The difference was that I redirected stderr to a >> file (you had mentioned stdout and stderr issues). Things were >> running fine until between 17:04 and 17:12 EDT yesterday. >> >> The error this time was really different, so maybe not related at >> all. It was getting name errors like: >> >> NameError (undefined method `options_of'' for class `Module'') >> >> but for all kinds of different methods and classes. >> >> Looking in the stderr file it looks as though there was a HUP signal >> caught. But no timestamp. I''m guessing it happened sometime between >> 17:04 and 17:12. >> >> Now, the specific errors I''ve been getting are different than those >> I''ve been complaining about. Is it possible that previously the HUP >> signal handling actually worked (mostly) but left the IO system in >> trouble? > > So you''re seeing a HUP signal for some reason, so that''s something you > have to track down. But, HUP does a "reload" of Rails, which tends to > not work so well. If you get a HUP and the server stops working after > that, then the only thing you can do is not send the process HUP > signals.I''m not sending a HUP signal, and I don''t know who did. Looking into it...> > Is someone doing a capistrano deployment that is doing a reload rather > than a restart? >No. When you force a reload (or restart) does Mongrel wait until all current requests are done, or does it do it right away? Cheers, Bob> > -- > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
On Jun 15, 2006, at 5:15 PM, Bob Hutchison wrote:>> So you''re seeing a HUP signal for some reason, so that''s something >> you >> have to track down. But, HUP does a "reload" of Rails, which >> tends to >> not work so well. If you get a HUP and the server stops working >> after >> that, then the only thing you can do is not send the process HUP >> signals. > > I''m not sending a HUP signal, and I don''t know who did. Looking > into it...I''ve installed on a VPS and the problem appears to have gone away. Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>