Hello all--
I encountered an issue last night that appears to be related to the
new routes implementation. We''re running edge rails for our
production e-commerce app (* ignores snickers from the crowd *), and
had to roll back when last night''s deployment failed. What we were
seeing in the log is:
------------------------------------------------------------------------
------------------------------
Processing Base#index (for 10.0.1.252 at 2006-06-05 23:22:56) [GET]
Session ID: 051255ad8aed403a821bafba0950d079
Parameters: {}
RegexpError (regular expression too big: / *** large regexp fragment
here ***):
/vendor/rails/actionpack/lib/action_controller/routing.rb:515:in
`initialize''
/vendor/rails/actionpack/lib/action_controller/routing.rb:515:in
`regexp_chunk''
/vendor/rails/actionpack/lib/action_controller/routing.rb:196:in
`recognition_extraction''
/vendor/rails/actionpack/lib/action_controller/routing.rb:194:in
`recognition_extraction''
/vendor/rails/actionpack/lib/action_controller/routing.rb:158:in
`write_recognition''
/vendor/rails/actionpack/lib/action_controller/routing.rb:247:in
`recognize''
/vendor/rails/actionpack/lib/action_controller/routing.rb:996:in
`recognize_path''
/vendor/rails/actionpack/lib/action_controller/routing.rb:995:in
`recognize_path''
/vendor/rails/actionpack/lib/action_controller/routing.rb:988:in
`recognize''
*** stack trace continues ***
------------------------------------------------------------------------
------------------------------
I can''t seem to duplicate this anywhere -- running the below code in
the console (taken from routing.rb:514-515 and tweaked for irb) runs
fine.
------------------------------------------------------------------------
------------------------------
possible_names =
ActionController::Routing.possible_controllers.collect { |name|
Regexp.escape name }
regexp = nil
Regexp.new("(?i-:(#{(regexp || Regexp.union
(*possible_names)).source}))")
------------------------------------------------------------------------
------------------------------
A bit of trivia here: our possible_names array contains 125 entries,
and the resulting regular expression source is ~ 3500 characters.
As a separate issue: I did notice in perusing the possible_names
array that there is now a lot of "stuff" which rails will recognize
as an "available" route out of the box (for example: /rails/
actionpack/examples/address_book), so it feels like $LOAD_PATH needs
to be filtered a bit, at least for a production environment. Is this
expanded route scope by design, or should I file a ticket on it?
Back to the main issue, any ideas on reproducing/troubleshooting the
Regexp exception? We''ve configured a parallel deployment of the app
and run it successfully on another port, but I''m not yet warm and
fuzzy that the issue won''t pop up again.
Thanks in advance,
Solomon
Solomon, I''ll definitely take a look at it today and see what I can find. In the meantime, please note that the current routes impl is NOT ready for production. It is known to be inefficient in several areas (particularly route generation) and buggy (as you discovered). We are working hard to eliminate the bugs, and are getting closer, but it is still experimental. - Jamis On Jun 6, 2006, at 10:49 AM, Solomon White wrote:> Hello all-- > > I encountered an issue last night that appears to be related to the > new routes implementation. We''re running edge rails for our > production e-commerce app (* ignores snickers from the crowd *), > and had to roll back when last night''s deployment failed. What we > were seeing in the log is: > > > ---------------------------------------------------------------------- > -------------------------------- > Processing Base#index (for 10.0.1.252 at 2006-06-05 23:22:56) [GET] > Session ID: 051255ad8aed403a821bafba0950d079 > Parameters: {} > > > RegexpError (regular expression too big: / *** large regexp > fragment here ***): > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 515:in `initialize'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 515:in `regexp_chunk'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 196:in `recognition_extraction'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 194:in `recognition_extraction'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 158:in `write_recognition'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 247:in `recognize'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 996:in `recognize_path'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 995:in `recognize_path'' > /vendor/rails/actionpack/lib/action_controller/routing.rb: > 988:in `recognize'' > *** stack trace continues *** > ---------------------------------------------------------------------- > -------------------------------- > > > I can''t seem to duplicate this anywhere -- running the below code > in the console (taken from routing.rb:514-515 and tweaked for irb) > runs fine. > > > ---------------------------------------------------------------------- > -------------------------------- > possible_names = > ActionController::Routing.possible_controllers.collect { |name| > Regexp.escape name } > regexp = nil > Regexp.new("(?i-:(#{(regexp || Regexp.union > (*possible_names)).source}))") > ---------------------------------------------------------------------- > -------------------------------- > > A bit of trivia here: our possible_names array contains 125 > entries, and the resulting regular expression source is ~ 3500 > characters. > > As a separate issue: I did notice in perusing the possible_names > array that there is now a lot of "stuff" which rails will recognize > as an "available" route out of the box (for example: /rails/ > actionpack/examples/address_book), so it feels like $LOAD_PATH > needs to be filtered a bit, at least for a production environment. > Is this expanded route scope by design, or should I file a ticket > on it? > > Back to the main issue, any ideas on reproducing/troubleshooting > the Regexp exception? We''ve configured a parallel deployment of > the app and run it successfully on another port, but I''m not yet > warm and fuzzy that the issue won''t pop up again. > > Thanks in advance, > > Solomon > _______________________________________________ > Rails-core mailing list > Rails-core@lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails-core
Solomon, Can you svn up and see if it works now? I''ve made the possible_controllers code smarter, so it should return a much smaller list of possible controllers. Definitely still room for improvement, but it''s better than it was. - Jamis On Jun 7, 2006, at 7:52 AM, Jamis Buck wrote:> Solomon, > > I''ll definitely take a look at it today and see what I can find. In > the meantime, please note that the current routes impl is NOT ready > for production. It is known to be inefficient in several areas > (particularly route generation) and buggy (as you discovered). We > are working hard to eliminate the bugs, and are getting closer, but > it is still experimental. > > - Jamis > > On Jun 6, 2006, at 10:49 AM, Solomon White wrote: > >> Hello all-- >> >> I encountered an issue last night that appears to be related to >> the new routes implementation. We''re running edge rails for our >> production e-commerce app (* ignores snickers from the crowd *), >> and had to roll back when last night''s deployment failed. What we >> were seeing in the log is: >> >> >> --------------------------------------------------------------------- >> --------------------------------- >> Processing Base#index (for 10.0.1.252 at 2006-06-05 23:22:56) [GET] >> Session ID: 051255ad8aed403a821bafba0950d079 >> Parameters: {} >> >> >> RegexpError (regular expression too big: / *** large regexp >> fragment here ***): >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 515:in `initialize'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 515:in `regexp_chunk'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 196:in `recognition_extraction'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 194:in `recognition_extraction'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 158:in `write_recognition'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 247:in `recognize'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 996:in `recognize_path'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 995:in `recognize_path'' >> /vendor/rails/actionpack/lib/action_controller/routing.rb: >> 988:in `recognize'' >> *** stack trace continues *** >> --------------------------------------------------------------------- >> --------------------------------- >> >> >> I can''t seem to duplicate this anywhere -- running the below code >> in the console (taken from routing.rb:514-515 and tweaked for irb) >> runs fine. >> >> >> --------------------------------------------------------------------- >> --------------------------------- >> possible_names = >> ActionController::Routing.possible_controllers.collect { |name| >> Regexp.escape name } >> regexp = nil >> Regexp.new("(?i-:(#{(regexp || Regexp.union >> (*possible_names)).source}))") >> --------------------------------------------------------------------- >> --------------------------------- >> >> A bit of trivia here: our possible_names array contains 125 >> entries, and the resulting regular expression source is ~ 3500 >> characters. >> >> As a separate issue: I did notice in perusing the possible_names >> array that there is now a lot of "stuff" which rails will >> recognize as an "available" route out of the box (for example: / >> rails/actionpack/examples/address_book), so it feels like >> $LOAD_PATH needs to be filtered a bit, at least for a production >> environment. Is this expanded route scope by design, or should I >> file a ticket on it? >> >> Back to the main issue, any ideas on reproducing/troubleshooting >> the Regexp exception? We''ve configured a parallel deployment of >> the app and run it successfully on another port, but I''m not yet >> warm and fuzzy that the issue won''t pop up again. >> >> Thanks in advance, >> >> Solomon >> _______________________________________________ >> Rails-core mailing list >> Rails-core@lists.rubyonrails.org >> http://lists.rubyonrails.org/mailman/listinfo/rails-core > > _______________________________________________ > Rails-core mailing list > Rails-core@lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails-core
Jamis-- Seems to have worked for us, we''re back on edge. How far from production-ready do you estimate the new routes implementation to be? We''re about to convert a fairly high-traffic site from its current JSP implementation to our rails app, and I need to decide whether to freeze before or after new routes... Thanks for your help, Solomon On Jun 7, 2006, at 10:17 AM, Jamis Buck wrote:> Solomon, > > Can you svn up and see if it works now? I''ve made the > possible_controllers code smarter, so it should return a much > smaller list of possible controllers. Definitely still room for > improvement, but it''s better than it was. > > - Jamis > > On Jun 7, 2006, at 7:52 AM, Jamis Buck wrote: > >> Solomon, >> >> I''ll definitely take a look at it today and see what I can find. >> In the meantime, please note that the current routes impl is NOT >> ready for production. It is known to be inefficient in several >> areas (particularly route generation) and buggy (as you >> discovered). We are working hard to eliminate the bugs, and are >> getting closer, but it is still experimental. >> >> - Jamis >> >> On Jun 6, 2006, at 10:49 AM, Solomon White wrote: >> >>> Hello all-- >>> >>> I encountered an issue last night that appears to be related to >>> the new routes implementation. We''re running edge rails for our >>> production e-commerce app (* ignores snickers from the crowd *), >>> and had to roll back when last night''s deployment failed. What >>> we were seeing in the log is: >>> >>> >>> -------------------------------------------------------------------- >>> ---------------------------------- >>> Processing Base#index (for 10.0.1.252 at 2006-06-05 23:22:56) [GET] >>> Session ID: 051255ad8aed403a821bafba0950d079 >>> Parameters: {} >>> >>> >>> RegexpError (regular expression too big: / *** large regexp >>> fragment here ***): >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 515:in `initialize'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 515:in `regexp_chunk'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 196:in `recognition_extraction'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 194:in `recognition_extraction'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 158:in `write_recognition'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 247:in `recognize'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 996:in `recognize_path'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 995:in `recognize_path'' >>> /vendor/rails/actionpack/lib/action_controller/routing.rb: >>> 988:in `recognize'' >>> *** stack trace continues *** >>> -------------------------------------------------------------------- >>> ---------------------------------- >>> >>> >>> I can''t seem to duplicate this anywhere -- running the below code >>> in the console (taken from routing.rb:514-515 and tweaked for >>> irb) runs fine. >>> >>> >>> -------------------------------------------------------------------- >>> ---------------------------------- >>> possible_names = >>> ActionController::Routing.possible_controllers.collect { |name| >>> Regexp.escape name } >>> regexp = nil >>> Regexp.new("(?i-:(#{(regexp || Regexp.union >>> (*possible_names)).source}))") >>> -------------------------------------------------------------------- >>> ---------------------------------- >>> >>> A bit of trivia here: our possible_names array contains 125 >>> entries, and the resulting regular expression source is ~ 3500 >>> characters. >>> >>> As a separate issue: I did notice in perusing the possible_names >>> array that there is now a lot of "stuff" which rails will >>> recognize as an "available" route out of the box (for example: / >>> rails/actionpack/examples/address_book), so it feels like >>> $LOAD_PATH needs to be filtered a bit, at least for a production >>> environment. Is this expanded route scope by design, or should I >>> file a ticket on it? >>> >>> Back to the main issue, any ideas on reproducing/troubleshooting >>> the Regexp exception? We''ve configured a parallel deployment of >>> the app and run it successfully on another port, but I''m not yet >>> warm and fuzzy that the issue won''t pop up again. >>> >>> Thanks in advance, >>> >>> Solomon >>> _______________________________________________ >>> Rails-core mailing list >>> Rails-core@lists.rubyonrails.org >>> http://lists.rubyonrails.org/mailman/listinfo/rails-core >> >> _______________________________________________ >> Rails-core mailing list >> Rails-core@lists.rubyonrails.org >> http://lists.rubyonrails.org/mailman/listinfo/rails-core > > _______________________________________________ > Rails-core mailing list > Rails-core@lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails-core
On Jun 18, 2006, at 4:53 PM, Solomon White wrote:> Jamis-- > > Seems to have worked for us, we''re back on edge. How far from > production-ready do you estimate the new routes implementation to > be? We''re about to convert a fairly high-traffic site from its > current JSP implementation to our rails app, and I need to decide > whether to freeze before or after new routes...Well, as far as I know, the only outstanding routing-related ticket is http://dev.rubyonrails.org/ticket/5320. I haven''t had a chance to look into it yet. So, if the rails trac is any indication, it sounds like not many people are having issues with the new routes, and we may be close to production-ready, though I''d like to hear from a lot more people that are actually using it before I make that declaration with any confidence. - Jamis
On 6/18/06, Jamis Buck <jamis@37signals.com> wrote:> On Jun 18, 2006, at 4:53 PM, Solomon White wrote: > > > Jamis-- > > > > Seems to have worked for us, we''re back on edge. How far from > > production-ready do you estimate the new routes implementation to > > be? We''re about to convert a fairly high-traffic site from its > > current JSP implementation to our rails app, and I need to decide > > whether to freeze before or after new routes... > > Well, as far as I know, the only outstanding routing-related ticket > is http://dev.rubyonrails.org/ticket/5320. I haven''t had a chance to > look into it yet. So, if the rails trac is any indication, it sounds > like not many people are having issues with the new routes, and we > may be close to production-ready, though I''d like to hear from a lot > more people that are actually using it before I make that declaration > with any confidence. >I''ve run into some nasty and repeatable memory leakage in recent Rails revisions. Running on 4450-something, I managed to make the Linux kernel''s OOM-killer shut down the Rails processes. Rolling back to 4358 fixed it, with no other changes. I''m trying to find time to narrow down the leak to a specific revision, but from the changelog, the only major change seems to be the new routing code. Anyone feel like looping ''ab'' against one of their apps, in production mode? Here''s the command I used: while true; do /usr/sbin/ab -n 1000 -c 30 http://example.com/ 2>/dev/null | grep quest.*mean ; echo -- ; done I''ve got two apps that don''t go above 21MB of usage during that process when running on 4358, and die by running out of memory on the latest edge. --Wilson.
Wilson-- Hmmm. Can''t reproduce that behavior here. I''ve got four dispatch fcgi processes running and they all hovered right around 50MB of memory usage (which is also where they are at under normal load) through 10 iterations of your ab command. We''re running v4459 of rails. How long was it taking for you to see the leak? On Jun 19, 2006, at 8:38 AM, Wilson Bilkovich wrote:> {snip}> Anyone feel like looping ''ab'' against one of their apps, in > production mode? > Here''s the command I used: > > while true; do /usr/sbin/ab -n 1000 -c 30 http://example.com/ > 2>/dev/null | grep quest.*mean ; echo -- ; done
On 6/20/06, Solomon White <solo@gatelys.com> wrote:> Wilson-- > > Hmmm. Can''t reproduce that behavior here. I''ve got four dispatch > fcgi processes running and they all hovered right around 50MB of > memory usage (which is also where they are at under normal load) > through 10 iterations of your ab command. We''re running v4459 of > rails. How long was it taking for you to see the leak? >The extra memory usage was visible in ''top'' within 15 seconds or so. Both of the apps in question are running on SQLite3. Perhaps that''s involved somehow. Good to know that it isn''t happening to everyone, though. I''ll set aside some time to try tracking it down. Thanks, --Wilson.