I''m getting exceptions from my server when a bot starts crawling random incorrect un-escaped URLs (up to a few thousand now), specifically with % in the URL. I''ve looked at a bunch of issues in rack here<https://github.com/rack/rack/issues/337> and here <https://github.com/rack/rack/issues/225> and rails here<https://github.com/rails/rails/issues/2622>, and looked at this SO thread<http://stackoverflow.com/questions/15769681/rails-rack-argumenterror-invalid-encoding-for-post-data> but there doesn''t seem to be a definitive solution. Is there a correct solution for GET errors? Do I have to monkeypatch rack? Here''s the stacktrace: /usr/local/lib/ruby/1.9.1/uri/common.rb:898:in `decode_www_form_component'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/utils.rb:41:in `unescape'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/utils.rb:94:in `block (2 levels) in parse_nested_query'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/utils.rb:94:in `map'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/utils.rb:94:in `block in parse_nested_query'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/utils.rb:93:in `each'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/utils.rb:93:in `parse_nested_query'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/request.rb:332:in `parse_query'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/http/request.rb:269:in `parse_query'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/request.rb:186:in `GET'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/http/request.rb:225:in `GET'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/http/parameters.rb:10:in `parameters'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/http/filter_parameters.rb:33:in `filtered_parameters'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_controller/metal/instrumentation.rb:21:in `process_action'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'' [GEM_ROOT]/gems/activerecord-3.2.12/lib/active_record/railties/controller_runtime.rb:18:in `process_action'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/abstract_controller/base.rb:121:in `process'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/abstract_controller/rendering.rb:45:in `process'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_controller/metal.rb:203:in `dispatch'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_controller/metal.rb:246:in `block in action'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/routing/route_set.rb:73:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/routing/route_set.rb:36:in `call'' [GEM_ROOT]/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'' [GEM_ROOT]/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'' [GEM_ROOT]/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/routing/route_set.rb:601:in `call'' [GEM_ROOT]/gems/omniauth-1.1.1/lib/omniauth/strategy.rb:177:in `call!'' [GEM_ROOT]/gems/omniauth-1.1.1/lib/omniauth/strategy.rb:157:in `call'' [GEM_ROOT]/gems/sass-3.2.7/lib/sass/plugin/rack.rb:54:in `call'' [GEM_ROOT]/gems/warden-1.2.1/lib/warden/manager.rb:35:in `block in call'' [GEM_ROOT]/gems/warden-1.2.1/lib/warden/manager.rb:34:in `catch'' [GEM_ROOT]/gems/warden-1.2.1/lib/warden/manager.rb:34:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/conditionalget.rb:25:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/head.rb:14:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/params_parser.rb:21:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/flash.rb:242:in `call'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/cookies.rb:341:in `call'' [GEM_ROOT]/gems/activerecord-3.2.12/lib/active_record/query_cache.rb:64:in `call'' [GEM_ROOT]/gems/activerecord-3.2.12/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'' [GEM_ROOT]/gems/activesupport-3.2.12/lib/active_support/callbacks.rb:405:in `_run__3427364536233887934__call__4357758965732378258__callbacks'' [GEM_ROOT]/gems/activesupport-3.2.12/lib/active_support/callbacks.rb:405:in `__run_callback'' [GEM_ROOT]/gems/activesupport-3.2.12/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'' [GEM_ROOT]/gems/activesupport-3.2.12/lib/active_support/callbacks.rb:81:in `run_callbacks'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/callbacks.rb:27:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'' [GEM_ROOT]/gems/railties-3.2.12/lib/rails/rack/logger.rb:32:in `call_app'' [GEM_ROOT]/gems/railties-3.2.12/lib/rails/rack/logger.rb:16:in `block in call'' [GEM_ROOT]/gems/activesupport-3.2.12/lib/active_support/tagged_logging.rb:22:in `tagged'' [GEM_ROOT]/gems/railties-3.2.12/lib/rails/rack/logger.rb:16:in `call'' [GEM_ROOT]/gems/actionpack-3.2.12/lib/action_dispatch/middleware/request_id.rb:22:in `call'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'' [GEM_ROOT]/gems/activesupport-3.2.12/lib/active_support/cache/strategy/local_cache.rb:72:in `call'' [GEM_ROOT]/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'' [GEM_ROOT]/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'' [GEM_ROOT]/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'' [GEM_ROOT]/gems/rack-cache-1.2/lib/rack/cache/context.rb:172:in `rescue in lookup'' [GEM_ROOT]/gems/rack-cache-1.2/lib/rack/cache/context.rb:168:in `lookup'' [GEM_ROOT]/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'' [GEM_ROOT]/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'' [GEM_ROOT]/gems/railties-3.2.12/lib/rails/engine.rb:479:in `call'' [GEM_ROOT]/gems/railties-3.2.12/lib/rails/application.rb:223:in `call'' [GEM_ROOT]/gems/railties-3.2.12/lib/rails/railtie/configurable.rb:30:in `method_missing'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_request_handler.rb:516:in `accept_and_process_next_request'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/utils.rb:479:in `safe_fork'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server.rb:180:in `start'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/rack/application_spawner.rb:129:in `start'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'' : /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'' /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/helper-scripts/passenger-spawn-server:99:in `<main>'' I saw this thread from over a year ago as well (https://groups.google.com/forum/?fromgroups=#!searchin/rubyonrails-talk/argumenterror$20encoding/rubyonrails-talk/X9xUwJEZ2Z8/9w8rNr9yuJ8J), but wasn''t sure what the correct thing to do was. :) Thanks for any ideas! Waynn -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msg/rubyonrails-talk/-/mDK_nMSo-zYJ. For more options, visit https://groups.google.com/groups/opt_out.
On Wed, May 8, 2013 at 7:20 PM, Waynn Lue <waynnlue-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> I''m getting exceptions from my server when a bot starts crawling random > incorrect un-escaped URLs (up to a few thousand now), specifically with % in > the URL. I''ve looked at a bunch of issues in rack here and here and rails > here, and looked at this SO thread but there doesn''t seem to be a definitive > solution. Is there a correct solution for GET errors? Do I have to > monkeypatch rack?Do you want to process these requests? If they are coming from a rogue bot, you may just want to catch them and drop them. Make sure you also have robots.txt in your public/ folder. As for the underlying handling of parsing such strings by rails, your guess is as good as mine... -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit https://groups.google.com/groups/opt_out.