Mongrel users - Oct 2006 - Mongrel can''t handle some URLs that IE6 sends

IE6 happily sends unsafe* characters unencoded if you''ve typed them
into the URL bar of your IE6 window.  This could happen if you copy &
paste a URL from an email or web page.

Mongrel doesn''t seem to handle these properly.  In 0.3.13.3 it would
print out something like:

   Sun Oct 15 23:05:38 CST 2006: BAD CLIENT (192.168.1.2): Invalid
HTTP format, parsing fails.

0.3.13.5 seems to ignore the request altogether (not even debug logs
say anything).

It seems other web servers like LighTPD handle requests like this
without issue.  Shouldn''t mongrel follow suit?  It isn''t
standard and
IE7 seems to have fixed this, but IE6 is still a large chunk of the
browser market, and I''d like for my users to get something more
meaningful than an IE error page if they''ve accidentally copied
something out of an email incorrectly.

If there won''t be a patch for Mongrel, is there some way I can get
LighTPD/pen to escape the request on Mongrel''s behalf?

* According to rfc2068: unsafe = CTL | SP | <"> | "#" |
"%" | "<" | ">"

On 10/15/06, Eden Li <eden.li at gmail.com> wrote:
> IE6 happily sends unsafe* characters unencoded if you''ve typed
them
> into the URL bar of your IE6 window.  This could happen if you copy &
> paste a URL from an email or web page.
>
> Mongrel doesn''t seem to handle these properly.  In 0.3.13.3 it
would
> print out something like:
>
>    Sun Oct 15 23:05:38 CST 2006: BAD CLIENT (192.168.1.2): Invalid
> HTTP format, parsing fails.
>
> 0.3.13.5 seems to ignore the request altogether (not even debug logs
> say anything).
>
> It seems other web servers like LighTPD handle requests like this
> without issue.  Shouldn''t mongrel follow suit?  It isn''t
standard and
> IE7 seems to have fixed this, but IE6 is still a large chunk of the
> browser market, and I''d like for my users to get something more
> meaningful than an IE error page if they''ve accidentally copied
> something out of an email incorrectly.
>
> If there won''t be a patch for Mongrel, is there some way I can get
> LighTPD/pen to escape the request on Mongrel''s behalf?
>
> * According to rfc2068: unsafe = CTL | SP | <"> | "#"
| "%" | "<" | ">"
You could probably write a mongrel handler to cuddle with the params.
-- 
Rick Olson
http://weblog.techno-weenie.net
http://mephistoblog.com

>> You could probably write a mongrel handler to cuddle with the params.
Aren''t people likely to blame it on Mongrel, even if it is most unfair
?

The usual conclusion is "this server can''t handle that".

My gut feeling is that a web server should handle gracefully ill-behaved
clients or complain explicitely.

(But, since I''m not the one who can write the code, take my remark as a
mere
whisper ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061015/b7b92634/attachment.html

Any examples out there on how I could do this?  The docs on rubyforge
are pretty thin in this area.  I got as far as authoring a mini
gem_plugin, but:

$ mongrel_rails start -S config/mongrel.conf
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with development environment...
** Rails loaded.
** Loading any Rails specific GemPlugins
** Loading config/mongrel.conf external config script
/usr/lib/ruby/gems/1.8/gems/gem_plugin-0.2.2/lib/gem_plugin.rb:163:in
`create'': Plugin category handlers does not exist (RuntimeError)
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/lib/mongrel/configurator.rb:252:in
`plugin''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/lib/mongrel/configurator.rb:337:in
`run_config''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/lib/mongrel/configurator.rb:337:in
`open''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/lib/mongrel/configurator.rb:337:in
`run_config''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/bin/mongrel_rails:117:in
`cloaker_''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/lib/mongrel/configurator.rb:138:in
`listener''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/bin/mongrel_rails:95:in
`cloaker_''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/lib/mongrel/configurator.rb:51:in
`initialize''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/bin/mongrel_rails:82:in
`run''
        from
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/lib/mongrel/command.rb:211:in
`run''
        from /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.5/bin/mongrel_rails:234
        from /usr/bin/mongrel_rails:18

Where mongrel.conf has:
uri "/", :handler => plugin("handlers/myplugin"),
:in_front => true

On 10/16/06, Rick Olson <technoweenie at gmail.com>
wrote:
> You could probably write a mongrel handler to cuddle with the params.

On 10/15/06, philippe lachaise <philippe.lachaise at gmail.com>
wrote:
> >> You could probably write a mongrel handler to cuddle with the
params.
>
> Aren''t people likely to blame it on Mongrel, even if it is most
unfair ?
>
> The usual conclusion is "this server can''t handle that".
>
> My gut feeling is that a web server should handle gracefully ill-behaved
> clients or complain explicitely.
>
> (But, since I''m not the one who can write the code, take my remark
as a mere
> whisper ;-)
That''s up to Zed I guess.  But, a plugin serves many purposes:

- lets you get on with your life since you don''t have to wait for a new
release
- lets others on older versions of mongrel easily upgrade
- provides Zed with working code (hopefully) being used in production.
 i''m sure that says a lot more than a feature request.

As much as I dislike the notion, I too think Mongrel should handle this stuff.

-- 
Rick Olson
http://weblog.techno-weenie.net
http://mephistoblog.com

Hmm.  It looks like a plugin can''t handle this.  The plugin I tested
never got its process method called.

Tracing through lib/mongrel.rb, the HttpParser#execute method throws
an HttpParserError before it has a chance to pass the request off to
the plugin.  Looks like this''ll have to be patched in the Ragel or C
code.

FWIW, the URI parsing libraries of both Perl and Python, which claim
to be RFC compliant, parse these out ''properly'' without
throwing an
error.

Also, from rfc2396:

   The angle-bracket "<" and ">" and double-quote
(") characters are
   excluded because they are often used as the delimiters around URI in
   text documents and protocol fields.

Which makes sense from the standards point of view, but if the request
has already gotten to your server, it shouldn''t barf on it.

On 10/16/06, Rick Olson <technoweenie at gmail.com>
wrote:
> On 10/15/06, philippe lachaise <philippe.lachaise at gmail.com>
wrote:
> > >> You could probably write a mongrel handler to cuddle with the
params.
> >
> > Aren''t people likely to blame it on Mongrel, even if it is
most unfair ?
> >
> > The usual conclusion is "this server can''t handle
that".
> >
> > My gut feeling is that a web server should handle gracefully
ill-behaved
> > clients or complain explicitely.
> >
> > (But, since I''m not the one who can write the code, take my
remark as a mere
> > whisper ;-)
>
> That''s up to Zed I guess.  But, a plugin serves many purposes:
>
> - lets you get on with your life since you don''t have to wait for
a new release
> - lets others on older versions of mongrel easily upgrade
> - provides Zed with working code (hopefully) being used in production.
>  i''m sure that says a lot more than a feature request.
>
> As much as I dislike the notion, I too think Mongrel should handle this
stuff.
>
> --
> Rick Olson
> http://weblog.techno-weenie.net
> http://mephistoblog.com
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

Here''s a patch of http_parser.rl against r358 that allows these three
characters and associated unit tests.

On 10/16/06, Eden Li <eden.li at gmail.com> wrote:
> Hmm.  It looks like a plugin can''t handle this.  The plugin I
tested
> never got its process method called.
>
> Tracing through lib/mongrel.rb, the HttpParser#execute method throws
> an HttpParserError before it has a chance to pass the request off to
> the plugin.  Looks like this''ll have to be patched in the Ragel or
C
> code.
>
> FWIW, the URI parsing libraries of both Perl and Python, which claim
> to be RFC compliant, parse these out ''properly'' without
throwing an
> error.
>
> Also, from rfc2396:
>
>    The angle-bracket "<" and ">" and double-quote
(") characters are
>    excluded because they are often used as the delimiters around URI in
>    text documents and protocol fields.
>
> Which makes sense from the standards point of view, but if the request
> has already gotten to your server, it shouldn''t barf on it.
>
> On 10/16/06, Rick Olson <technoweenie at gmail.com> wrote:
> > On 10/15/06, philippe lachaise <philippe.lachaise at gmail.com>
wrote:
> > > >> You could probably write a mongrel handler to cuddle
with the params.
> > >
> > > Aren''t people likely to blame it on Mongrel, even if it
is most unfair ?
> > >
> > > The usual conclusion is "this server can''t handle
that".
> > >
> > > My gut feeling is that a web server should handle gracefully
ill-behaved
> > > clients or complain explicitely.
> > >
> > > (But, since I''m not the one who can write the code, take
my remark as a mere
> > > whisper ;-)
> >
> > That''s up to Zed I guess.  But, a plugin serves many
purposes:
> >
> > - lets you get on with your life since you don''t have to wait
for a new release
> > - lets others on older versions of mongrel easily upgrade
> > - provides Zed with working code (hopefully) being used in production.
> >  i''m sure that says a lot more than a feature request.
> >
> > As much as I dislike the notion, I too think Mongrel should handle
this stuff.
> >
> > --
> > Rick Olson
> > http://weblog.techno-weenie.net
> > http://mephistoblog.com
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ie6urls.diff
Type: text/x-patch
Size: 5778 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mongrel-users/attachments/20061016/c9dbcff6/attachment.bin

This has come up before, btw.

-Ross.

On 10/15/06, Eden Li <eden.li at gmail.com> wrote:
> Hmm.  It looks like a plugin can''t handle this.  The plugin I
tested
> never got its process method called.
>
> Tracing through lib/mongrel.rb, the HttpParser#execute method throws
> an HttpParserError before it has a chance to pass the request off to
> the plugin.  Looks like this''ll have to be patched in the Ragel or
C
> code.
>
> FWIW, the URI parsing libraries of both Perl and Python, which claim
> to be RFC compliant, parse these out ''properly'' without
throwing an
> error.
>
> Also, from rfc2396:
>
>    The angle-bracket "<" and ">" and double-quote
(") characters are
>    excluded because they are often used as the delimiters around URI in
>    text documents and protocol fields.
>
> Which makes sense from the standards point of view, but if the request
> has already gotten to your server, it shouldn''t barf on it.
>
> On 10/16/06, Rick Olson <technoweenie at gmail.com> wrote:
> > On 10/15/06, philippe lachaise <philippe.lachaise at gmail.com>
wrote:
> > > >> You could probably write a mongrel handler to cuddle
with the params.
> > >
> > > Aren''t people likely to blame it on Mongrel, even if it
is most unfair ?
> > >
> > > The usual conclusion is "this server can''t handle
that".
> > >
> > > My gut feeling is that a web server should handle gracefully
ill-behaved
> > > clients or complain explicitely.
> > >
> > > (But, since I''m not the one who can write the code, take
my remark as a mere
> > > whisper ;-)
> >
> > That''s up to Zed I guess.  But, a plugin serves many
purposes:
> >
> > - lets you get on with your life since you don''t have to wait
for a new release
> > - lets others on older versions of mongrel easily upgrade
> > - provides Zed with working code (hopefully) being used in production.
> >  i''m sure that says a lot more than a feature request.
> >
> > As much as I dislike the notion, I too think Mongrel should handle
this stuff.
> >
> > --
> > Rick Olson
> > http://weblog.techno-weenie.net
> > http://mephistoblog.com
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

Care to point to the thread?

On 10/16/06, Ross Singer <ross.singer at library.gatech.edu>
wrote:
> This has come up before, btw.

On Mon, 16 Oct 2006 07:48:16 +0800
"Eden Li" <eden.li at gmail.com> wrote:
> Here''s a patch of http_parser.rl against r358 that allows these
three
> characters and associated unit tests.
Rock on!  That''s what I''m talking about.  I''ll apply
this soon and post up a pre-release.

Thanks Eden.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Starts here:
http://rubyforge.org/pipermail/mongrel-users/2006-August/001162.html

This patch should basically eliminate it, though, I guess :)

Which is awesome.
-Ross.

On 10/16/06, Eden Li <eden.li at gmail.com> wrote:
> Care to point to the thread?
>
> On 10/16/06, Ross Singer <ross.singer at library.gatech.edu> wrote:
> > This has come up before, btw.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

"Zed A. Shaw" <zedshaw-dd7LMGGEL7NBDgjK7y7TUQ at
public.gmane.org> wrote:
> On Mon, 16 Oct 2006 07:48:16 +0800
> "Eden Li" <eden.li-Re5JQEeQqe8AvxtiuMwx3w at
public.gmane.org> wrote:
> 
> > Here''s a patch of http_parser.rl against r358 that allows
these three
> > characters and associated unit tests.
> 
> 
> Rock on!  That''s what I''m talking about.  I''ll
apply this soon and
> post up a pre-release.
>  
> Thanks Eden.
I was late to the party, but just committed this to trunk:

------------------------------------------------------------------------
r996 | normalperson | 2008-03-27 14:46:28 -0700 (Thu, 27 Mar 2008) | 12 lines
Changed paths:
   M ext/http11/http11_parser.c
   M ext/http11/http11_parser_common.rl
   M test/test_http11.rb

http11_parser: accept ''"'' (double-quote),
''<'', and ''>'' characters in URLs

Some broken web browsers don''t properly escape ", <, and >
characters
in URLs, however these URLs to occasionally legitimate and sometimes
show up.

This patch was submitted by Eden Li here:
  http://rubyforge.org/pipermail/mongrel-users/2006-October/001845.html

This patch was accepted by Zed Shaw here:
  http://rubyforge.org/pipermail/mongrel-users/2006-October/001847.html

------------------------------------------------------------------------

-- 
Eric Wong

Oh wow; good catch.

Evan

On Thu, Mar 27, 2008 at 5:50 PM, Eric Wong <normalperson at yhbt.net>
wrote:
> "Zed A. Shaw" <zedshaw-dd7LMGGEL7NBDgjK7y7TUQ at
public.gmane.org> wrote:
>  > On Mon, 16 Oct 2006 07:48:16 +0800
>  > "Eden Li" <eden.li-Re5JQEeQqe8AvxtiuMwx3w at
public.gmane.org> wrote:
>  >
>  > > Here''s a patch of http_parser.rl against r358 that
allows these three
>  > > characters and associated unit tests.
>  >
>  >
>  > Rock on!  That''s what I''m talking about. 
I''ll apply this soon and
>  > post up a pre-release.
>  >
>  > Thanks Eden.
>
>  I was late to the party, but just committed this to trunk:
>
>  ------------------------------------------------------------------------
>  r996 | normalperson | 2008-03-27 14:46:28 -0700 (Thu, 27 Mar 2008) | 12
lines
>  Changed paths:
>    M ext/http11/http11_parser.c
>    M ext/http11/http11_parser_common.rl
>    M test/test_http11.rb
>
>  http11_parser: accept ''"'' (double-quote),
''<'', and ''>'' characters in URLs
>
>  Some broken web browsers don''t properly escape ", <, and
> characters
>  in URLs, however these URLs to occasionally legitimate and sometimes
>  show up.
>
>  This patch was submitted by Eden Li here:
>   http://rubyforge.org/pipermail/mongrel-users/2006-October/001845.html
>
>  This patch was accepted by Zed Shaw here:
>   http://rubyforge.org/pipermail/mongrel-users/2006-October/001847.html
>
>  ------------------------------------------------------------------------
>
>  --
>  Eric Wong
>  _______________________________________________
>  Mongrel-users mailing list
>  Mongrel-users at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
Evan Weaver
Cloudburst, LLC

Sweet, when can I remove these lines from my nginx.conf ?

--

# nginx does not appear to properly sanitize urls in the form
http://www.redbubble.com/people/photography/art/
</p
# which causes mongrel to slam the door all the way down the cluster,  
causing a minor hickup. Both behaviours are
# wrong, but until one party fixes the bug, this hack protects us

location ~ ''<'' {
    return 400;
}

location ~ ''>'' {
    return 400;
}

location ~ ''"'' {
    return 400;
}

--

Cheers

Dave

On 28/03/2008, at 9:18 AM, Evan Weaver wrote:
> Oh wow; good catch.
>
> Evan
>
> On Thu, Mar 27, 2008 at 5:50 PM, Eric Wong <normalperson at yhbt.net>
> wrote:
>> "Zed A. Shaw" <zedshaw-dd7LMGGEL7NBDgjK7y7TUQ at
public.gmane.org>
>> wrote:
>>> On Mon, 16 Oct 2006 07:48:16 +0800
>>> "Eden Li" <eden.li-Re5JQEeQqe8AvxtiuMwx3w at
public.gmane.org> wrote:
>>>
>>>> Here''s a patch of http_parser.rl against r358 that
allows these
>>>> three
>>>> characters and associated unit tests.
>>>
>>>
>>> Rock on!  That''s what I''m talking about. 
I''ll apply this soon and
>>> post up a pre-release.
>>>
>>> Thanks Eden.
>>
>> I was late to the party, but just committed this to trunk:
>>
>>
------------------------------------------------------------------------
>> r996 | normalperson | 2008-03-27 14:46:28 -0700 (Thu, 27 Mar 2008)  
>> | 12 lines
>> Changed paths:
>>   M ext/http11/http11_parser.c
>>   M ext/http11/http11_parser_common.rl
>>   M test/test_http11.rb
>>
>> http11_parser: accept ''"'' (double-quote),
''<'', and ''>'' characters
>> in URLs
>>
>> Some broken web browsers don''t properly escape ", <,
and > characters
>> in URLs, however these URLs to occasionally legitimate and sometimes
>> show up.
>>
>> This patch was submitted by Eden Li here:
>>  http://rubyforge.org/pipermail/mongrel-users/2006-October/ 
>> 001845.html
>>
>> This patch was accepted by Zed Shaw here:
>>  http://rubyforge.org/pipermail/mongrel-users/2006-October/ 
>> 001847.html
>>
>>
------------------------------------------------------------------------
>>
>> --
>> Eric Wong
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>
>
>
> -- 
> Evan Weaver
> Cloudburst, LLC
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

On Thu, Mar 27, 2008 at 7:24 PM, Dave Cheney <dave at cheney.net>
wrote:
> Sweet, when can I remove these lines from my nginx.conf ?
>
Hold down a bit man, it just got into the code base ;-)

Regards,
-- 
Luis Lavena
Multimedia systems
-
Human beings, who are almost unique in having the ability to learn from
the experience of others, are also remarkable for their apparent
disinclination to do so.
Douglas Adams