Hi, I am having a bit of a problem with my app that is using mongrel. For a particular vendor we use, they are generating links that look like: http://umlaut.library.gatech.edu/resolve?genre=article&issn=00224898&title=Journal+of+Terramechanics&volume=43&issue=4&date=20061001&atitle=Requirements+and+system+design+for+a+robot+performing+selective+cleaning+in+young+forest+stands.&spage=505&sid=EBSCO:aph&pid=Vestlund%2c+Karin%3bHellstr%c3%b6m%2c+Thomas>>2183092720061001aph(pardon the awfulness of OpenURL query strings) Which for IE gives the user: Thu Aug 24 18:12:48 EDT 2006: BAD CLIENT (127.0.0.1): Invalid HTTP format, parsing fails. I am not entirely sure, but I think too many of these eventually bring mongrel down completely (mongrel keeps stopping, it may or may not be related). Firefox and its ilk properly escape these query strings, so those users aren''t affected by it (except, of course, when mongrel goes down completely). Removing the ">>" eliminates the problem in IE. I''ve placed a call to the vendor in question to please stop doing this, but I don''t know if I''ll get any response, and, besides, it''s quite possible I''ll see more invalid characters from other sources in the future. Webrick also fails in exactly the same way on the same queries. FastCGI doesn''t (as far as I can tell), but I also haven''t had any luck getting FastCGI working with this app. I''m using mongrel 0.3.13.3 in production and 0.3.13.4 in development (the link above). Thanks for any help you can offer on this, -Ross. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060824/6d316ecb/attachment.html
Well, the urgency of this has waned a bit, since I successfully got lighty/fcgi working last night. Still, I''ve been pretty fond on mongrel, so far, so I''d like to migrate back to it eventually. Thanks, -Ross. On 8/24/06, Ross Singer <ross.singer at library.gatech.edu> wrote:> > Hi, > > I am having a bit of a problem with my app that is using mongrel. > > For a particular vendor we use, they are generating links that look like: > http://umlaut.library.gatech.edu/resolve?genre=article&issn=00224898&title=Journal+of+Terramechanics&volume=43&issue=4&date=20061001&atitle=Requirements+and+system+design+for+a+robot+performing+selective+cleaning+in+young+forest+stands.&spage=505&sid=EBSCO:aph&pid=Vestlund%2c+Karin%3bHellstr%c3%b6m%2c+Thomas > >>2183092720061001aph > > (pardon the awfulness of OpenURL query strings) > > Which for IE gives the user: > Thu Aug 24 18:12:48 EDT 2006: BAD CLIENT (127.0.0.1): Invalid HTTP format, > parsing fails. > > I am not entirely sure, but I think too many of these eventually bring > mongrel down completely (mongrel keeps stopping, it may or may not be > related). > > Firefox and its ilk properly escape these query strings, so those users > aren''t affected by it (except, of course, when mongrel goes down > completely). > > Removing the ">>" eliminates the problem in IE. > > I''ve placed a call to the vendor in question to please stop doing this, > but I don''t know if I''ll get any response, and, besides, it''s quite possible > I''ll see more invalid characters from other sources in the future. > > Webrick also fails in exactly the same way on the same queries. FastCGI > doesn''t (as far as I can tell), but I also haven''t had any luck getting > FastCGI working with this app. > > I''m using mongrel 0.3.13.3 in production and 0.3.13.4 in development (the > link above). > > Thanks for any help you can offer on this, > -Ross. >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060825/6665e5fd/attachment.html
You can''t really blame mongrel for a bug in someone elses code. Zed is very clear on the website that the parser is strict. Properly urlencoding the query string is basic web programming 101 stuff. If it was me I would be nice but make it clear to your vendor that this is really just not acceptable and you expect to have it fixed asap. Chris
While I agree with this fundamentally, pragmatically this is an unrealistic expectation. I get this vendor to fix it, another vendor pops up with another dumb unescaped character. You can type these characters into the query string on IE or Safari and get the same result, it wouldn''t have to be vendor provided. I am not ''blaming'' anybody. I''m just pointing out that it''s a pretty chaotic world and, as a result, it''s fairly important that (within reason) a webserver can handle the things that are thrown at it. My webserver is the one thing that /I/ can control. -Ross. On 8/25/06, snacktime <snacktime at gmail.com> wrote:> > You can''t really blame mongrel for a bug in someone elses code. Zed > is very clear on the website that the parser is strict. Properly > urlencoding the query string is basic web programming 101 stuff. If > it was me I would be nice but make it clear to your vendor that this > is really just not acceptable and you expect to have it fixed asap. > > Chris > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060825/dd45ad81/attachment.html
On 8/25/06, Ross Singer <ross.singer at library.gatech.edu> wrote:> While I agree with this fundamentally, pragmatically this is an unrealistic > expectation. I get this vendor to fix it, another vendor pops up with > another dumb unescaped character.For you maybe. I am actually using the parser in another project specifically because I like how strict it is. In the end it''s up to Zed though.
On Fri, 2006-08-25 at 07:19 -0400, Ross Singer wrote:> Well, the urgency of this has waned a bit, since I successfully got > lighty/fcgi working last night. > > Still, I''ve been pretty fond on mongrel, so far, so I''d like to > migrate back to it eventually.Hey Ross, I missed this. Here''s what you do to let me see if the client really is wrong: 1) get the latest 0.3.13.4 pre-release: gem install mongrel --source=http://mongrel.rubyforge.org/releases/ 2) Start your mongrel application 3) Before you make your request, hit with a USR1 signal: killall -USR1 mongrel_rails (don''t do killall if you''ve got other ones running you want to leave alone). 4) Hit your web page like normal, and now Mongrel will dump the full data it received and the params it parsed before it complained. Send me the logs and I''ll take a look. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Fri, 2006-08-25 at 15:49 -0400, Ross Singer wrote:> While I agree with this fundamentally, pragmatically this is an > unrealistic expectation. I get this vendor to fix it, another vendor > pops up with another dumb unescaped character. >It''s most likely not a problem with unescaped characters but something else (I won''t know until you send me the logs).> You can type these characters into the query string on IE or Safari > and get the same result, it wouldn''t have to be vendor provided.Well, this is the problem: almost all security attacks against web servers take advantage of how loose they interpret the protocol in order to handle user errors that really the browser should be dealing with. If you think about it, you have two situations: 1) Dumb user typing into a browser -- browser fixes it. 2) Trained programmer writing an API -- follow the RFC. So the trade-off of being more strict is that a few folks who don''t do either of the above have to go change their interactions.> I am not ''blaming'' anybody. I''m just pointing out that it''s a pretty > chaotic world and, as a result, it''s fairly important that (within > reason) a webserver can handle the things that are thrown at it. >Yep, that''s understandable, but it''s also why so many other web servers get hacked so often. The HTTP RFC is very complex, and the best way I''ve found to deal with it is to use a strict parser. That being said, the parser has been wrong in the past so shoot me the logs and I''ll look at how to update it.> My webserver is the one thing that /I/ can control.This also bring up another point, not every solution is right for every job. Mongrel''s the hot thing but I''m the first to suggest another solution if Mongrel doesn''t work out, and have many times told people to not adopt Mongrel if they''ve got something that works. If fcgi works then stick with it. Another option you might like is litespeed: http://litespeedtech.com/community/wiki/doku.php?id=litespeed_wiki:ruby_rails It''s fast, commercial, has support, pretty easy to setup and they do fast work. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Zed,
Thanks for getting back to me on this.
Here is the log from the offending request:
---
Fri Aug 25 20:01:09 EDT 2006: BAD CLIENT (127.0.0.1): Invalid HTTP
format, parsing fails.
Fri Aug 25 20:01:09 EDT 2006: REQUEST DATA: "GET
/resolve?genre=article&issn=00224898&title=Journal+of+Terramechanics&volume=43&issue=4&date=20061001&atitle=Requirements+and+system+design+for+a+robot+performing+selective+cleaning+in+young+forest+stands.&spage=505&sid=EBSCO:aph&pid=Vestlund%2c+Karin%3bHellstr%C3%B6m%2c+Thomas>>2183092720061001aph
HTTP/1.1\r\nHost: umlaut.library.gatech.edu\r\nUser-Agent: Mozilla/5.0
(Macintosh; U; PPC Mac OS X; en) AppleWebKit/312.8 (KHTML, like Gecko)
Safari/312.6\r\nAccept: */*\r\nAccept-Encoding: gzip,
deflate\r\nAccept-Language: en\r\nCookie:
_session_id=8999f072c7fcf77d13e741c9ee8faccc\r\nMax-Forwards:
10\r\nX-Forwarded-For: 24.98.252.118\r\nX-Forwarded-Host:
umlaut.library.gatech.edu\r\nX-Forwarded-Server:
umlaut.library.gatech.edu\r\n\r\n"
---
PARAMS: {"REQUEST_PATH"=>"/resolve",
"REQUEST_METHOD"=>"GET"}
---
Is this what you were looking for?
IE and Safari (and presumably other KHTML based browsers) don''t escape
the ">>". Gecko and Opera do.
Again, here''s the URL:
http://umlaut.library.gatech.edu/resolve?genre=article&issn=00224898&title=Journal+of+Terramechanics&volume=43&issue=4&date=20061001&atitle=Requirements+and+system+design+for+a+robot+performing+selective+cleaning+in+young+forest+stands.&spage=505&sid=EBSCO:aph&pid=Vestlund%2c+Karin%3bHellstr?m%2c+Thomas>>2183092720061001aph
The ">>" is the critical part.
On 8/25/06, Zed Shaw <zedshaw at zedshaw.com>
wrote:> On Fri, 2006-08-25 at 15:49 -0400, Ross Singer wrote:
> > While I agree with this fundamentally, pragmatically this is an
> > unrealistic expectation. I get this vendor to fix it, another vendor
> > pops up with another dumb unescaped character.
> >
> It''s most likely not a problem with unescaped characters but
something
> else (I won''t know until you send me the logs).
>
> > You can type these characters into the query string on IE or Safari
> > and get the same result, it wouldn''t have to be vendor
provided.
>
> Well, this is the problem: almost all security attacks against web
> servers take advantage of how loose they interpret the protocol in order
> to handle user errors that really the browser should be dealing with.
>
I completely understand this.
> If you think about it, you have two situations:
>
> 1) Dumb user typing into a browser -- browser fixes it.
Except it''s not.
> 2) Trained programmer writing an API -- follow the RFC.
Agreed. But, again, YMMV.
>
> So the trade-off of being more strict is that a few folks who
don''t do
> either of the above have to go change their interactions.
>
Right. I guess I''m just pointing out "real world behavior"
that broke
mongrel. But, yes, you''re right, it''s sort of
''encouraging'' bad
behavior to adjust mongrel to it.
> > I am not ''blaming'' anybody. I''m just
pointing out that it''s a pretty
> > chaotic world and, as a result, it''s fairly important that
(within
> > reason) a webserver can handle the things that are thrown at it.
> >
> Yep, that''s understandable, but it''s also why so many
other web servers
> get hacked so often. The HTTP RFC is very complex, and the best way
> I''ve found to deal with it is to use a strict parser.
>
Sure. Understandable.
> That being said, the parser has been wrong in the past so shoot me the
> logs and I''ll look at how to update it.
>
> > My webserver is the one thing that /I/ can control.
>
> This also bring up another point, not every solution is right for every
> job. Mongrel''s the hot thing but I''m the first to
suggest another
> solution if Mongrel doesn''t work out, and have many times told
people to
> not adopt Mongrel if they''ve got something that works.
>
Well, I liked mongrel for the simple fact that it''s uber-easy to
deploy (and, until last night, I couldn''t get fcgi to work, period).
If I''m helping another library install my app, the ability to say,
"Type gem install mongrel; gem install mongrel_cluster" and
they''re
basically running is a ton better than the alternatives.
I mean, yes, fcgi is working (and, like I said, this is new), but
''options'' are nice. Especially if a library is stuck Solaris
or AIX
machine that they don''t have much control on.
But, yeah, I see both sides of this.
Thanks,
-Ross.
> If fcgi works then stick with it. Another option you might like is
> litespeed:
>
>
http://litespeedtech.com/community/wiki/doku.php?id=litespeed_wiki:ruby_rails
>
> It''s fast, commercial, has support, pretty easy to setup and they
do
> fast work.
>
>
> --
> Zed A. Shaw
> http://www.zedshaw.com/
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
On Fri, 2006-08-25 at 20:08 -0400, Ross Singer wrote:> Zed, > > Thanks for getting back to me on this. > > Here is the log from the offending request: ><snip>> Is this what you were looking for?Yep I''ll look at that. But right from the RFC you have this: unsafe = (CTL | " " | "\"" | "#" | "%" | "<" | ">"); Even they label that as "unsafe" so I''m going to be hard pressed to allow this in. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On 8/25/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> > On Fri, 2006-08-25 at 20:08 -0400, Ross Singer wrote: > > Zed, > > > > Thanks for getting back to me on this. > > > > Here is the log from the offending request: > > > <snip> > > Is this what you were looking for? > > Yep I''ll look at that. But right from the RFC you have this: > > unsafe = (CTL | " " | "\"" | "#" | "%" | "<" | ">"); > > Even they label that as "unsafe" so I''m going to be hard pressed to > allow this in.Again, I wouldn''t say I''m opposed to this. :) All I''m sayin'' "Real World Scenario". Also "85% browser market share". I can probably talk EBSCO into fixing their bad URLs, but I don''t have any control over Internet Explorer or Microsoft. I''ve got no influence there. I''m not trying to be passive-aggressive at all about this (I really want to stress that), I''m just stating that this is something that''s bound to come up again. Because IE sucks. I guess Safari does too. But good lord a lot people use them. -Ross. --> Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060825/e8d24c95/attachment.html
Also, I think I want to reiterate at this point about how resolution on this is not as pressing for me anymore. You''re right about the fcgi thing. So... I guess take this however you want :) -Ross. On 8/25/06, Ross Singer <ross.singer at library.gatech.edu> wrote:> > On 8/25/06, Zed Shaw <zedshaw at zedshaw.com> wrote: > > > On Fri, 2006-08-25 at 20:08 -0400, Ross Singer wrote: > > > Zed, > > > > > > Thanks for getting back to me on this. > > > > > > Here is the log from the offending request: > > > > > <snip> > > > Is this what you were looking for? > > > > Yep I''ll look at that. But right from the RFC you have this: > > > > unsafe = (CTL | " " | "\"" | "#" | "%" | "<" | ">"); > > > > Even they label that as "unsafe" so I''m going to be hard pressed to > > allow this in. > > > Again, I wouldn''t say I''m opposed to this. :) > > All I''m sayin'' "Real World Scenario". Also "85% browser market share". > > I can probably talk EBSCO into fixing their bad URLs, but I don''t have any > control over Internet Explorer or Microsoft. I''ve got no influence there. > > I''m not trying to be passive-aggressive at all about this (I really want > to stress that), I''m just stating that this is something that''s bound to > come up again. > > Because IE sucks. I guess Safari does too. > > But good lord a lot people use them. > > -Ross. > > > -- > > > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060825/2f62e066/attachment-0001.html