Mongrel users - Oct 2006 - [SEC] Mongrel Temporary Fix For cgi.rb 99% CPU DoS Attack

On 10/25/06, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
>
> There is a DoS for Ruby''s cgi.rb that is easily exploitable.  The
attack
> involves sending a malformed multipart MIME body in an HTTP request.  The
> full explanation of the attack as well as how to fix it RIGHT NOW is given
> below.
>
> I''m putting this fix into the Mongrel pre-release process to give
Matz
> time to get an official release out.  If he doesn''t within the
next few days
> then I''ll turn this into an official Mongrel release.

To underline and bold: you''re unaffected if you''re in
production on FastCGI.

The vulnerability has been reported to security at ruby-lang.org and the
various OS distros. Matz fixed it in 1.8 CVS but hasn''t backported,
hence
the full disclosure and hotfix now.

Track it at http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-5467(should
be up shortly).

jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061025/d8ddcfb1/attachment-0001.html

Can anyone help with a way to have a DIV follow along while a user is
scrolling down a long screen?  I have seen it on several sites, but
don''t know how to accomplish it.

Thanks,
Nathan


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk
-~----------~----~----~----~------~----~------~--~---

This is important so please read this message very carefully.

There is a DoS for Ruby''s cgi.rb that is easily exploitable.  The
attack involves sending a malformed multipart MIME body in an HTTP request.  The
full explanation of the attack as well as how to fix it RIGHT NOW is given
below.

Most of the work was done by Jeremy Kemper and Jamis Buck.  They did all the
work of building the hot fix gem you''ll install and getting the right
people to finally agree to get this out.

The original report is attached to this message so you can read it in full.

I''m putting this fix into the Mongrel pre-release process to give Matz
time to get an official release out.  If he doesn''t within the next few
days then I''ll turn this into an official Mongrel release.

FULL DISCLOSURE

There has been an exploitable bug in the Ruby CGI library named cgi.rb which
allows:

Anyone on the Internet to...
Send a single HTTP request to...
Any Ruby program (NOT just Mongrel) using...
cgi.rb multipart parsing with...
A malformed MIME body that...
Causes the Ruby process to go into a 99% CPU infinite loop killing it.

I broke this down so that it''s as clear as possible, and so you
don''t miss that it''s for ANY program using cgi.rb mime
parsing.  Not just Rails and Mongrel.

What happens is that the final MIME boundary is sometimes given as:

-ASDFADSFASFD--

Rather than:

--ADFADSFADSF--

And this causes cgi.rb to go into an infinite loop waiting for more input that
isn''t coming.  This is caused by any system that reads directly from an
input stream that returns "" rather than EOF.

The fix described below has a full exploit/tester script demonstrating the
defect.  It also doesn''t matter if you have file uploads on your site
or not.  I can point this script at your site on any URI and cause a DoS on your
site.

WHO''S AFFECTED

Currently, the following servers are affected:

* Mongrel -- Reads from a socket so gets "" rather than EOF.
* Litespeed -- Affected but has an internal timeout that nails the process.
* CGI Standalone -- Impacted since reading from a normal input stream.
* Any other custom server using the above similar operations.

Looks like FastCGI''s FCGIInputStream, WEBrick and mod_ruby are not
vulnerable since they either read from a domain socket or don''t use
normal cgi.rb.

THE FIX

Everyone using Mongrel can get the fix immediately by installing the latest
pre-release version 0.3.14:

sudo gem install mongrel --source=http://mongrel.rubyforge.org/releases

Win32 people and anyone who can''t upgrade that way can get the fix by
doing this:

1) gem install cgi_multipart_eof_fix
--source=http://mongrel.rubyforge.org/releases
2) Edit your environment.rb to have:  require
''cgi_multipart_eof_fix''
3) Restart your services.

People using other frameworks can get the fix by simply requiring rubygems and
this fix in some start-up location for your framework.

THE PATCH

If you can''t do the hot fix gem install, then there is also a patch for
cgi.rb attached to this e-mail.  You can apply the patch with the following
process:

1) Find the original cgi.rb file in your install. Mine''s in
/usr/lib/ruby/1.8
2) cd /usr/lib/ruby/1.8
3) sudo patch < ~/cgi_multipart_eof_fix.patch

You can look at the patch.  It''s literally changing one line, so you
can edit by hand if you get really desperate.

FUTURE DEFECTS

Based on how the cgi.rb file is coded it''s most likely that there will
be more of these kinds of defects in the future.  If you find a defect like
this, then please don''t flip out.  Just report it to me or anyone else,
and I''ll cook up another one of these hot fix releases rather than wait
for an official fix.  I promise immediate turn-around from now on using a
hot-fix gem if I can''t get an official fix within a few days.

Suggestions on how to do a more standardized hot-fix release process are much
appreciated.

Flame wars about screwing goats or the merits of full-disclosure are not
appreciated.

GETTING HELP

I''ll be in the Mongrel lingr room:

http://www.lingr.com/room/3yXhqKbfPy8

And on irc.freenode.org in #rubyonrails, #rails-security, and #ruby-lang
fielding questions and helping people.  If I don''t answer right away
then wait a bit.

I''ll also answer help e-mails directly if you can''t access any
of the above.

---
Zed A. Shaw
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Jeremy_Kemper_DoS_report.txt
Type: application/octet-stream
Size: 904 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mongrel-users/attachments/20061025/09b5f29a/attachment.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cgi_multipart_eof_fix.patch
Type: application/octet-stream
Size: 378 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mongrel-users/attachments/20061025/09b5f29a/attachment-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ChangeLog
Type: application/octet-stream
Size: 3868 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mongrel-users/attachments/20061025/09b5f29a/attachment-0002.obj

This is important so please read this message very carefully.

There is a DoS for Ruby''s cgi.rb that is easily exploitable.  The
attack involves sending a malformed multipart MIME body in an HTTP request.  The
full explanation of the attack as well as how to fix it RIGHT NOW is given
below.

I''m putting this fix into the Mongrel pre-release process to give Matz
time to get an official release out.  If he doesn''t within the next few
days then I''ll turn this into an official Mongrel release.

FULL DISCLOSURE

There has been an exploitable bug in the Ruby CGI library named cgi.rb which
allows:

Anyone on the Internet to...
Send a single HTTP request to...
Any Ruby program (NOT just Mongrel) using...
cgi.rb multipart parsing with...
A malformed MIME body that...
Causes the Ruby process to go into a 99% CPU infinite loop killing it.

I broke this down so that it''s as clear as possible, and so you
don''t miss that it''s for ANY program using cgi.rb mime
parsing.  Not just Rails and Mongrel.

What happens is that the final MIME boundary is sometimes given as:

-ASDFADSFASFD--

Rather than:

--ADFADSFADSF--

And this causes cgi.rb to go into an infinite loop waiting for more input that
isn''t coming.  This is caused by any system that reads directly from an
input stream that returns "" rather than EOF.

The fix described below has a full exploit/tester script demonstrating the
defect.  It also doesn''t matter if you have file uploads on your site
or not.  I can point this script at your site on any URI and cause a DoS on your
site.

WHO''S AFFECTED

Currently, the following servers are affected:

* Mongrel -- Reads from a StringIO so gets "" rather than EOF.
* Litespeed -- Affected but has an internal timeout that nails the process.
* CGI Standalone -- Impacted since reading from a normal input stream.
* Any other custom server using the above similar operations.

Looks like FastCGI''s FCGIInputStream, WEBrick and mod_ruby are not
vulnerable since they either read from a domain socket or don''t use
normal cgi.rb.

THE FIX

Everyone using Mongrel can get the fix immediately by installing the latest
pre-release version 0.3.14:

sudo gem install mongrel --source=http://mongrel.rubyforge.org/releases

Win32 people and anyone who can''t upgrade that way can get the fix by
doing this:

1) gem install cgi_multipart_eof_fix
--source=http://mongrel.rubyforge.org/releases
2) Edit your environment.rb to have:  require
''cgi_multipart_eof_fix''
3) Restart your services.

People using other frameworks can get the fix by simply requiring rubygems and
this fix in some start-up location for your framework.

THE PATCH

If you can''t do the hot fix gem install, then there is also a patch for
cgi.rb attached to this e-mail.  You can apply the patch with the following
process:

1) Find the original cgi.rb file in your install. Mine''s in
/usr/lib/ruby/1.8
2) cd /usr/lib/ruby/1.8
3) sudo patch < ~/cgi_multipart_eof_fix.patch

You can look at the patch.  It''s literally changing one line, so you
can edit by hand if you get really desperate.

FUTURE DEFECTS

Based on how the cgi.rb file is coded it''s most likely that there will
be more of these kinds of defects in the future.  If you find a defect like
this, then please don''t flip out.  Just report it to me or anyone else,
and I''ll cook up another one of these hot fix releases rather than wait
for an official fix.  I promise immediate turn-around from now on using a
hot-fix gem if I can''t get an official fix within a few days.

Suggestions on how to do a more standardized hot-fix release process are much
appreciated.

Flame wars about screwing goats or the merits of full-disclosure are not
appreciated.

GETTING HELP

I''ll be in the Mongrel lingr room:

http://www.lingr.com/room/3yXhqKbfPy8

And on irc.freenode.org in #rubyonrails, #rails-security, and #ruby-lang
fielding questions and helping people.  If I don''t answer right away
then wait a bit.

I''ll also answer help e-mails directly if you can''t access any
of the above.

CREDITS

Most of the work was done by Jeremy Kemper and Jamis Buck.  They did all the
work of building the hot fix gem you''ll install and getting the right
people to finally agree to get this out.

The original report is attached to this message so you can read it in full.

---
Zed A. Shaw


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk
-~----------~----~----~----~------~----~------~--~---

I don''t think it''s possible to this with CSS, I think you need
some
javascript that essentially resets the style.top on the div element on
the page scroll event.

On 10/25/06, Nathan Leach
<nathan.leach-8O96a5ImwvDQT0dZR+AlfA@public.gmane.org>
wrote:
>
> Can anyone help with a way to have a DIV follow along while a user is
> scrolling down a long screen?  I have seen it on several sites, but
> don''t know how to accomplish it.
>
> Thanks,
> Nathan
>
>
> >
>

-- 
Thomas Mango
tsmango-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk
-~----------~----~----~----~------~----~------~--~---

> I don''t think it''s possible to this with CSS, I think you
need some
> javascript that essentially resets the style.top on the div element on
> the page scroll event.
Actually it is... depending on what you mean by "follow along".  If
you
don''t mind it being fixed you can...

http://www.cssplay.co.uk/layouts/fixed.html
>
> On 10/25/06, Nathan Leach
<nathan.leach-8O96a5ImwvDQT0dZR+AlfA@public.gmane.org> wrote:
>>
>> Can anyone help with a way to have a DIV follow along while a user is
>> scrolling down a long screen?  I have seen it on several sites, but
>> don''t know how to accomplish it.
>>
>> Thanks,
>> Nathan
>>
>>
>>>
>>
>
>
> -- 
> Thomas Mango
> tsmango-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk
-~----------~----~----~----~------~----~------~--~---

Zed A. Shaw wrote:
>There is a DoS for Ruby''s cgi.rb that is easily exploitable.  The
attack involves sending a malformed multipart MIME body in an HTTP request.  The
full explanation of the attack as well as how to fix it RIGHT NOW is given
below.
>  
>
Another quick fix is to download the latest cgi.rb from the Ruby CVS 
repository and install that as your system cgi.rb:

    # wget -O cgi.rb 
''http://www.ruby-lang.org/cgi-bin/cvsweb.cgi/~checkout~/ruby/lib/cgi.rb?rev=1.91;content-type=application%2Fx-ruby''
    # install -m 644 cgi.rb /usr/lib/ruby/1.8/cgi.rb

You will still get spinners, however (though the malformed boundary 
spinners do seem to go away).

Using 0.3.13.5 + the revision 356 patch (not really sure how necessary 
that was), along with replacing cgi.rb, has solved most of our mongrel woes.

Also, if you''re not using monit yet, seriously consider embracing it
now.

- Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/
PS. In our testing, we did see WEBrick affected by this as well. It is 
not just Mongrel. FCGI appears to weather this particular problem just fine.

On Wed, 25 Oct 2006 19:51:58 -0400
"Ian C. Blenke" <ian at blenke.com> wrote:
> Zed A. Shaw wrote:
> 
> >There is a DoS for Ruby''s cgi.rb that is easily exploitable. 
The attack involves sending a malformed multipart MIME body in an HTTP request. 
The full explanation of the attack as well as how to fix it RIGHT NOW is given
below.
> Using 0.3.13.5 + the revision 356 patch (not really sure how necessary 
> that was), along with replacing cgi.rb, has solved most of our mongrel
woes.
>
If you ever get desperate for a previous pre-release of Mongrel, you can just go
here:

http://mongrel.rubyforge.org/releases/gems/

And find almost everything for all time.
 
BTW, how was 0.3.14 pre-release for you?  I''ve got reports it somehow
breaks X-Sendfile support.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://safari.oreilly.com/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Zed A. Shaw wrote:
>If you ever get desperate for a previous pre-release of Mongrel, you can
just go here:
>
>http://mongrel.rubyforge.org/releases/gems/
>
>And find almost everything for all time.
>  
>
Yes. I''ve done this on occasion, that is a _very_ useful resource.
Thank
you for leaving it up!
>BTW, how was 0.3.14 pre-release for you?  I''ve got reports it
somehow breaks X-Sendfile support.
>  
>
We haven''t experienced a problem with it yet.

In lingr chat, Evan mentioned his problem was with apache2.2, camping, 
and 0.3.14.

We''re using debian ruby 1.8.4-1, apache 2.2.3, mod_proxy_balancer, no 
camping, and mongrel 0.3.13.5 (is this pre-0.3.14?) plus the patch for 
svn rev 356, and ruby cvs cgi.rb rev 1.19. I haven''t seen any missing 
file errors in the apache logs since the upgrade from 0.3.13.3.

- Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/

Ian C. Blenke wrote:
> We''re using debian ruby 1.8.4-1, apache 2.2.3, mod_proxy_balancer,
no
> camping, and mongrel 0.3.13.5 (is this pre-0.3.14?) plus the patch for 
> svn rev 356, and ruby cvs cgi.rb rev 1.19. I haven''t seen any
missing
> file errors in the apache logs since the upgrade from 0.3.13.3.

It just occurred to me, we''re serving public/ files statically from 
apache2.2 without involving mongrel at all. In this case, anything in 
public/ would be sent directly without the need for mongrel to do an 
X-Sendfile direction to Apache.

  # Redirect all non-static requests to cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]

If I comment out the RewriteCond, and shunt all requests to mongrel, I 
don''t see any missing static content or errors in the logs, but then 
again I haven''t done anything special to enable sendfile support (is 
there a gem I should install for mongrel to use to help test this?)

It looks like Coda Hale has experienced this, and is suggesting that 
folks do _not_ use sendfile if you can have something else test for 
static content and send it instead:

    
http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/

Unless this has changed recently... (as most things rails tend to do)

- Ian C. Blenke <ian at blenke.com> http://ian.blenke.com

On Thu, 26 Oct 2006 10:11:36 -0400
"Ian C. Blenke" <ian at blenke.com> wrote:
> Zed A. Shaw wrote:
> 
> We''re using debian ruby 1.8.4-1, apache 2.2.3, mod_proxy_balancer,
no
> camping, and mongrel 0.3.13.5 (is this pre-0.3.14?) plus the patch for 
> svn rev 356, and ruby cvs cgi.rb rev 1.19. I haven''t seen any
missing
> file errors in the apache logs since the upgrade from 0.3.13.3.
What''s this "patch for svn rev 356" you speak of?  Is it
something I should include?

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://safari.oreilly.com/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Zed A. Shaw wrote:
>What''s this "patch for svn rev 356" you speak of?  Is it
something I should include?
>  
>
I stole it blindly from Michael Moen''s mongrel-users list posts
recently:

    http://rubyforge.org/pipermail/mongrel-users/2006-October/001928.html
    http://rubyforge.org/pipermail/mongrel-users/2006-October/001938.html

I figured that a few lines couldn''t hurt:

    mongrel$ svn diff -r 355:356 trunk/

    Index: trunk/lib/mongrel.rb
    ==================================================================    ---
trunk/lib/mongrel.rb        (revision 355)
    +++ trunk/lib/mongrel.rb        (revision 356)
    @@ -219,7 +219,9 @@
             read_body(remain, content_length, dispatcher)
           end

    -      @body.rewind if body
    +      raise HttpParserError.new("BAD CLIENT: Actual body length 
does not match Content-Length") if @body.pos != content_length
    +
    +      @body.rewind if @body
         end


    @@ -687,7 +689,6 @@
                   reap_dead_workers("max processors")
                 else
                   thread = Thread.new(client) {|c| process_client(c) }
    -              thread.abort_on_exception = true
                   thread[:started_on] = Time.now
                   @workers.add(thread)
   

- Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/

Just in case this trips anyone else up...

I applied Ian''s suggested patch

# wget -O cgi.rb
''http://www.ruby-lang.org/cgi-bin/cvsweb.cgi/~checkout~/ruby/lib/cgi.rb?rev=1.91;content-type=application%2Fx-ruby''
# install -m 644 cgi.rb /usr/lib/ruby/1.8/cgi.rb

but this caused my app to throw an error - looks like something to do
with the FileColumn plugin.

undefined method `size'' for
#<CGI::QueryExtension::MorphingBody:0xb757a62c>
 [RAILS_ROOT]/vendor/plugins/file_column-0.3.2/lib/file_column.rb:57:in
`assign''

Anyway I fixed this by running Ian''s patch with

http://www.ruby-lang.org/cgi-bin/cvsweb.cgi/~checkout~/ruby/lib/cgi.rb?rev=1.68.2.18;content-type=application%2Fx-ruby

which is the current cgi.rb from the ruby1.8 branch rather than the Main branch.

Thanks for the heads up Ian & Zed

Sam

On Thu, 26 Oct 2006 17:31:46 -0400
"Ian C. Blenke" <ian at blenke.com> wrote:
> Zed A. Shaw wrote:
> 
> >What''s this "patch for svn rev 356" you speak of? 
Is it something I should include?
> >  
> >
> 
> I stole it blindly from Michael Moen''s mongrel-users list posts
recently:
> 
>     http://rubyforge.org/pipermail/mongrel-users/2006-October/001928.html
>     http://rubyforge.org/pipermail/mongrel-users/2006-October/001938.html
> 
Ah, so this is in the latest pre-release already then.  Ok, great.  I just have
to find out why X-Sendfile is suddenly not working for Camping behind Apache
(other situations?).

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://safari.oreilly.com/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Sam Giffney wrote:
>Anyway I fixed this by running Ian''s patch with
>
>http://www.ruby-lang.org/cgi-bin/cvsweb.cgi/~checkout~/ruby/lib/cgi.rb?rev=1.68.2.18;content-type=application%2Fx-ruby
>
>which is the current cgi.rb from the ruby1.8 branch rather than the Main
branch.
>  
>
Yeah, I think Zed''s gem fix is the best approach - it patches the 
missing end boundary spin problem without any other side effects:
   
    gem install cgi_multipart_eof_fix 
--source=http://mongrel.rubyforge.org/releases

I''ve dropped the cgi.rb update directly from CVS and moved to this fix 
myself, which seems to work just fine.

Kudos Zed.

 - Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/