Rails core - Aug 2007 - XML Parsing (included malformed DTD), 99% CPU DoS Attack

Hi, Rails Experts!

I think this is important so please read this message.

There is a DoS for Ruby on Rails that is easily exploitable.
The attack involves sending a malformed xml document in
an HTTP request.


There has been an exploitable spec in the Ruby on Rails
which allows:

Anyone on the Internet to...
Send a single HTTP request to...
Any Rails Web application program using...
Rexml parsing with...
A malformed Xml document that...
Causes the Ruby process to go into a 99% CPU heavily-loaded loop.


What happens is that the Xml document (included DTD) is given as:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE foobar [
      <!ENTITY x0 "hello">
      <!ENTITY x1 "&x0;&x0;">
      <!ENTITY x2 "&x1;&x1;">
      <!ENTITY x3 "&x2;&x2;">
---- repeat 100 times -----------------
      <!ENTITY x100 "&x99;&x99;">
]>
<SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema"

xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-
instance">
      <SOAP-ENV:Body>
              <ns1:echo
xmlns:ns1="aaa"><b>&x100;</b>
              </ns1:echo>
      </SOAP-ENV:Body>
</SOAP-ENV:Envelope>




And this causes REXML to go into an heavily-loaded xml parsing loop.

Currently, as far as I know, Rails 1.1.x and Rails 1.2.x are affected.
Maybe, all of the Rails based Web application.

And Mongrel, Lighttpd + FastCGI, and WEBrick are vulnerable.



I''m looking for a hot fix release, or a avoidance technique.
(For example, How to disable the DTD for Rails XML Parser.)

Please any one help me!


---
3arrows


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

> I''m looking for a hot fix release, or a avoidance technique.
> (For example, How to disable the DTD for Rails XML Parser.)
Hi 3arrows,

Thanks for the report. I''ve immediately tried to contact the REXML
guys. This will be a problem for all Ruby applications that accept XML
from the outside world. So a hotfix should preferably come from them,
so it can benefit everyone in the Ruby community.

But in the mean time, it''d be great if others also looked into ways of
stopping this from the Rails side.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

On 8/4/07, DHH <david.heinemeier@gmail.com> wrote:
>
>
> > I''m looking for a hot fix release, or a avoidance technique.
> > (For example, How to disable the DTD for Rails XML Parser.)
>
> Hi 3arrows,
>
> Thanks for the report. I''ve immediately tried to contact the REXML
> guys. This will be a problem for all Ruby applications that accept XML
> from the outside world. So a hotfix should preferably come from them,
> so it can benefit everyone in the Ruby community.
>
> But in the mean time, it''d be great if others also looked into
ways of
> stopping this from the Rails side.

Is this a DOM mode only thing?  Is the SAX streaming affected as well?

-Daniel

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

On Aug 3, 10:55 am, "Daniel N" <has....@gmail.com>
wrote:
> On 8/4/07, DHH <david.heineme...@gmail.com> wrote:
>
>
>
> > > I''m looking for a hot fix release, or a avoidance
technique.
> > > (For example, How to disable the DTD for Rails XML Parser.)
>
> > Hi 3arrows,
>
> > Thanks for the report. I''ve immediately tried to contact the
REXML
> > guys. This will be a problem for all Ruby applications that accept XML
> > from the outside world. So a hotfix should preferably come from them,
> > so it can benefit everyone in the Ruby community.
>
> > But in the mean time, it''d be great if others also looked
into ways of
> > stopping this from the Rails side.
I''ve done some investigation into what''s going on. 
It''s the &x100;
entity causing issues.  At the top of the doctype you define

<!ENTITY x0 "hello">

So, &x0; is ''hello''.  Then:

<!ENTITY x1 "&x0;&x0;">

&x1; is ''hellohello''

That cycle continues, so &x100; would have to navigate backwards
through all 99 entities and figure out what the value is.  It''s pretty
big :)  This is probably something for REXML or XmlSimple to fix.  In
the meantime, someone could create a filter that scrubs doctype
entities before processing them.

Also, don''t bother pinging rails apps, you can test it locally like
this:

require ''rubygems''
require ''xmlsimple''
XmlSimple.xml_in ''bad-file.xml''


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

Lame one-line fix until someone smarter than me comes up with one:

REXML::Document.class_eval { def doctype() nil end }

This hides any doctype so XmlSimple will ignore it.  I think it''s good
enough until this gets properly patched.

script/plugin install
http://svn.techno-weenie.net/projects/plugins/xml_simple_doctype_fix

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

Thanks a lot, Rick and you all!

Rick''s code works very well on my test confirming the risk of this
Dos.
It will resolve serious, widespread problems.

(But, I''ve never checked the adverse affect.)
I''ll go on additional tests.


This setting is desirable from the viewpoint of security of the Rails
based Web Application.
As far as I know, Major commercial server applications reject the
outside DTD(Doctypes) by default setting.


I should like to thank you promptly and fully for your cooperation.




On 8月4日, 午前1:50, "Rick Olson" <technowee...@gmail.com>
wrote:
> Lame one-line fix until someone smarter than me comes up with one:
>
> REXML::Document.class_eval { def doctype() nil end }
>
> This hides any doctype so XmlSimple will ignore it.  I think it''s
good
> enough until this gets properly patched.
>
> script/plugin
installhttp://svn.techno-weenie.net/projects/plugins/xml_simple_doctype_fix

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

This is an exponential expansion of entities -- the test XML is
requiring REXML to produce a string that is 5 * 2^100 characters long.

This isn''t something for REXML to fix, because it isn''t a bug;
it is a
side-effect of the proper operation of the entity expansion.  You have
two options:  either apply the doctype-erasing process that was
suggested by Rick, or have the code that processes the SOAP message
check for DOS attacks.

Cheers,

--- SER


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

Here''s a little more information:

There are two solutions: either remove the doctype declaration, or
have the code that interprets the SOAP message check for DOS attacks.
Removing the doctype can be done either prior to parsing:

	REXML::Document.class_eval { def doctype() nil end }

(as suggested by Rick) or after parsing:

	d = REXML::Document.new( bad_xml )
	d << REXML::DocType.new

Unfortunately for both of these cases, the XML spec requires that
undefined entities be reported as errors, so that trying to read the
text will result in an Exception.  Therefore, it really is up to the
SOAP toolkit to perform sanity checks on the DocType to see if an
exploit like this is being attempted -- you should NOT blindly delete
the DocType in an attempt to avoid this exploit.

Cheers,

--- SER


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

Is this important enough to warrant a message on the ''rubyonrails-
security'' group/mailing list? Not everybody monitors this one.

NeilW

On Aug 3, 4:47 pm, DHH <david.heineme...@gmail.com>
wrote:
> But in the mean time, it''d be great if others also looked into
ways of
> stopping this from the Rails side.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

On 8/3/07, Rick Olson <technoweenie@gmail.com>
wrote:
>
> Lame one-line fix until someone smarter than me comes up with one:
>
> REXML::Document.class_eval { def doctype() nil end }
>
> This hides any doctype so XmlSimple will ignore it.  I think it''s
good
> enough until this gets properly patched.
>
> script/plugin install
> http://svn.techno-weenie.net/projects/plugins/xml_simple_doctype_fix
Also, if you don''t need to support XML uploads in your application,
you can disable Rails use of XMLSimple by including the following in
your environment:

  ActionController::Base.param_parsers.delete(Mime::XML)

This is untested but should work if I''m reading the source correctly.
If a core member could confirm, that''d be great.

Jeremy

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

> Also, if you don''t need to support XML uploads in your
application,
> you can disable Rails use of XMLSimple by including the following in
> your environment:
>
>   ActionController::Base.param_parsers.delete(Mime::XML)
>
> This is untested but should work if I''m reading the source
correctly.
> If a core member could confirm, that''d be great.
That should work fine,  the other options is to override
XmlSimple.xml_in to a no-op or raise an exception

-- 
Cheers

Koz

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---