Rails core - Sep 2011 - Hitting 500 status code on invalid UTF-8 byte sequence in params

Hello Core,

Background: with Redmine-1.2 on Rails-2.3.11 we''re experiencing a
"500
Internal Server Error" every time a mail with invalid UTF-8 byte
sequence is hitting the MailHandler (the exception comes from the DB
complaining about invalid string in the INSERT statement.)

Since we do care about internal errors we''ve installed
exception_notification plugin and this results in mailbox flood for
our Redmine admins, coming from these internal errors.  Besides that,
not being able to properly handle invalid input is just a shame. ;-)

Now, most of the time (100% so far, to be honest,) these failing mails
are spam, but it would be really nice if we could alleviate the
problem by repairing the invalid byte sequences prior to passing
params to the application.  This far we have short of the options but
disabling exceptions_notification, which is not what we''d really like
to do.

It is my belief, that trying to fix this in the application realm
isn''t worth the effort: it has to be a core fix instead.

From what I can see, the problem has been noted in a more recent
version of Rails:

https://github.com/rails/rails/blob/3-1-stable/actionpack/lib/action_dispatch/http/parameters.rb#L39

      # TODO: Validate that the characters are UTF-8. If they aren''t,
      # you''ll get a weird error down the road, but our form handling
      # should really prevent that from happening
      def encode_params(params)

That''s exactly what happens to us, and since the data isn''t
coming
from a webform we don''t enjoy the mentioned form handling.

I''d vote for a fix along the lines of
Iconv.conv("#{$KCODE}//IGNORE",
$KCODE, ...), but I''m not sure what would be a good place to apply
that, more so that 2.3 and 3.x differ greatly in this regard, AFAICT.

--
Regards,
Alex

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

On Sep 14, 5:15 pm, Alex Shulgin <alex.shul...@gmail.com>
wrote:
>
> From what I can see, the problem has been noted in a more recent
> version of Rails:
>
> https://github.com/rails/rails/blob/3-1-stable/actionpack/lib/action_...
>
>       # TODO: Validate that the characters are UTF-8. If they
aren''t,
>       # you''ll get a weird error down the road, but our form
handling
>       # should really prevent that from happening
>       def encode_params(params)
>
> That''s exactly what happens to us, and since the data
isn''t coming
> from a webform we don''t enjoy the mentioned form handling.
>
> I''d vote for a fix along the lines of
Iconv.conv("#{$KCODE}//IGNORE",
> $KCODE, ...), but I''m not sure what would be a good place to apply
> that, more so that 2.3 and 3.x differ greatly in this regard, AFAICT.
I wonder if this will get more attention if I actually submit a
patch...

What would be the best place to apply the Iconv translation?  I mean
in the Rails world params are hashed already, so it might make sense
to do this at some earlier stage, like in Rack?

--
Alex

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.