Mechanize users - Jun 2009 - Nokogiri encoding bug

Hi there,

I spotted in the list archive someone having the same problem as I''ve 
just run into. I just thought I''d feed back some info in case it was 
useful for future googlers.  Here''s a typical backtrace:

$ ruby -Itest -Ilib test/test_blank_form.rb
Loaded suite test/test_blank_form
Started
E
Finished in 0.003698 seconds.

   1) Error:
test_blank_form_query_string(BlankFormTest):
TypeError: can''t convert nil into String
     /home/alex/gems/mechanize/lib/www/mechanize/util.rb:41:in `iconv''
     /home/alex/gems/mechanize/lib/www/mechanize/util.rb:41:in 
`from_native_charset''
     /home/alex/gems/mechanize/lib/www/mechanize/form.rb:153:in 
`from_native_charset''
     /home/alex/gems/mechanize/lib/www/mechanize/form.rb:144:in
`proc_query''
     /home/alex/gems/mechanize/lib/www/mechanize/form.rb:143:in `map''
     /home/alex/gems/mechanize/lib/www/mechanize/form.rb:143:in
`proc_query''
     /home/alex/gems/mechanize/lib/www/mechanize/form.rb:167:in 
`build_query''
     /home/alex/gems/mechanize/lib/www/mechanize/form.rb:166:in `each''
     /home/alex/gems/mechanize/lib/www/mechanize/form.rb:166:in 
`build_query''
     test/test_blank_form.rb:11:in `test_blank_form_query_string''

1 tests, 0 assertions, 0 failures, 1 errors

Versions are:

mechanize: 0.9.3.20090617085936, fresh from github
nokogiri: 1.2.3, gem installed
Nokogiri::LIBXML_VERSION: 2.6.32
OS: 64-bit Ubuntu Jaunty
ruby -v: ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]

I''ve tracked it back to the WWW::Mechanize::Page#parser method, 
specifically line 75:

   @parser = mech.html_parser.parse(html_body, nil, @encoding)

For a page I''m working on, @encoding is "utf-8" before this
call (it''s
set in a Content-Type header), but @parser.encoding is nil immediately 
afterwards, and that''s what eventually ends up being passed to iconv as
the encoding value. I''m going to try following this upstream, so there 
might be a pull request coming in at some point if I get anywhere with 
it. That being said, a quick fix would be to simply ignore the value 
that comes back from the parser. Since we''ve already got the encoding, 
what more can the parser tell us? I don''t understand that bit yet.

Thanks,
-- 
Alex

Alex Young wrote:
> Hi there,
<snip>
> That being said, a quick fix would be to simply ignore the value 
> that comes back from the parser. Since we''ve already got the
encoding,
> what more can the parser tell us? I don''t understand that bit yet.
Just a quick follow-up. The easiest patch to do this is:

diff --git a/lib/www/mechanize/page.rb b/lib/www/mechanize/page.rb
index 1f7d884..ac6909e 100644
--- a/lib/www/mechanize/page.rb
+++ b/lib/www/mechanize/page.rb
@@ -64,7 +64,7 @@ module WWW
        end

        def encoding
-        parser.respond_to?(:encoding) ? parser.encoding : nil
+        (parser.respond_to?(:encoding) ? parser.encoding : nil) || 
@encoding
        end

        def parser

That results in all the tests passing, except for the following:

   1) Failure:
test_another_mostly_broken_charset(TestPage) [./test/test_page.rb:32]:
<"UTF8"> expected but was
<nil>.

   2) Failure:
test_encoding_override_after_parser_was_initialized(TestPage) 
[./test/test_page.rb:58]:
<"ISO-8859-2"> expected but was
<"windows-1255">.

   3) Failure:
test_encoding_override_before_parser_initialized(TestPage) 
[./test/test_page.rb:47]:
<"ISO-8859-2"> expected but was
<"windows-1255">.

   4) Failure:
test_page_decoded_with_charset(TestPage) [./test/test_page.rb:99]:
<"EUC-JP"> expected but was
<nil>.

   5) Failure:
test_set_encoding(TestPage) [./test/test_page.rb:69]:
<"UTF-8"> expected but was
<nil>.

291 tests, 1502 assertions, 5 failures, 0 errors

There''s clearly something screwy going on with libxml2''s
HTMLParser, but
I don''t have much more than that yet.

Hope this helps.

-- 
Alex