Mechanize users - Aug 2010 - Redirection patch

Hi, all,

I found that I had to make the following patch to Mechanize for one of the
sites I''m scraping:

    alias fetch_page_original_version fetch_page
    def fetch_page(params)
      params[:uri] = params[:uri].gsub(/^https?:/i) {|m| m.downcase } if
String == params[:uri].class
      fetch_page_original_version(params)
    end

(Also, here: http://pastie.org/1077542)

The problem was that the site was returning a 302 redirect with a Location
header that looked like: httpS://www.blah-blah-blah... The weirdly
capitalized protocol was causing EOF errors, so it needed to be adjusted.

My versions:

mechanize (1.0.0)
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]

Hopefully, this is helpful to someone.

Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/mechanize-users/attachments/20100805/e04b055d/attachment.html>

Hi Aaron,

On Thu, Aug 5, 2010 at 11:20 AM, Aaron Starr <astarr at wiredquote.com>
wrote:
> Hi, all,
>
> I found that I had to make the following patch to Mechanize for one of the
> sites I''m scraping:
>
>     alias fetch_page_original_version fetch_page
>     def fetch_page(params)
>       params[:uri] = params[:uri].gsub(/^https?:/i) {|m| m.downcase } if
> String == params[:uri].class
>       fetch_page_original_version(params)
>     end
>
> (Also, here: http://pastie.org/1077542)
>
> The problem was that the site was returning a 302 redirect with a Location
> header that looked like: httpS://www.blah-blah-blah... The weirdly
> capitalized protocol was causing EOF errors, so it needed to be adjusted.
>
> My versions:
>
> mechanize (1.0.0)
> ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
>
> Hopefully, this is helpful to someone.
>
I''ve opened the following issue to record this information and
hopefully
pull the fix into a future version of mechanize:

http://github.com/tenderlove/mechanize/issues#issue/44

Thanks for reporting it!

>
> Aaron
>
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/mechanize-users/attachments/20100805/34f8a045/attachment.html>