Mechanize users - Oct 2007 - basic_auth problem since 0.6.9

I have a site that I don''t think "returns" a basic_auth
request, but is
able to use basic_auth.

In the past on 0.6.8, I could use the following code:

require ''rubygems''
# gem ''mechanize'', ''=0.6.8''
require ''mechanize''

agent = WWW::Mechanize.new
agent.basic_auth("username", "password")
site_response = agent.get("https://link.to.site/")

Since 0.6.9, this doesn''t work any longer.

I "think" prior to 0.6.9 Mechanize would just try basic_auth if I used
the
basic_auth method, but now I think if I read prior posts by Aaron 
correctly, Mechanize is making an initial call out to the web page I want 
to connect to to try and find what authentication the web server supports.

In my case (and I don''t know if this might be common, or just totally 
wrong, but this is how the site does it), the web server doesn''t
respond
with basic_auth, so I don''t think mechanize tries using it.  Or maybe
it
is something else entirely.

Connecting with 0.6.10 to "other" sites that give back the basic_auth 
response seems to works fine, so works except in this one case.

If everything I''ve said it true, I wonder if there is a way to force 
Mechanize to just try using basic auth if I use the basic auth method to 
set username password, regardless of what the server says it supports.  I 
just checkout out the source and will try to just hack something, but
I''m
such a new programmer, it''ll likely be a kludge.  Anyhow, thought
I''d
report it.

Here is a 0.6.10 -r 445 logger log:

irb(main):011:0> site_request = agent.get("https://link.to.site")
I, [2007-10-05T12:46:44.841861 #4958]  INFO -- : Net::HTTP::Get: /
D, [2007-10-05T12:46:45.213989 #4958] DEBUG -- : request-header: 
accept-language => en-us,en;q0.5
D, [2007-10-05T12:46:45.214105 #4958] DEBUG -- : request-header: 
connection => keep-alive
D, [2007-10-05T12:46:45.214169 #4958] DEBUG -- : request-header: accept => 
*/*
D, [2007-10-05T12:46:45.214227 #4958] DEBUG -- : request-header: 
accept-encoding => gzip,identity
D, [2007-10-05T12:46:45.214285 #4958] DEBUG -- : request-header: 
user-agent => WWW-Mechanize/0.6.10 
(http://rubyforge.org/projects/mechanize/)
D, [2007-10-05T12:46:45.214344 #4958] DEBUG -- : request-header: 
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2007-10-05T12:46:45.214402 #4958] DEBUG -- : request-header: 
keep-alive => 300
D, [2007-10-05T12:46:45.348892 #4958] DEBUG -- : Read 432 bytes
D, [2007-10-05T12:46:45.349286 #4958] DEBUG -- : Read 1456 bytes
D, [2007-10-05T12:46:45.349583 #4958] DEBUG -- : Read 1510 bytes
D, [2007-10-05T12:46:45.349661 #4958] DEBUG -- : response-header: vary => 
negotiate,BackhandSSL
D, [2007-10-05T12:46:45.349726 #4958] DEBUG -- : response-header: 
connection => close
D, [2007-10-05T12:46:45.349784 #4958] DEBUG -- : response-header: 
bobo-exception-file => /path/to/CORE2.py
D, [2007-10-05T12:46:45.349845 #4958] DEBUG -- : response-header: 
content-location => index.html.asis, index.html.asis
D, [2007-10-05T12:46:45.349903 #4958] DEBUG -- : response-header: 
content-type => text/html; charset=UTF-8
D, [2007-10-05T12:46:45.349962 #4958] DEBUG -- : response-header: 
x-powered-by => Zope (www.zope.org), Python (www.python.org)
D, [2007-10-05T12:46:45.350021 #4958] DEBUG -- : response-header: date => 
Fri, 05 Oct 2007 18:28:43 GMT
D, [2007-10-05T12:46:45.350078 #4958] DEBUG -- : response-header: 
bobo-exception-value => bobo exception
D, [2007-10-05T12:46:45.350136 #4958] DEBUG -- : response-header: 
bobo-exception-type => LoginRequired
D, [2007-10-05T12:46:45.350196 #4958] DEBUG -- : response-header: 
bobo-exception-line => 958
D, [2007-10-05T12:46:45.350253 #4958] DEBUG -- : response-header: server 
=> Apache/1.3.26 (Unix) mod_backhand/1.2.2 mod_fastcgi/2.2.12 
mod_ssl/2.8.10 OpenSSL/0.9.6e
D, [2007-10-05T12:46:45.350311 #4958] DEBUG -- : response-header: 
content-length => 1510
D, [2007-10-05T12:46:45.350369 #4958] DEBUG -- : response-header: tcn => 
choice, choice
I, [2007-10-05T12:46:45.362057 #4958]  INFO -- : status: 500
WWW::Mechanize::ResponseCodeError: 500 => Net::HTTPInternalServerError
         from 
/usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.10/lib/mechanize.rb:185:in 
`get''
         from (irb):11


Thanks,

Gary

> If everything I''ve said it true, I wonder if there is a way to
force
> Mechanize to just try using basic auth if I use the basic auth method to
set
> username password, regardless of what the server says it supports.  I just 
> checkout out the source and will try to just hack something,
Yeah, this is a kludge, but as a test, it does work:

def basic_auth(user, password)
-->   @auth_basic = 1  # set a variable that I really do want basic auth
   auth(user, password)
end

In def fetch_page(...,...)


There is a series of checks for the result class in this method:

---------------

     return page if res_klass <= Net::HTTPSuccess

     if res_klass == Net::HTTPNotModified
     elsif res_klass <= Net::HTTPRedirection
     elsif res_klass <= Net::HTTPUnauthorized
       raise ResponseCodeError.new(page) unless @user || @password
       raise ResponseCodeError.new(page) if @auth_hash.has_key?(uri.host)
       if response[''www-authenticate''] =~ /Digest/i
         @auth_hash[uri.host] = :digest
         @digest = response[''www-authenticate'']
       else
         @auth_hash[uri.host] = :basic
       end

-----------------


If I manually force the auth_hash to be basic, I can connect:

  if @auth_basic == 1
        @auth_hash[uri.host] = :basic
      end

This is a kludge, but it does work.

> If everything I''ve said it true, I wonder if there is a way to
force
> Mechanize to just try using basic auth if I use the basic auth method to 
> set username password, regardless of what the server says it supports. 
> I just checkout out the source and will try to just hack something,
Yeah, this is a kludge, but as a test, it does work:

In the basic_auth method:

def basic_auth(user, password)
-->   @auth_basic = 1  # set a marker that I really do want basic auth
   auth(user, password)
end

Then, in the fetch_page method:

def fetch_page(...,...)

There is a series of checks for the attempt to connect to the site the 
first time.  If the rez_klass returns an HTTPUnauthorized it sets the 
@auth_hash.

---------------

     return page if res_klass <= Net::HTTPSuccess

     if res_klass == Net::HTTPNotModified
     elsif res_klass <= Net::HTTPRedirection
     elsif res_klass <= Net::HTTPUnauthorized
       raise ResponseCodeError.new(page) unless @user || @password
       raise ResponseCodeError.new(page) if @auth_hash.has_key?(uri.host)
       if response[''www-authenticate''] =~ /Digest/i
         @auth_hash[uri.host] = :digest
         @digest = response[''www-authenticate'']
       else
         @auth_hash[uri.host] = :basic
       end

-----------------

In my case, the server doesn''t correctly respond with HTTPUnauthorized,
but if I manually force the auth_hash to be basic, I can connect again:

  if @auth_basic
        @auth_hash[uri.host] = :basic
      end

This is a kludge, but it does work.

> If everything I''ve said it true, I wonder if there is a way to
force
> Mechanize to just try using basic auth if I use the basic auth method to 
> set username password, regardless of what the server says it supports. I 
> just checkout out the source and will try to just hack something,
Yeah, this is a kludge, but as a test, it does work:

In the basic_auth method:

def basic_auth(user, password)
-->   @auth_basic = 1  # set a marker that I really do want basic auth
   auth(user, password)
end

Then, in the fetch_page method:

def fetch_page(...,...)

There is a series of checks for the attempt to connect to the site the 
first time.  If the
rez_klass returns an HTTPUnauthorized it sets the @auth_hash.

---------------

     return page if res_klass <= Net::HTTPSuccess

     if res_klass == Net::HTTPNotModified
     elsif res_klass <= Net::HTTPRedirection
     elsif res_klass <= Net::HTTPUnauthorized
       raise ResponseCodeError.new(page) unless @user || @password
       raise ResponseCodeError.new(page) if @auth_hash.has_key?(uri.host)
       if response[''www-authenticate''] =~ /Digest/i
         @auth_hash[uri.host] = :digest
         @digest = response[''www-authenticate'']
       else
         @auth_hash[uri.host] = :basic
       end

-----------------

In my case, the server doesn''t correctly respond with HTTPUnauthorized,
but if I manually force
the auth_hash to be basic, I can connect again:

  if @auth_basic
        @auth_hash[uri.host] = :basic
      end

This is a kludge, but it does work.