thr3ads.net - Mechanize users - [Mechanize-users] mechanize: 400 Bad Request [Nov 2006]

If this information is useful, please help other people find it:
Share via:

Friedrich, Axel

2006-Nov-07 20:39 UTC

[Mechanize-users] mechanize: 400 Bad Request

Hello,

when trying to access a certain HTML-frame, I get:
"in `request'': Unhandled response
(WWW::Mechanize::ResponseCodeError)"

and the page returns:  "400 Bad Request"

* Why?
* How to solve this?

With browser, it works.

In the logs below, I marked 4 lines with "***", where I see possible  
differences in the URI. But I don''t know, if this is the reason for the
malfuncion and how to fix this.

mechanize:
Net::HTTP::Get:  
https://www.frankfurter-fondsbank.de/../diverse/navigation.jsp;jsessionid=CdEfG!-34567!-7654?menu=1

firefox-LiveHTTPHeaders:
GET /diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1 HTTP/1.1


Thank you for any help!

Axel

axel ? friedrich ? _smail  AT   gmx ? de


Details
???????
ruby 1.8.4 (2005-12-24) [i386-mswin32]
Windows 98SE


Code
????
require ''rubygems''
require ''mechanize''

agent = WWW::Mechanize.new
page agent.get("https://www.frankfurter-fondsbank.de/login/Logon.jsp")

... login stuff...

begin
   framePage = agent.click( page2.frames.text(''nav'') )   #
This works not
   # framePage = agent.click( page2.frames.text(''head'') )  #
This works
rescue WWW::Mechanize::ResponseCodeError => ex
     puts ex.page.body
end


page2 looks like this
?????????????????????
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN"
"http://www.w3.org/TR/html4/frameset.dtd">
<html>
...
<!-- Wenn man nicht von start.jsp kommt, dann FS loesen (zB neutrale
Login Seite)-->
<script type="text/javascript">
if (parent.frames.length > 0 && parent.frames[0].name !=
''mainframe'')
        
parent.location="/diverse/indexKunde.jsp;jsessionid=fafdsfsd!-132!-12132";
</script>

<frameset rows="100,*" framespacing="0"
border="0" frameborder="0">
         <frame name="head"
src="/brokerdesign/default/head.jsp;jsessionid=fafdsfsd!-132!-12132"
scrolling="no" marginwidth="0" marginheight="0"
noresize=""/>
         <frameset cols="199,*" framespacing="0"
border="0" frameborder="0">
                 <frameset rows="*,75,0" framespacing="0"
border="0"
frameborder="0">
                     <frame name="nav"
src="../diverse/navigation.jsp;jsessionid=fafdsfsd!-132!-12132?menu=1"
scrolling="no" marginwidth="0" marginheight="0"
noresize=""/>
                     <frame name="ffblogo"
src="../diverse/ffblogoKunde.jsp;jsessionid=fafdsfsd!-132!-12132"
scrolling="no" marginwidth="0" marginheight="0"
noresize=""/>
                     <frame name="keepSessionAlive"
src="../images/spacer.gif" scrolling="no"
marginwidth="0"
marginheight="0" noresize=""/>
                 </frameset>
                 <frame name="main"
src="../action/depotuebersichtinit;jsessionid=fafdsfsd!-132!-12132"
scrolling="auto" marginwidth="0" marginheight="0"
noresize=""/>
         </frameset>
                     <noframes>
                     </noframes>
</frameset>
</html>


log of mechanize (time stamps removed)
?????????????????????????????????????
# Logfile created on Tue Nov 07 09:03:04 (MEZ) Mitteleurop?ische Zeit 2006  
by logger.rb/1.5.2.7
  INFO -- : Net::HTTP::Get:  
https://www.frankfurter-fondsbank.de/login/Logon.jsp
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => Mozilla/5.0 (Windows; U; Windows  
NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: content-type => text/html; charset=ISO-8859-1
DEBUG -- : response-header: x-powered-by => Servlet/2.4 JSP/2.0
DEBUG -- : response-header: date => Tue, 07 Nov 2006 08:03:07 GMT
DEBUG -- : response-header: server => Apache
DEBUG -- : response-header: content-length => 4253
  INFO -- : status: 200
DEBUG -- : query: "login=10001&pin=2002"
  INFO -- : Net::HTTP::Post:  
https://www.frankfurter-fondsbank.de/login_neutral;jsessionid=CdEfG!-34567!-7654
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => Mozilla/5.0 (Windows; U; Windows  
NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6
DEBUG -- : request-header: content-type =>  
application/x-www-form-urlencoded
DEBUG -- : request-header: referer =>  
https://www.frankfurter-fondsbank.de/login/Logon.jsp
DEBUG -- : request-header: content-length => 28
DEBUG -- : response-header: cache-control => no-cache, no-store, max-age=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 01 Jan 1970 00:00:00 GMT
DEBUG -- : response-header: content-type => text/html
DEBUG -- : response-header: x-powered-by => Servlet/2.4 JSP/2.0
DEBUG -- : response-header: date => Tue, 07 Nov 2006 08:03:10 GMT
DEBUG -- : response-header: server => Apache
DEBUG -- : response-header: content-length => 2722
DEBUG -- : response-header: pragma => no-cache
  INFO -- : status: 200
*** INFO -- : Net::HTTP::Get:  
https://www.frankfurter-fondsbank.de/../diverse/navigation.jsp;jsessionid=CdEfG!-34567!-7654?menu=1
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => Mozilla/5.0 (Windows; U; Windows  
NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6
*** DEBUG -- : request-header: referer =>  
https://www.frankfurter-fondsbank.de/login_neutral;jsessionid=CdEfG!-34567!-7654
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: content-type => text/html; charset=iso-8859-1
DEBUG -- : response-header: date => Tue, 07 Nov 2006 08:03:13 GMT
DEBUG -- : response-header: server => Apache
DEBUG -- : response-header: content-length => 304
  INFO -- : status: 400



log of Firefox with LiveHTTPHeaders (partialy only, it is realy long)
??????????????????????????????????????????????????????????????????????
https://www.frankfurter-fondsbank.de/login/Logon.jsp

GET /login/Logon.jsp HTTP/1.1
Host: www.frankfurter-fondsbank.de
User-Agent: Mozilla/5.0 (Windows; U; Win98; de; rv:1.8.0.1) Gecko/20060111  
Firefox/1.5.0.1
Accept:  
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

HTTP/1.x 200 OK
Date: Tue, 07 Nov 2006 19:57:49 GMT
Server: Apache
Content-Length: 4253
X-Powered-By: Servlet/2.4 JSP/2.0
Connection: close
Content-Type: text/html; charset=ISO-8859-1
----------------------------------------------------------
https://www.frankfurter-fondsbank.de/login_neutral;jsessionid=AbCdE!-1234!-5678

POST /login_neutral;jsessionid=AbCdE!-1234!-5678 HTTP/1.1
Host: www.frankfurter-fondsbank.de
User-Agent: Mozilla/5.0 (Windows; U; Win98; de; rv:1.8.0.1) Gecko/20060111  
Firefox/1.5.0.1
Accept:  
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: https://www.frankfurter-fondsbank.de/login/Logon.jsp
Content-Type: application/x-www-form-urlencoded
Content-Length: 28
login=10001&pin=2002
HTTP/1.x 200 OK
Date: Tue, 07 Nov 2006 19:58:17 GMT
Server: Apache
Cache-Control: no-cache, no-store, max-age=0
Pragma: no-cache
Content-Length: 2722
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Powered-By: Servlet/2.4 JSP/2.0
Connection: close
Content-Type: text/html
----------------------------------------------------------
https://www.frankfurter-fondsbank.de/brokerdesign/default/head.jsp;jsessionid=AbCdE!-1234!-5678

GET /brokerdesign/default/head.jsp;jsessionid=AbCdE!-1234!-5678 HTTP/1.1
Host: www.frankfurter-fondsbank.de
User-Agent: Mozilla/5.0 (Windows; U; Win98; de; rv:1.8.0.1) Gecko/20060111  
Firefox/1.5.0.1
Accept:  
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer:  
https://www.frankfurter-fondsbank.de/login_neutral;jsessionid=AbCdE!-1234!-5678

HTTP/1.x 200 OK
Date: Tue, 07 Nov 2006 19:58:19 GMT
Server: Apache
Content-Length: 5052
X-Powered-By: Servlet/2.4 JSP/2.0
Connection: close
Content-Type: text/html; charset=ISO-8859-1
----------------------------------------------------------
https://www.frankfurter-fondsbank.de/diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1

GET /diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1 HTTP/1.1
Host: www.frankfurter-fondsbank.de
User-Agent: Mozilla/5.0 (Windows; U; Win98; de; rv:1.8.0.1) Gecko/20060111  
Firefox/1.5.0.1
Accept:  
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer:  
https://www.frankfurter-fondsbank.de/login_neutral;jsessionid=AbCdE!-1234!-5678

HTTP/1.x 200 OK
Date: Tue, 07 Nov 2006 19:58:19 GMT
Server: Apache
Content-Length: 6984
X-Powered-By: Servlet/2.4 JSP/2.0
Connection: close
Content-Type: text/html
----------------------------------------------------------
...

...
----------------------------------------------------------
https://www.frankfurter-fondsbank.de/images/buttons/but_sitemap.gif

GET /images/buttons/but_sitemap.gif HTTP/1.1
Host: www.frankfurter-fondsbank.de
User-Agent: Mozilla/5.0 (Windows; U; Win98; de; rv:1.8.0.1) Gecko/20060111  
Firefox/1.5.0.1
Accept: image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer:  
https://www.frankfurter-fondsbank.de/brokerdesign/default/head.jsp;jsessionid=AbCdE!-1234!-5678

HTTP/1.x 200 OK
Date: Tue, 07 Nov 2006 19:59:06 GMT
Server: Apache
Last-Modified: Tue, 04 May 2004 07:25:42 GMT
Etag: "18e0a6c-16c-37a3c980"
Accept-Ranges: bytes
Content-Length: 364
Connection: close
Content-Type: image/gif
----------------------------------------------------------
https://www.frankfurter-fondsbank.de/diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1

*** GET /diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1  
HTTP/1.1
Host: www.frankfurter-fondsbank.de
User-Agent: Mozilla/5.0 (Windows; U; Win98; de; rv:1.8.0.1) Gecko/20060111  
Firefox/1.5.0.1
Accept:  
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

HTTP/1.x 200 OK
Date: Tue, 07 Nov 2006 19:59:24 GMT
Server: Apache
Content-Length: 6984
X-Powered-By: Servlet/2.4 JSP/2.0
Connection: close
Content-Type: text/html
----------------------------------------------------------
https://www.frankfurter-fondsbank.de/DesignServlet;jsessionid=AbCdE!-1234!-5678?element=stylesheet

GET /DesignServlet;jsessionid=AbCdE!-1234!-5678?element=stylesheet HTTP/1.1
Host: www.frankfurter-fondsbank.de
User-Agent: Mozilla/5.0 (Windows; U; Win98; de; rv:1.8.0.1) Gecko/20060111  
Firefox/1.5.0.1
Accept: text/css,*/*;q=0.1
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
*** Referer:  
https://www.frankfurter-fondsbank.de/diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1

HTTP/1.x 200 OK
Date: Tue, 07 Nov 2006 19:59:27 GMT
Server: Apache
Content-Length: 2303
X-Powered-By: Servlet/2.4 JSP/2.0
Connection: close
Content-Type: text/css
----------------------------------------------------------

Aaron Patterson

2006-Nov-08 22:59 UTC

head link

[Mechanize-users] mechanize: 400 Bad Request

On Tue, Nov 07, 2006 at 09:39:04PM +0100, Friedrich, Axel
wrote:> 
> Hello,
> 
> when trying to access a certain HTML-frame, I get:
> "in `request'': Unhandled response
(WWW::Mechanize::ResponseCodeError)"
> 
> and the page returns:  "400 Bad Request"
> 
> * Why?
> * How to solve this?
> 
> With browser, it works.
> 
> In the logs below, I marked 4 lines with "***", where I see
possible
> differences in the URI. But I don''t know, if this is the reason
for the
> malfuncion and how to fix this.
> 
> mechanize:
> Net::HTTP::Get:  
>
https://www.frankfurter-fondsbank.de/../diverse/navigation.jsp;jsessionid=CdEfG!-34567!-7654?menu=1
> 
> firefox-LiveHTTPHeaders:
> GET /diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1 HTTP/1.1
It looks like mechanize doesn''t support relative paths in links.  This
is a bug, and I''ll look in to fixing it.
> 
> 
> Thank you for any help!
> 
> Axel
> 
> axel ? friedrich ? _smail  AT   gmx ? de
-- 
Aaron Patterson
http://tenderlovemaking.com/

Aaron Patterson

2006-Nov-13 08:05 UTC

head link

[Mechanize-users] mechanize: 400 Bad Request

On Wed, Nov 08, 2006 at 02:59:40PM -0800, Aaron Patterson
wrote:> On Tue, Nov 07, 2006 at 09:39:04PM +0100, Friedrich, Axel wrote:
> > 
> > Hello,
> > 
> > when trying to access a certain HTML-frame, I get:
> > "in `request'': Unhandled response
(WWW::Mechanize::ResponseCodeError)"
> > 
> > and the page returns:  "400 Bad Request"
> > 
> > * Why?
> > * How to solve this?
> > 
> > With browser, it works.
> > 
> > In the logs below, I marked 4 lines with "***", where I see
possible
> > differences in the URI. But I don''t know, if this is the
reason for the
> > malfuncion and how to fix this.
> > 
> > mechanize:
> > Net::HTTP::Get:  
> >
https://www.frankfurter-fondsbank.de/../diverse/navigation.jsp;jsessionid=CdEfG!-34567!-7654?menu=1
> > 
> > firefox-LiveHTTPHeaders:
> > GET /diverse/navigation.jsp;jsessionid=AbCdE!-1234!-5678?menu=1
HTTP/1.1
> 
> It looks like mechanize doesn''t support relative paths in links. 
This
> is a bug, and I''ll look in to fixing it.
Okay, I''m stumped.  I thought this was a bug in Mechanize, but
I''ve
written a few test cases to try to reproduce the error, and I can''t!

Are you sure the following line is being executed?
> >    framePage = agent.click( page2.frames.text(''nav'')
)   # This works not
I copied the html you provided in to a file and tried to reproduce the error,
but mechanize handles the relative link correctly.  Would it be possible for
you to write a short test case that reproduces the error?

-- 
Aaron Patterson
http://tenderlovemaking.com/

Friedrich, Axel

2006-Nov-19 14:06 UTC

head link

[Mechanize-users] mechanize: 400 Bad Request

Hello,

finally, I found time to create a test case:

With browser, the following page shows up correctly:

http://cafriedrich.netfirms.com/root/dir1/firstpage.htm



With the "standard mechanize code", I get the before mentioned error:
----------------------------------------------------------------------

require ''rubygems''
require ''mechanize''
require ''logger''
require ''fileutils''

# ========== CONFIG ==================================@logFilePath =
"G:/Ruby/FrankfurterFondsBank/downloaded/log.txt"
@downloadDir = ''G:/Ruby/FrankfurterFondsBank/downloaded/''
@firstPage = "http://cafriedrich.netfirms.com/root/dir1/firstpage.htm"
# ========== END CONFIG ==============================

FileUtils.mkpath( File.dirname(@logFilePath) )
FileUtils.mkpath( @downloadDir )

agent = WWW::Mechanize.new{|a| a.log = Logger.new(@logFilePath) }

page = agent.get(@firstPage)
txt = page.body
File.open("@downloadDir" + "firstpage.htm",
''w'') {|f|  f.write(txt)  }

puts "\n\n================= frames:=================  "
page.frames.each { |f| puts sprintf("%-15s %-1s \n", f.text, f.href) }
puts "\n=================  /frames =================  \n\n"

begin
   mainframe = agent.click( page.frames.text(''main'') )
   txt = mainframe.body
   File.open(@downloadDir + "mainframe.htm", ''w'')
{|f|  f.write(txt)  }

   frameleft1 = agent.click( page.frames.text(''left1'') )
   txt = page.body
   File.open(@downloadDir + "left1.htm", ''w'') {|f| 
f.write(txt)  }
rescue WWW::Mechanize::ResponseCodeError => ex
   puts "====== ERROR =========="
   puts ex.page.body
   puts "====== /ERROR =========="
end

puts "-- Exiting now."



With the "workaround code" below, it works:
-------------------------------------------

require ''rubygems''
require ''mechanize''
require ''logger''
require ''fileutils''

# ========== CONFIG ==================================@logFilePath =
"G:/Ruby/FrankfurterFondsBank/downloaded/log.txt"
@downloadDir = ''G:/Ruby/FrankfurterFondsBank/downloaded/''
@firstPage = "http://cafriedrich.netfirms.com/root/dir1/firstpage.htm"
# ========== END CONFIG ==============================

FileUtils.mkpath( File.dirname(@logFilePath) )
FileUtils.mkpath( @downloadDir )

agent = WWW::Mechanize.new{|a| a.log = Logger.new(@logFilePath) }

page = agent.get(@firstPage)
txt = page.body
File.open("@downloadDir" + "firstpage.htm",
''w'') {|f|  f.write(txt)  }

puts "\n\n================= frames:=================  "
page.frames.each { |f| puts sprintf("%-15s %-1s \n", f.text, f.href) }
puts "\n=================  /frames =================  \n\n"

begin
   mainframe = agent.click( page.frames.text(''main'') )
   txt = mainframe.body
   File.open(@downloadDir + "mainframe.htm", ''w'')
{|f|  f.write(txt)  }

   href = ''http://cafriedrich.netfirms.com/root/''        #
<-  WORKAROUND
   href <<
page.frames.text(''left1'').href.sub(''../'',
'''') # <-  WORKAROUND
   frameleft1 = agent.get(href)                          # <-  WORKAROUND
   txt = page.body
   File.open(@downloadDir + "left1.htm", ''w'') {|f| 
f.write(txt)  }
rescue WWW::Mechanize::ResponseCodeError => ex
   puts "====== ERROR =========="
   puts ex.page.body
   puts "====== /ERROR =========="
end

puts "-- Exiting now."

Aaron Patterson

2006-Nov-20 19:31 UTC

head link

[Mechanize-users] mechanize: 400 Bad Request

On Sun, Nov 19, 2006 at 03:06:17PM +0100, Friedrich, Axel
wrote:> 
> Hello,
> 
> finally, I found time to create a test case:
Thank you!  I understand the bug now.  When mechanize sees that the url
is relative, it looks at the last url in its history to determine where
to go next.  In the case of frames, that is not correct, since your
request flow is not linear.  I''ll fix this issue for the next release.

Here''s my workaround where the rescue block never gets called:

  require ''rubygems''
  require ''mechanize''
  
  # ========== CONFIG ==================================  @firstPage =
"http://cafriedrich.netfirms.com/root/dir1/firstpage.htm"
  # ========== END CONFIG ==============================  
  agent = WWW::Mechanize.new
  page = agent.get(@firstPage)
  
  begin
     mainframe = agent.click( page.frames.text(''main'') )
     agent.history.pop  # <= This makes it work!
     frameleft1 = agent.click( page.frames.text(''left1'') )
  rescue WWW::Mechanize::ResponseCodeError => ex
     puts "Error"
  end
  
  puts "-- Exiting now."

-- 
Aaron Patterson
http://tenderlovemaking.com/

Friedrich, Axel

2006-Nov-21 17:29 UTC

head link

[Mechanize-users] mechanize: 400 Bad Request

> I''ll fix this issue for the next release.
> Here''s my workaround
Thank you!

- Axel

Maybe Matching Threads

Search for more apparently analagous threads

Mechanize users - Nov 2006 - mechanize: 400 Bad Request

[Mechanize-users] mechanize: 400 Bad Request

[Mechanize-users] mechanize: 400 Bad Request

[Mechanize-users] mechanize: 400 Bad Request

[Mechanize-users] mechanize: 400 Bad Request

[Mechanize-users] mechanize: 400 Bad Request

[Mechanize-users] mechanize: 400 Bad Request

Maybe Matching Threads