Josh Stratton
2013-May-22 23:22 UTC
[Mechanize-users] different page from ruby''s mechanize than browser
My friend wrote a script, which uses ruby''s mechanize to get a cookie
off a
page after login.
* start mechanize
* go through login page form
* get cookie value from page delivered by form
The script seems to work for him even on my machine, but doesn''t for
me.
I''ve tried doing this interactively and I noticed I get a different
page
title returned from Mechanize than the one for my browser. When I log in,
the page title is "SSL VPN - Home", but when I look at the title
attribute
on my returned submission object I get "Please wait..."
irb(main):084:0> intermediate.title
=> "Please wait..."
irb(main):085:0> intermediate.iframes
=> [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
Is Mechanize possibly not waiting long enough to get the destination page?
The "Please wait..." seems to hint at some intermediate page that gets
refreshed or redirected that the Mechanize session isn''t getting to.
Either
way I don''t get the same title value.
Am I at some intermediate page my browser redirects from that mechanize
gets stuck at?
=> #<Mechanize::Page
{url
#<URI::HTTPS:0x00000002b49338 URL:
https://vpn1.example.com/dana/home/starter0.cgi?check=yes>}
{meta_refresh}
{title "Please wait..."}
{iframes #<Mechanize::Page::Frame nil
"/dana-na/html/blank.html">}
{frames}
{links}
{forms
#<Mechanize::Form
{name "frmGrab"}
{method "POST"}
{action "/dana/home/starter0.cgi"}
{fields
[hidden:0x15b9860 type: hidden name: xsauth value:
a0395604a9f4f531504a7f6b6dc86051]
[hidden:0x15b94a0 type: hidden name: tz_offset value: ]
[hidden:0x15b92e8 type: hidden name: clienttime value: ]
[hidden:0x15b8ff0 type: hidden name: url value: ]
[hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0]
[hidden:0x15b8898 type: hidden name: java_enabled value: 0]
[hidden:0x15b8690 type: hidden name: power_user value: 0]
[hidden:0x15b84ec type: hidden name: grab value: 1]
[hidden:0x15b8348 type: hidden name: browserproxy value: ]
[hidden:0x15b8168 type: hidden name: browsertype value: ]
[hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ]
[hidden:0x15b7df8 type: hidden name: check value: yes]
[hidden:0x15b7c54 type: hidden name: nextpage value: ]
[hidden:0x15bb8f4 type: hidden name: mid value: ]
[hidden:0x15bb5d4 type: hidden name: signin value: ]
[hidden:0x15bb2dc type: hidden name: alias value: ]
[hidden:0x15bb05c type: hidden name: id value: ]
[hidden:0x15baeb8 type: hidden name: username value: ]
[hidden:0x15bacb0 type: hidden name: password value: ]
[hidden:0x15bab20 type: hidden name: occurrence value: ]}
{radiobuttons}
{checkboxes}
{file_uploads}
{buttons}>}>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/mechanize-users/attachments/20130522/046e28de/attachment.html>
Timothy Snowhite
2013-May-23 00:12 UTC
[Mechanize-users] different page from ruby''s mechanize than browser
Use intermediate.body to see the HTML returned, and look for a meta-refresh element or a JavaScript redirect. On Wednesday, May 22, 2013, Josh Stratton wrote:> My friend wrote a script, which uses ruby''s mechanize to get a cookie off > a page after login. > > * start mechanize > * go through login page form > * get cookie value from page delivered by form > > The script seems to work for him even on my machine, but doesn''t for me. > I''ve tried doing this interactively and I noticed I get a different page > title returned from Mechanize than the one for my browser. When I log in, > the page title is "SSL VPN - Home", but when I look at the title attribute > on my returned submission object I get "Please wait..." > > irb(main):084:0> intermediate.title > => "Please wait..." > irb(main):085:0> intermediate.iframes > => [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> > Is Mechanize possibly not waiting long enough to get the destination page? > The "Please wait..." seems to hint at some intermediate page that gets > refreshed or redirected that the Mechanize session isn''t getting to. Either > way I don''t get the same title value. > > Am I at some intermediate page my browser redirects from that mechanize > gets stuck at? > > => #<Mechanize::Page > {url > #<URI::HTTPS:0x00000002b49338 URL: > https://vpn1.example.com/dana/home/starter0.cgi?check=yes>} > {meta_refresh} > {title "Please wait..."} > {iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">} > {frames} > {links} > {forms > #<Mechanize::Form > {name "frmGrab"} > {method "POST"} > {action "/dana/home/starter0.cgi"} > {fields > [hidden:0x15b9860 type: hidden name: xsauth value: > a0395604a9f4f531504a7f6b6dc86051] > [hidden:0x15b94a0 type: hidden name: tz_offset value: ] > [hidden:0x15b92e8 type: hidden name: clienttime value: ] > [hidden:0x15b8ff0 type: hidden name: url value: ] > [hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0] > [hidden:0x15b8898 type: hidden name: java_enabled value: 0] > [hidden:0x15b8690 type: hidden name: power_user value: 0] > [hidden:0x15b84ec type: hidden name: grab value: 1] > [hidden:0x15b8348 type: hidden name: browserproxy value: ] > [hidden:0x15b8168 type: hidden name: browsertype value: ] > [hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ] > [hidden:0x15b7df8 type: hidden name: check value: yes] > [hidden:0x15b7c54 type: hidden name: nextpage value: ] > [hidden:0x15bb8f4 type: hidden name: mid value: ] > [hidden:0x15bb5d4 type: hidden name: signin value: ] > [hidden:0x15bb2dc type: hidden name: alias value: ] > [hidden:0x15bb05c type: hidden name: id value: ] > [hidden:0x15baeb8 type: hidden name: username value: ] > [hidden:0x15bacb0 type: hidden name: password value: ] > [hidden:0x15bab20 type: hidden name: occurrence value: ]} > {radiobuttons} > {checkboxes} > {file_uploads} > {buttons}>}> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20130522/bb7aa160/attachment.html>
Josh Stratton
2013-May-23 16:12 UTC
[Mechanize-users] different page from ruby''s mechanize than browser
Looking at the html, I don''t see a javascript redirect or meta-refresh. http://pastebin.com/4SPArnUR I do see a reference to an iframe, but I''m not sure if it''s related. I''m not sure how to use it either. irb(main):114:0> intermediate.iframes.length => 1 irb(main):112:0> page = intermediate.iframes.first => #<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> irb(main):116:0> page.class => Mechanize::Page::Frame irb(main):117:0> page.links NoMethodError: undefined method `links'' for #<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> from (irb):117 from /usr/bin/irb:12:in `<main>'' I don''t see any relevant data on it. I''m confused because the functions I see in the docs like "links" don''t seem to exist on it. On Wed, May 22, 2013 at 5:12 PM, Timothy Snowhite <snowhitster at gmail.com>wrote:> Use intermediate.body to see the HTML returned, and look for a > meta-refresh element or a JavaScript redirect. > > > On Wednesday, May 22, 2013, Josh Stratton wrote: > >> My friend wrote a script, which uses ruby''s mechanize to get a cookie off >> a page after login. >> >> * start mechanize >> * go through login page form >> * get cookie value from page delivered by form >> >> The script seems to work for him even on my machine, but doesn''t for me. >> I''ve tried doing this interactively and I noticed I get a different page >> title returned from Mechanize than the one for my browser. When I log in, >> the page title is "SSL VPN - Home", but when I look at the title attribute >> on my returned submission object I get "Please wait..." >> >> irb(main):084:0> intermediate.title >> => "Please wait..." >> irb(main):085:0> intermediate.iframes >> => [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> >> Is Mechanize possibly not waiting long enough to get the destination >> page? The "Please wait..." seems to hint at some intermediate page that >> gets refreshed or redirected that the Mechanize session isn''t getting to. >> Either way I don''t get the same title value. >> >> Am I at some intermediate page my browser redirects from that mechanize >> gets stuck at? >> >> => #<Mechanize::Page >> {url >> #<URI::HTTPS:0x00000002b49338 URL: >> https://vpn1.example.com/dana/home/starter0.cgi?check=yes>} >> {meta_refresh} >> {title "Please wait..."} >> {iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">} >> {frames} >> {links} >> {forms >> #<Mechanize::Form >> {name "frmGrab"} >> {method "POST"} >> {action "/dana/home/starter0.cgi"} >> {fields >> [hidden:0x15b9860 type: hidden name: xsauth value: >> a0395604a9f4f531504a7f6b6dc86051] >> [hidden:0x15b94a0 type: hidden name: tz_offset value: ] >> [hidden:0x15b92e8 type: hidden name: clienttime value: ] >> [hidden:0x15b8ff0 type: hidden name: url value: ] >> [hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0] >> [hidden:0x15b8898 type: hidden name: java_enabled value: 0] >> [hidden:0x15b8690 type: hidden name: power_user value: 0] >> [hidden:0x15b84ec type: hidden name: grab value: 1] >> [hidden:0x15b8348 type: hidden name: browserproxy value: ] >> [hidden:0x15b8168 type: hidden name: browsertype value: ] >> [hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ] >> [hidden:0x15b7df8 type: hidden name: check value: yes] >> [hidden:0x15b7c54 type: hidden name: nextpage value: ] >> [hidden:0x15bb8f4 type: hidden name: mid value: ] >> [hidden:0x15bb5d4 type: hidden name: signin value: ] >> [hidden:0x15bb2dc type: hidden name: alias value: ] >> [hidden:0x15bb05c type: hidden name: id value: ] >> [hidden:0x15baeb8 type: hidden name: username value: ] >> [hidden:0x15bacb0 type: hidden name: password value: ] >> [hidden:0x15bab20 type: hidden name: occurrence value: ]} >> {radiobuttons} >> {checkboxes} >> {file_uploads} >> {buttons}>}> >> > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20130523/58d5b441/attachment-0001.html>
Godfrey Chan
2013-May-23 16:18 UTC
[Mechanize-users] different page from ruby''s mechanize than browser
there''s a unload handler attached to the body which then perform a bunch of crazy stuff, and then finally submit a form which causes the page to navigate away. You''ll either have to reverse engineer that, or use a headless browser with javascript support.? Godfrey ? Sent from Mailbox for iPhone On Thu, May 23, 2013 at 9:13 AM, Josh Stratton <strattonbrazil at gmail.com> wrote:> Looking at the html, I don''t see a javascript redirect or meta-refresh. > http://pastebin.com/4SPArnUR > I do see a reference to an iframe, but I''m not sure if it''s related. I''m > not sure how to use it either. > irb(main):114:0> intermediate.iframes.length > => 1 > irb(main):112:0> page = intermediate.iframes.first > => #<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> > irb(main):116:0> page.class > => Mechanize::Page::Frame > irb(main):117:0> page.links > NoMethodError: undefined method `links'' for #<Mechanize::Page::Frame nil > "/dana-na/html/blank.html"> > from (irb):117 > from /usr/bin/irb:12:in `<main>'' > I don''t see any relevant data on it. I''m confused because the functions I > see in the docs like "links" don''t seem to exist on it. > On Wed, May 22, 2013 at 5:12 PM, Timothy Snowhite <snowhitster at gmail.com>wrote: >> Use intermediate.body to see the HTML returned, and look for a >> meta-refresh element or a JavaScript redirect. >> >> >> On Wednesday, May 22, 2013, Josh Stratton wrote: >> >>> My friend wrote a script, which uses ruby''s mechanize to get a cookie off >>> a page after login. >>> >>> * start mechanize >>> * go through login page form >>> * get cookie value from page delivered by form >>> >>> The script seems to work for him even on my machine, but doesn''t for me. >>> I''ve tried doing this interactively and I noticed I get a different page >>> title returned from Mechanize than the one for my browser. When I log in, >>> the page title is "SSL VPN - Home", but when I look at the title attribute >>> on my returned submission object I get "Please wait..." >>> >>> irb(main):084:0> intermediate.title >>> => "Please wait..." >>> irb(main):085:0> intermediate.iframes >>> => [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> >>> Is Mechanize possibly not waiting long enough to get the destination >>> page? The "Please wait..." seems to hint at some intermediate page that >>> gets refreshed or redirected that the Mechanize session isn''t getting to. >>> Either way I don''t get the same title value. >>> >>> Am I at some intermediate page my browser redirects from that mechanize >>> gets stuck at? >>> >>> => #<Mechanize::Page >>> {url >>> #<URI::HTTPS:0x00000002b49338 URL: >>> https://vpn1.example.com/dana/home/starter0.cgi?check=yes>} >>> {meta_refresh} >>> {title "Please wait..."} >>> {iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">} >>> {frames} >>> {links} >>> {forms >>> #<Mechanize::Form >>> {name "frmGrab"} >>> {method "POST"} >>> {action "/dana/home/starter0.cgi"} >>> {fields >>> [hidden:0x15b9860 type: hidden name: xsauth value: >>> a0395604a9f4f531504a7f6b6dc86051] >>> [hidden:0x15b94a0 type: hidden name: tz_offset value: ] >>> [hidden:0x15b92e8 type: hidden name: clienttime value: ] >>> [hidden:0x15b8ff0 type: hidden name: url value: ] >>> [hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0] >>> [hidden:0x15b8898 type: hidden name: java_enabled value: 0] >>> [hidden:0x15b8690 type: hidden name: power_user value: 0] >>> [hidden:0x15b84ec type: hidden name: grab value: 1] >>> [hidden:0x15b8348 type: hidden name: browserproxy value: ] >>> [hidden:0x15b8168 type: hidden name: browsertype value: ] >>> [hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ] >>> [hidden:0x15b7df8 type: hidden name: check value: yes] >>> [hidden:0x15b7c54 type: hidden name: nextpage value: ] >>> [hidden:0x15bb8f4 type: hidden name: mid value: ] >>> [hidden:0x15bb5d4 type: hidden name: signin value: ] >>> [hidden:0x15bb2dc type: hidden name: alias value: ] >>> [hidden:0x15bb05c type: hidden name: id value: ] >>> [hidden:0x15baeb8 type: hidden name: username value: ] >>> [hidden:0x15bacb0 type: hidden name: password value: ] >>> [hidden:0x15bab20 type: hidden name: occurrence value: ]} >>> {radiobuttons} >>> {checkboxes} >>> {file_uploads} >>> {buttons}>}> >>> >> >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20130523/92892f6d/attachment.html>
Timothy Snowhite
2013-May-23 16:49 UTC
[Mechanize-users] different page from ruby''s mechanize than browser
I believe intermediate.iframes.first.click will load the iframe for you. It might help to run Charles (a debugging proxy) while you log in to the site via a browser to see what all requests get sent throughout the login process. On Thursday, May 23, 2013, Godfrey Chan wrote:> there''s a unload handler attached to the body which then perform a bunch > of crazy stuff, and then finally submit a form which causes the page to > navigate away. > > You''ll either have to reverse engineer that, or use a headless browser > with javascript support. > > Godfrey > ? > Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone > > > On Thu, May 23, 2013 at 9:13 AM, Josh Stratton <strattonbrazil at gmail.com>wrote: > > Looking at the html, I don''t see a javascript redirect or meta-refresh. > > http://pastebin.com/4SPArnUR > > I do see a reference to an iframe, but I''m not sure if it''s related. I''m > not sure how to use it either. > > irb(main):114:0> intermediate.iframes.length > => 1 > > irb(main):112:0> page = intermediate.iframes.first > => #<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> > > irb(main):116:0> page.class > => Mechanize::Page::Frame > irb(main):117:0> page.links > > NoMethodError: undefined method `links'' for #<Mechanize::Page::Frame nil > "/dana-na/html/blank.html"> > > from (irb):117 > from /usr/bin/irb:12:in `<main>'' > > I don''t see any relevant data on it. I''m confused because the functions I > see in the docs like "links" don''t seem to exist on it. > > > > > On Wed, May 22, 2013 at 5:12 PM, Timothy Snowhite <snowhitster at gmail.com>wrote: > > Use intermediate.body to see the HTML returned, and look for a > meta-refresh element or a JavaScript redirect. > > > On Wednesday, May 22, 2013, Josh Stratton wrote: > > My friend wrote a script, which uses ruby''s mechanize to get a cookie off > a page after login. > > * start mechanize > * go through login page form > * get cookie value from page delivered by form > > The script seems to work for him even on my machine, but doesn''t for me. > I''ve tried doing this interactively and I noticed I get a different page > title returned from Mechanize than the one for my browser. When I log in, > the page title is "SSL VPN - Home", but when I look at the title attribute > on my returned submission object I get "Please wait..." > > irb(main):084:0> intermediate.title > => "Please wait..." > irb(main):085:0> intermediate.iframes > => [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html"> > Is Mechanize possibly not waiting long enough to get the destination page? > The "Please wait..." seems to hint at some intermediate page that gets > refreshed or redirected that the Mechanize session isn''t getting to. Either > way I don''t get the same title value. > > Am I at some intermediate page my browser redirects from that mechanize > gets stuck at? > > => #<Mechanize::Page > {url > #<URI::HTTPS:0x00000002b49338 URL: > https://vpn1.example.com/dana/home/starter0.cgi?check=yes>} > {meta_refresh} > {title "Please wait..."} > {iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">} > {frames} > {links} > {forms > #<Mechanize::Form > {name "frmGrab"} > {method "POST"} > {action "/dana/home/starter0.cgi"} > {fields > [hidden:0x15b9860 type: hidden name: xsauth value: > a0395604a9f4f531504a7f6b6dc86051] > [hidden:0x15b94a0 type: hidden name: tz_offset value: ] > [hidden:0x15b92e8 type: hidden name: clienttime value: ] > [hidden:0x15b8ff0 type: hidden name: url value: ] > [hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0] > [hidden:0x15b8898 type: hidden name: java_enabled value: 0] > [hidden:0x15b8690 type: hidden name: power_user value: 0] > [hidden:0x15b84ec type: hidden name: grab value: 1] > [hidden:0x15b8348 type: hidden name: browserproxy value: ] > [hidden:0x15b8168 > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20130523/e133c436/attachment.html>