similar to: Single method call to retrieve the entire page in HTML?

Hi all, Trying to figure out a quick way to do something. I have a yahoo groups list, and it has listings that look like this: </td></tr> <tr><td colspan="2" class="ygrp-g"> <span class="ygrp-nowrap"> <em><a href="/group/freecycledc/">freecycledc</a></em> </span> <a

I am trying to get a page which includes a form, but the form is missing from the WWW::Mechanize::Page object. I retrieve it via: page = web_agent.submit(a_different_form) For debugging this problem, I then immediately write the resulting page to two different logs: File.open(''big.html'',''wb'') { |f| f.write(page.body) }

Hey everyone, I''ve been refactoring Mechanize for an 0.7.0 release. Basically I''m trying to clean the code up and there are a few features that I think are unnecessary, but I would like to ask people first. 1) REXML as a parser. I want to remove support for REXML. I don''t use it. Hpricot seems to do everything I need. 2) 1.8.2 thru 1.8.4 support

I am new to Mechanize and was wondering if there was a built-in method to get the elements that are on the page that are not part of a form. A couple of examples would be my banking site lists my entries and I want them to go into an array so that I can handle them. Or another site I use, does some categorization for me and I would like to manipulate it and present it differently to a user.

Hello to the list and thanks to Aaron for the cool software. I''ve been fooling around with Mechanize and Hpricot for a couple of days and from the docs I''ve read, the following code SHOULD work but doesn''t. I''ve tried the same code on a couple of different sites and I get the same exception for each. Any pointers or suggestions are appreciated.

I knew that hpricot was a dependency, but hoe and rubyforge....is this right? Version 0.6.9 for ruby Mike B. ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

Is there a decent way to add a field to a form before posting it? I haven''t tried using HPricot manipulations just yet, since I can''t ever find really solid docs on hpricot.... Form#[]= doesn''t work because it first searches only pre-existing fields. I''m investigating how to write a patch now. But I thought maybe someone here might have an idea.

All, Been a while since I''ve "mechanized" and I''m running into an idiot simple problem. I''m looking for more detailed examples to refresh my memory. My question is how to get a list of links off of a page based on information in the img src attribute, specifically if the img tag contains a "mp3.gif/jpg". I, for the life of me, can''t seem

mechanize version 0.6.6 has been released! http://mechanize.rubyforge.org/ The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. Changes: =

Hello all, I am trying to click image maps ("area" nodes), and while Perl''s mechanize treats "a", "area", "frame", "iframe" and "meta" tags as links, it seems to me that Ruby''s Mechanize thinks only "a" tags can be links (at least my little experiment, namely: page.links.each {|link| puts link.node.name}

Hi, I''m using Mechanize, and I''ve developed a lot of code around it. I''d like to be able to check the Etag header during a get to see if the page has changed, as well as some other http header information. Can I do that without hacking Mechanize myself? Does anyone have any examples of how to do this? William

Greetings, I''m using Mechanize to scrap dozens of pages and have noticed the size of my ruby process keeps growing. I set Mechanize.max_history to 0 with no effect on the memory use. I wrote a little test to show the objects left on the heap after mechanizing a single page and then doing a garbage collection. Sample list appended below. I can supply the test code if it helps.

version 0.62 I''ve got a form that''s giving me trouble (server reports invalid session ID). Unfortunately, the form is submitted using SSL so I can''t use a sniffer to see the differences between the ruby mechanize version and the perl mechanize version that is working. Is there a way to change the form action within mechanize? Or a way to display the request headers?

I got this in one of my pages, and I''m not exactly sure what it means. ideas? Mike B. ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

Hi, Does anyone have the formula for getting logged into LinkedIn? Here''s my current attempt: require ''rubygems'' require ''mechanize'' agent = WWW::Mechanize.new home_page = agent.get(''http://www.linkedin.com'') signin_page = agent.click home_page.links.text(''Sign in'') puts "\nSIGNIN PAGE"

I poked around the web a little and didn''t run across how to use https when it asks for certificate validation. I''m trying to connect to devices that don''t have valid certificates, and in this case, I don''t care if they are or not. So when I use my browser to ge to the site, firefox asks me to allow the certificate, then one other question, then I get the

I have a Watir script that I would like to convert to a Mechanize script. The watir code looks like this require ''watir'' include Watir def test ie = IE.new ie.goto(''http://cpref.gsm.com/inter.asp?r=8084'') ie.text_field(:name, ''inter_string'').set(''Potassium'') ie.button(:id, ''image1'').click

Hello, I am using Mechanize to post a form to a website. When I do this by hand in my browser the response takes about 35s to come back (it''s a long page full of tables and graphics). When I do this with Mechanize, the server starts to respond and then appears to hang. The obvious conclusion is that my code is wrong but I am reasonably sure that I haven''t altered it

hi there, I''m not sure if it is better to use the ml or the forum so I ended to post on both ... I''m trying to login into a page that is using javascript to set a cookie: <script language="JavaScript"> {text "\n" " \n" " \n" " document.cookie = ''vhp_js=v995105608h85339294p;path=/;domain=.venere.com;expire=0'';

Hi all, I''ve been using mechanize for a while and it rocks. Docs are pretty clear and so far I''ve been able to do it on my own. However, I''m stuck in a weird situation in a script to download my contact list from hotmail. I''ve used Firebug to check all urls, and tested it by hand while logged in via browser. Even in the script everything works well until the