similar to: Only partially reading a page!

All, Another easy question. In Hpricot, on a doc that I am using, I can do a .to_html method and retrieve the entire page. However, this doesn''t seem to work in Mechanize. My goal is to the text of the page and put it into a database to make it searchable with ferret (using the acts_as_ferret plugin in Rails). Does anyone have a good suggestion short of iterating over the entire

Hey all! Just to preface, I am fairly new to RoR, and brand new to using hpricot. I am using the following code to scrape this xpath: "/html/body/div/div[5]/div/div[2]/div[2]/div[2]" from this url: "http://www.greatnonprofits.org/" Here is my code to do so (taken from igvita.com''s related blogpost): ************* require ''rubygems'' require

hi - i have strings that i need to extract keywords from. the string might have html tags, urls, etc. i need to extract the keywords from the string. i imagine i''m not the first guy to have to tackle this problem. is there a gem i can use or anyone have any ideas how to approach this? thanks, dino

Hey all, I want to scrap some content from a website to rails application and wants to authenticate the url of the website which I want to scarp so please help me out.... Thank You!!! -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to

HI, I want to grab some information about university names, and I found this term called "web scraping" I search about it in google, and there are tools in ruby. One of them is nokogiri but I''m a bit confused because it seems that it only gets information that its already in an html or xml I found a webpage that have a list of university names as a <select>

require ''rubygems'' require ''hpricot'' require ''open-uri'' doc = Hpricot(open("http://graph.facebook.com/226723089703")) @doc = doc.to_s.gsub(/"id.+?likes":/,"").gsub("{","").gsub("}","") puts @doc OR require ''rubygems''

Hello to the list and thanks to Aaron for the cool software. I''ve been fooling around with Mechanize and Hpricot for a couple of days and from the docs I''ve read, the following code SHOULD work but doesn''t. I''ve tried the same code on a couple of different sites and I get the same exception for each. Any pointers or suggestions are appreciated.

Is there a decent way to add a field to a form before posting it? I haven''t tried using HPricot manipulations just yet, since I can''t ever find really solid docs on hpricot.... Form#[]= doesn''t work because it first searches only pre-existing fields. I''m investigating how to write a patch now. But I thought maybe someone here might have an idea.

I am new to Mechanize and was wondering if there was a built-in method to get the elements that are on the page that are not part of a form. A couple of examples would be my banking site lists my entries and I want them to go into an array so that I can handle them. Or another site I use, does some categorization for me and I would like to manipulate it and present it differently to a user.

On Sat, Jul 25, 2009 at 9:14 PM, Jeffrey Roberts<jeffrey.l.roberts at gmail.com> wrote: > Hello all, I have looked up and down on google for a solution to this going > on several days now, I am really hoping someone here can help me out. > > I have all my deps in order, and I believe the error is that it is looking > in /usr/lib when it should be looking in /usr/lib64, I am

Hey everyone, I''ve been refactoring Mechanize for an 0.7.0 release. Basically I''m trying to clean the code up and there are a few features that I think are unnecessary, but I would like to ask people first. 1) REXML as a parser. I want to remove support for REXML. I don''t use it. Hpricot seems to do everything I need. 2) 1.8.2 thru 1.8.4 support

I knew that hpricot was a dependency, but hoe and rubyforge....is this right? Version 0.6.9 for ruby Mike B. ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

Hi. Recently I started rebuilding my old Mechanize script, which I used to automatically log in to a certain site and retrieve files from it. Old version worked great, however, when I did the update it started complaining. Here''s the log of the error: /Users/lukastolyarov/.gem/ruby/1.8/gems/mechanize-1.0.0/lib/mechanize/form/field.rb:30:in `<=>'': undefined method

Hi All I''m trying to build an application which requires to scrap information from a webpage. On trying to perform the action, I get an error while trying to convert the html data to JSON. Has anyone experienced this before and if so can you please tell me how to solve this problem ? Please see below for code snippet and error log. Thanks in advance Anush require

mechanize version 0.6.6 has been released! http://mechanize.rubyforge.org/ The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. Changes: =

Hi, Anyone know a way to traverse/query the DOM in a response? I''m trying to write reusable story step implementations and as much as possible want to work with the actual response from a previous GET. As an example, I often have buttons that POST/PUT a hidden value. My goal is a reusable step implementation that takes the button label and works the existing response DOM to make an

Hello all, I am trying to click image maps ("area" nodes), and while Perl''s mechanize treats "a", "area", "frame", "iframe" and "meta" tags as links, it seems to me that Ruby''s Mechanize thinks only "a" tags can be links (at least my little experiment, namely: page.links.each {|link| puts link.node.name}

Hi, I''m getting a "Could not open any of [xml2, xslt, exslt] (LoadError)" error when trying to run a simple Ruby program taken from the EXAMPLES.rdoc file of the Mechanize gem. The error is in this line of the Nokogiri module of libxml.rb: ffi_lib ''xml2'', ''xslt'', ''exslt'' Not sure if there are missing gems, and if so,

Hi This may be a dumb question with an obvious answer. It would seem that an input form field identified with an ''id'' qualifier and not with a ''name'' qualifier is not recognised by Mechanize - at least it isn''t in the form field list. Is there any way of getting at these elements or am I, as I suspect, fresh out of luck. But you never know ...

Hi, I still have the following error while loading backgroundrb : /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/ packet-0.1.14/lib/packet/packet_master.rb:116:in `exec'': No such file or directory - packet_worker_runner 14:11:maintenance_worker:64:/home/ antz/declix.com/production/lib/workers (Errno::ENOENT) from