similar to: Does Amazon.com blocks scraping?

Displaying 20 results from an estimated 11000 matches similar to: "Does Amazon.com blocks scraping?"

2010 Jan 26
1
Does Amazon.com block scraping?
Hi there Does anyone know if Amazon.com has any sort of server side script that tries to block scraping activities? I first noticed that if I didn?t change the agent alias, it would fetch a page exactly like the normal one, but without the intial search field(maybe a silly way to prevent scraping). Then after it, I changed to some other alias, and submit a search. I got the result page as
2007 Apr 20
2
Running script does not return the correct page
Hello all, I have tried to post this yesterday, but noticed I was actually not subscribed yet... Well, here we go again: If I run this script, and observe the output, the results are not there at all (try to do the same in the browser). Any suggestions? require ''rubygems'' require ''mechanize'' agent = WWW::Mechanize.new agent.user_agent_alias =
2007 Apr 19
1
Do you have any idea what could be the problem with this script?
Hello all, If I run this script, and observe the output, the results are not there at all (try to do the same in the browser). Any suggestions? require ''rubygems'' require ''mechanize'' agent = WWW::Mechanize.new agent.user_agent_alias = ''Mac Safari'' page =
2008 Jan 15
1
Converting Watir script to Mechanize
I have a Watir script that I would like to convert to a Mechanize script. The watir code looks like this require ''watir'' include Watir def test ie = IE.new ie.goto(''http://cpref.gsm.com/inter.asp?r=8084'') ie.text_field(:name, ''inter_string'').set(''Potassium'') ie.button(:id, ''image1'').click
2007 Jul 12
1
WWW::Mechanize::Link.inspect needs some TLC
The problem: users trying to debug Mechanize apps with Komodo are finding the debugger times out once it''s loaded a web page. They don''t run into this in the ruby-debug debugger, or running in normal mode. The reason: Komodo''s debugger is graphical, which means that whenever it hits a breakpoint it automatically shows the contents of each local variable. It has a
2007 May 06
1
How to submit this form?
Hey all, maybe I am missing the obvious again, but I can not submit the form on buy.com: require ''rubygems'' require ''mechanize'' agent = WWW::Mechanize.new agent.user_agent_alias = ''Mac Safari'' page = agent.get("http://www.buy.com") search_form = page.forms.with.name("searchbox").first p search_form.buttons the last
2007 Jan 04
5
Mashup Tutorial
Hi: I am looking for a good hands on web mashup tutorial? or books for that matter.. Google gives me bits and pieces of the info but not any step by step or hands on tutorial.. Thanks a bundle -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk"
2007 May 20
1
Tips on testing
Hey WWW::Mechanize users, I''ve been using Mechanize for about 3 or 4 months now, and I''m rather enjoying it. I''ve done scraping in PHP & Python, starting with raw text and HTTP requests, moving up to regex and Curl, and finally landed myself here, making me much more productive when scraping pages. At the moment, I''m in the middle of a decent sized
2007 Oct 10
1
Scraping AOL Webmail to login and fetch contacts?
I''m helping with a gem that is going to published under the contentfree project on rubyforge (http://rubyforge.org/projects/contentfree/). The gem is called "blackbook" and basically it will go and fetch your contacts from the major webmail providers. So far Gmail, Yahoo!, and MSN have been completed. We are trying to finish up with fetching contacts from AOL Webmail. However
2008 Jul 17
3
Convert data to utf-8
Hello, I''m trying to find a solution to convert everything returned by mechanize to utf-8, no matter if the original page is utf-8 or iso and I really don''t know where to start from... agent = WWW::Mechanize.new { |a| a.log = Logger.new(File::join(RAILS_ROOT, "log/mechanize.log")) } one_page = agent.get("www.google.fr") My first problem is that one_page
2006 Nov 22
1
to_absolute_uri typo in 0.6.3?
I just started using Mechanize, and started using Ruby about thirty seconds before that, but one of the sites I''m scraping does a redirect on form submission to a badly-formed relative URL: index.cfm?action=bing&bang=boom=1|a=|b=|c= (etc.) Interestingly, Mechanize 0.6.2 handled this OK, but in 0.6.3 this causes a URI::InvalidURIError exception from URI.parse() in to_absolute_uri
2009 Jul 29
6
Doubt in nil object with ajax
hi this is my htnl <html> <head> <%= javascript_include_tag :defaults %> </head> <body> <%= form_tag nil, { :id => ''search_form'' } %> <%= text_field ''recipe'', ''name'' %> <%= end_form_tag %> <div id="recipe"> </div> <%= observe_form :search_form, :frequency =>
2007 Nov 12
3
Weird error downloading a gzip''ed file
Hi all, I''ve been using mechanize for a while and it rocks. Docs are pretty clear and so far I''ve been able to do it on my own. However, I''m stuck in a weird situation in a script to download my contact list from hotmail. I''ve used Firebug to check all urls, and tested it by hand while logged in via browser. Even in the script everything works well until the
2013 Feb 19
4
AWS OpsWorks announcement using Chef
I wonder if PuppetLabs will work with Amazon to try to add Puppet as an option to OpsWorks as well ? I don''t think people using Puppet with AWS in a stable fashion would try to move to OpsWorks and migrate everything to Chef, but new customers/startups would think twice in choosing Puppet if the service offered in AWS OpsWorks is based on Chef. I haven''t tried OpsWorks yet,
2010 Jul 08
4
help
Cordial saludo Estoy realizando un análisis con datos categóricos, donde tengo una tabla 3*3, son tres métodos de estudio comparado con tres niveles de asimilación del problema, lo ideal seria poder decidirse por algún método de estudio en especial, lo único que he encontrado es el test para independencia chi-cuadrado, pero no he podido encontrar algo que me sugiera por un método u otro, me
2007 Sep 11
3
Production mode bug with ruby/amazon
Hi all, I''ve got an issue that only ever appears in production mode. I''ve got my app hooked up to amazon''s web service (ECS, using ruby/amazon from http://www.caliban.org/ruby/ruby-amazon.shtml). In the dev environment, everything''s peachy. But in production mode, after I''ve left the app up for a couple of hours (sometimes, minutes), I get an
2016 Jun 28
2
Utilizar una application web usando R
Hola Comunidad, Estoy intentado usar una application web (http://rsscalculatorforpets.royalcanin-world.com/) con R. La idea es tener los datos en R y importalos a la applicacion web y bajar los resultados otra vez a R. Alguna idea de donde puedo encontrar informacion relacionada con este tema? Muchas gracias, Marcal Plans Pujolras UPC-ESAB. Campus del Baix Llobregat . Edifici D4. C. Esteve
2006 Jul 25
7
Paypal - Instant Payment Notifications
Has anyone any experience of getting the Paypal plug in to work? I''ve set myself up on the Paypal sandbox, I''ve setup my sending and receiving accounts, i''ve got the Paypal button on my view, I can go and make payments ok, but I can''t get the Instant Payment Notification part to work. At the end of the payment process the user is left in a paypal screen
2011 Dec 15
4
Scrapping Content from a website
Hey all, I want to scrap some content from a website to rails application and wants to authenticate the url of the website which I want to scarp so please help me out.... Thank You!!! -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to
2007 Apr 03
2
Scraping and saving.
Hi, I''m working to scrape and save some ebooks. Mechanize has been wonderful so far. The link I''m having trouble with is this one. http://www.webscription.net/SendZip.aspx?SKU=0671578499&ProductID=379&format=H When I click that in the browser it saves it to a file named H_1632.zip. How do I get that name from the page. I suspect to save this to a file I would just do