[Mechanize-users] Does Amazon.com blocks scraping?

Jimmy McGrath
2010-01-27T00:56:41Z

Apologies to the list if this is the second time this message is sent, I
had an issue with my mail client and I think my original attempt failed.
------------------------------------------------------------------------

No worries Felipe, thought you would have probably been on the case
already about the Amazon web services, but couldn''t hurt to suggest it
just in case.

Now, on to your issue:

You say you are getting the following on the console:
> #ActionController::RoutingError (No route matches
> "/aan/2009-09-#09/static/amazon/ iframeproxy-1.html" with
{:method=>:get}):
>
This screams out Rails to me. I doubt it has anything to do with
Mechanize at all (anyone with more experience with Mechanize please feel
free to correct me). Are you sure you are not making a request to the
rails framework somewhere?

As to the command:
> @match =
@page.search("/html/body/div[4]/div/div/div[2]/div[3]/div/div/div[3]/div/a")
If I perform the following search:
@page.search("/html/body/div[4]/div")

I get an empty array as a response, so that search argument is borked. I
don''t think that is the best way to get a link out of the page anyway
as
that will be very brittle. I think that you should look into the
nokogiri documentation for a way to search for links with a specific css
class. I can''t tell you off the top of my head how to do it and I need
to get back to work, so I''ll have to leave you to work it out yourself.

BTW: In Answer to the original question: "Does Amazon block scraping",
I
don''t think that they are attempting to block scraping at all, more
likely they don''t recognise the mechanise user agent string and they
get
confused. It would be an interesting exercise to use the user agent
switch plug-in in firefox and set it as mechanize, then see how amazon
renders within firefox, but as I said, I have to get back to work.

Ciao.

Felipe Jord?o A. P. Mattosinho wrote:
> Thanks Jimmy,
>
> First thanks for answering it. About the web service I know about it,
> however the thing is that I am doing a project in my university where I
need
> to scrap content, or to be more specific product reviews. So I just used
> Amazon.com as an example, but I would never scrap something when I have a
> service that can give me everything I need. The thing is just to use a
> scraping technique!But thanks anyway for you suggestion!
> Well, besides that these are the commands:
>
> @@mech = WWW::Mechanize.new
>
> #HERE IF I DON?T SET THE ALIAS I RECEIVE THE SAME MAIN PAGE BUT WITHOUT THE
> SEARCH #FIELD
>
> @@mech.user_agent_alias = ''Mac Safari''
>
> page = @@mech.get("http://www.amazon.com")
>
> search_form = page.form("site-search")
>
> search_form["field-keywords"] = "Nikon Coolpix P90"
>
> @page = @@mech.submit(search_form, search_form.buttons.first)
>
>
> PP @page
>
>
> #HERE AFTER printing the page I confirm that I received the page I was
> #expecting, however on the console I  get this "
> #ActionController::RoutingError (No route matches
> "/aan/2009-09-#09/static/amazon/ iframeproxy-1.html" with
{:method=>:get}):
> "
>
> #Now when I want to get the first match in the result page from that XPATH
> # I receive a null object (which is strange for me since I have the content
> # stored on @page variable)
>
> @match >
@page.search("/html/body/div[4]/div/div/div[2]/div[3]/div/div/div[3]/div/a")
>
> # HERE @match is nil, which sounds strange to me.
>
> NOW A QUESTIOON CONCERNING THE MAILGROUP ITSELF: Is there a way to receive
> single messages and the digest. Or if I enable the digest mode, I stop
> receiving single messages?
>
>
> Felipe
>
> -