Displaying 20 results from an estimated 10000 matches similar to: "Does Amazon.com blocks scraping?"
2010 Jan 26
1
Does Amazon.com block scraping?
Hi there
Does anyone know if Amazon.com has any sort of server side script that tries
to block scraping activities? I first noticed that if I didn?t change the
agent alias, it would fetch a page exactly like the normal one, but without
the intial search field(maybe a silly way to prevent scraping). Then after
it, I changed to some other alias, and submit a search. I got the result
page as
2007 Apr 20
2
Running script does not return the correct page
Hello all,
I have tried to post this yesterday, but noticed I was actually not
subscribed yet... Well, here we go again:
If I run this script, and observe the output, the results are not there
at all (try to do the same in the browser). Any suggestions?
require ''rubygems''
require ''mechanize''
agent = WWW::Mechanize.new
agent.user_agent_alias =
2007 Apr 19
1
Do you have any idea what could be the problem with this script?
Hello all,
If I run this script, and observe the output, the results are not there
at all (try to do the same in the browser). Any suggestions?
require ''rubygems''
require ''mechanize''
agent = WWW::Mechanize.new
agent.user_agent_alias = ''Mac Safari''
page =
2008 Jan 15
1
Converting Watir script to Mechanize
I have a Watir script that I would like to convert to a Mechanize
script. The watir code looks like this
require ''watir''
include Watir
def test
ie = IE.new
ie.goto(''http://cpref.gsm.com/inter.asp?r=8084'')
ie.text_field(:name, ''inter_string'').set(''Potassium'')
ie.button(:id, ''image1'').click
2007 Jul 12
1
WWW::Mechanize::Link.inspect needs some TLC
The problem: users trying to debug Mechanize apps with Komodo
are finding the debugger times out once it''s loaded a web
page. They don''t run into this in the ruby-debug debugger,
or running in normal mode.
The reason: Komodo''s debugger is graphical, which means that
whenever it hits a breakpoint it automatically shows the
contents of each local variable. It has a
2007 May 06
1
How to submit this form?
Hey all,
maybe I am missing the obvious again, but I can not submit the form on
buy.com:
require ''rubygems''
require ''mechanize''
agent = WWW::Mechanize.new
agent.user_agent_alias = ''Mac Safari''
page = agent.get("http://www.buy.com")
search_form = page.forms.with.name("searchbox").first
p search_form.buttons
the last
2007 Jan 04
5
Mashup Tutorial
Hi:
I am looking for a good hands on web mashup tutorial? or books for that
matter.. Google gives me bits and pieces of the info but not any step by
step or hands on tutorial..
Thanks a bundle
--
Posted via http://www.ruby-forum.com/.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk"
2007 May 20
1
Tips on testing
Hey WWW::Mechanize users,
I''ve been using Mechanize for about 3 or 4 months now, and I''m rather
enjoying it. I''ve done scraping in PHP & Python, starting with raw text and
HTTP requests, moving up to regex and Curl, and finally landed myself here,
making me much more productive when scraping pages.
At the moment, I''m in the middle of a decent sized
2007 Oct 10
1
Scraping AOL Webmail to login and fetch contacts?
I''m helping with a gem that is going to published under the
contentfree project on rubyforge
(http://rubyforge.org/projects/contentfree/).
The gem is called "blackbook" and basically it will go and fetch your
contacts from the major webmail providers. So far Gmail, Yahoo!, and
MSN have been completed.
We are trying to finish up with fetching contacts from AOL Webmail.
However
2008 Jul 17
3
Convert data to utf-8
Hello, I''m trying to find a solution to convert everything returned by
mechanize to utf-8, no matter if the original page is utf-8 or iso and I
really don''t know where to start from...
agent = WWW::Mechanize.new { |a| a.log =
Logger.new(File::join(RAILS_ROOT, "log/mechanize.log")) }
one_page = agent.get("www.google.fr")
My first problem is that one_page
2006 Nov 22
1
to_absolute_uri typo in 0.6.3?
I just started using Mechanize, and started using Ruby about thirty
seconds before that, but one of the sites I''m scraping does a redirect
on form submission to a badly-formed relative URL:
index.cfm?action=bing&bang=boom=1|a=|b=|c= (etc.)
Interestingly, Mechanize 0.6.2 handled this OK, but in 0.6.3 this causes
a URI::InvalidURIError exception from URI.parse() in to_absolute_uri
2009 Jul 29
6
Doubt in nil object with ajax
hi this is my htnl
<html>
<head>
<%= javascript_include_tag :defaults %>
</head>
<body>
<%= form_tag nil, { :id => ''search_form'' } %>
<%= text_field ''recipe'', ''name'' %>
<%= end_form_tag %>
<div id="recipe">
</div>
<%= observe_form :search_form, :frequency =>
2007 Nov 12
3
Weird error downloading a gzip''ed file
Hi all,
I''ve been using mechanize for a while and it rocks. Docs are pretty clear
and so far I''ve been able to do it on my own.
However, I''m stuck in a weird situation in a script to download my contact
list from hotmail.
I''ve used Firebug to check all urls, and tested it by hand while logged in
via browser.
Even in the script everything works well until the
2013 Feb 19
4
AWS OpsWorks announcement using Chef
I wonder if PuppetLabs will work with Amazon to try to add Puppet as an
option to OpsWorks as well ?
I don''t think people using Puppet with AWS in a stable fashion would try to
move to OpsWorks and migrate everything to Chef, but new customers/startups
would think twice in choosing Puppet if the service offered in AWS OpsWorks
is based on Chef.
I haven''t tried OpsWorks yet,
2010 Jul 08
4
help
Cordial saludo
Estoy realizando un análisis con datos categóricos, donde tengo una tabla 3*3, son tres métodos de estudio comparado con tres niveles de asimilación del problema, lo ideal seria poder decidirse por algún método de estudio en especial, lo único que he encontrado es el test para independencia chi-cuadrado, pero no he podido encontrar algo que me sugiera por un método u otro, me
2007 Sep 11
3
Production mode bug with ruby/amazon
Hi all,
I''ve got an issue that only ever appears in production mode. I''ve got
my app hooked up to amazon''s web service (ECS, using ruby/amazon from
http://www.caliban.org/ruby/ruby-amazon.shtml).
In the dev environment, everything''s peachy. But in production mode,
after I''ve left the app up for a couple of hours (sometimes, minutes),
I get an
2016 Jun 28
2
Utilizar una application web usando R
Hola Comunidad,
Estoy intentado usar una application web
(http://rsscalculatorforpets.royalcanin-world.com/) con R. La idea es
tener los datos en R y importalos a la applicacion web y bajar los
resultados otra vez a R.
Alguna idea de donde puedo encontrar informacion relacionada con este
tema?
Muchas gracias,
Marcal Plans Pujolras
UPC-ESAB.
Campus del Baix Llobregat . Edifici D4. C. Esteve
2006 Jul 25
7
Paypal - Instant Payment Notifications
Has anyone any experience of getting the Paypal plug in to work?
I''ve set myself up on the Paypal sandbox, I''ve setup my sending and
receiving accounts, i''ve got the Paypal button on my view, I can go and
make payments ok, but I can''t get the Instant Payment Notification part
to work. At the end of the payment process the user is left in a paypal
screen
2011 Dec 15
4
Scrapping Content from a website
Hey all,
I want to scrap some content from a website to rails application and
wants to authenticate the url of the website which I want to scarp so
please help me out....
Thank You!!!
--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To post to this group, send email to
2007 Apr 03
2
Scraping and saving.
Hi,
I''m working to scrape and save some ebooks. Mechanize has been
wonderful so far. The link I''m having trouble with is this one.
http://www.webscription.net/SendZip.aspx?SKU=0671578499&ProductID=379&format=H
When I click that in the browser it saves it to a file named
H_1632.zip. How do I get that name from the page. I suspect to save
this to a file I would just do