Mechanize users - May 2007 - Tips on testing

Hey WWW::Mechanize users,

I''ve been using Mechanize for about 3 or 4 months now, and I''m
rather
enjoying it. I''ve done scraping in PHP & Python, starting with raw
text and
HTTP requests, moving up to regex and Curl, and finally landed myself here,
making me much more productive when scraping pages.

At the moment, I''m in the middle of a decent sized project, which
requires
scraping lots of pages. Now, I''d like to be able to refactor much of
it,
however I''m not entirely happy with the tests that I''m
writing. I''m creating
way too many global variables, and monkey patching classes instead of
creating mocks. I would ideally like to have something like
ActiveRecord''s
fixtures, where I can serialize a page along with some meta-data about it,
along with a mock instance of my Mechanize instance.

Has anyone else done any extensive testing with Mechanize? I''d like to
hear
about what kinds of tips and tricks you employed to come up with something
that worked well.

Thanks!

-Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mechanize-users/attachments/20070520/fcd240d3/attachment.html

Hi Farley!

On Sun, May 20, 2007 at 05:12:45PM -0400, Farley Knight
wrote:
> Hey WWW::Mechanize users,
> 
> I''ve been using Mechanize for about 3 or 4 months now, and
I''m rather
> enjoying it. I''ve done scraping in PHP & Python, starting with
raw text and
> HTTP requests, moving up to regex and Curl, and finally landed myself here,
> making me much more productive when scraping pages.
> 
> At the moment, I''m in the middle of a decent sized project, which
requires
> scraping lots of pages. Now, I''d like to be able to refactor much
of it,
> however I''m not entirely happy with the tests that I''m
writing. I''m creating
> way too many global variables, and monkey patching classes instead of
> creating mocks. I would ideally like to have something like
ActiveRecord''s
> fixtures, where I can serialize a page along with some meta-data about it,
> along with a mock instance of my Mechanize instance.
> 
> Has anyone else done any extensive testing with Mechanize? I''d
like to hear
> about what kinds of tips and tricks you employed to come up with something
> that worked well.
Sorry for taking so long to reply.  Typically what I''ll do for testing
is moch out my mechanize objects.  Sometimes I like to think of it as a
web service, and test it using similar patterns.  Unfortunately
people''s
websites can change, and you''ll only really run in to that during live
runs.

I suggest checking out flexmock:

  http://flexmock.rubyforge.org/

Good luck!  Hope that helps.

-- 
Aaron Patterson
http://tenderlovemaking.com/