rspec users - Feb 2009 - Cucumber + Webrat + Selenium + ActionMailer

Hi,

We are upgrading to cucumber 0.1.99 in the process we also upgraded to
the latest versions of webrat and rspec.  However now we are having
difficulties checking for ActionMailer.deliveries in the selenium
features.  I seem to recall being able to check for these without
problems using webrat''s wrapper for selenium, but maybe my memory is
tricking me...

Lately we were solving this by using Polomium, here is our config
http://gist.github.com/39038

However, with the new configuration for webrat selenium features:

Webrat.configure do |config|
  config.mode = :selenium
end

We can''t seem to get Polonium working nicely with webrat.

Is there something wrong with our configuration or how are you guys
checking for ActionMailer deliveries in your selenium stories?

Cheers,

Rai
-- 
Posted via http://www.ruby-forum.com/.

Raimond Garcia wrote:
> Hi,
>
> We are upgrading to cucumber 0.1.99 in the process we also upgraded to
> the latest versions of webrat and rspec.  However now we are having
> difficulties checking for ActionMailer.deliveries in the selenium
> features.  I seem to recall being able to check for these without
> problems using webrat''s wrapper for selenium, but maybe my memory
is
> tricking me...
>
> Lately we were solving this by using Polomium, here is our config
> http://gist.github.com/39038
>
> However, with the new configuration for webrat selenium features:
>
> Webrat.configure do |config|
>   config.mode = :selenium
> end
>
> We can''t seem to get Polonium working nicely with webrat.
>
> Is there something wrong with our configuration or how are you guys
> checking for ActionMailer deliveries in your selenium stories?
>
> Cheers,
>
> Rai
>   
One option is to use ARMailer[1] to queue your mail.  As long as your 
selenium process and test process are using the same DB without 
transactions getting in the way then ARMailer should work fine for what 
you want to do.

Thanks to the guys at weplay EmailSpec[2] works happily with ARMailer so 
you can use the same helpers that EmailSpec provides for the regular 
in-memory ActionMailer test deliveries.

-Ben


1. 
http://seattlerb.rubyforge.org/ar_mailer/classes/ActionMailer/ARMailer.html
2. http://github.com/bmabey/email-spec/

On Wed, Feb 18, 2009 at 3:02 PM, Ben Mabey <ben at benmabey.com>
wrote:
>
> One option is to use ARMailer[1] to queue your mail.  As long as your
> selenium process and test process are using the same DB without
transactions
> getting in the way then ARMailer should work fine for what you want to do.
Is changing production infrastructure solely for the sake of making
tests work a recommended practice?  To me it feels a bit backwards.

Raimond: I can''t address the problems you''re having with
Polonium, as
I''d never heard of it until this thread (and Googling it was
surprisingly hard; I finally found
http://github.com/pivotal/polonium).  But it does seem to me like
Webrat and its various dependencies all manage to break each other''s
APIs with every minor release, so this doesn''t surprise me. 
I''ve had
to roll back versions of Nokogiri several times for things like this.

If you''re at the level of integration testing, checking
ActionMailer.deliveries may not be the way to go anyway.  I''d look
instead to see if the e-mails were actually sent.  As I see it, even
if ActionMailer thinks it''s done its job, mail could still fail for
other reasons and that''s a failed system interaction.  Here''s
an idea
I''ve seen that''s independent of delivery method; it''s
basically just
overriding the ActionMailer recipients in test mode so that all
e-mails go to one address:
http://www.dweebd.com/ruby/capture-actionmailer-output-for-your-qa-team/
>From that point, your tests could use Ruby''s standard mail
libraries
to retrieve the mail and scan it, or scrape ''mail'' from the
shell, or
read the entrails of your enemies, or whatever else you like to
confirm that the right stuff is in the message.



-- 
Have Fun,
   Steve Eley (sfeley at gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org

Stephen Eley wrote:
> On Wed, Feb 18, 2009 at 3:02 PM, Ben Mabey <ben at benmabey.com>
wrote:
>   
>> One option is to use ARMailer[1] to queue your mail.  As long as your
>> selenium process and test process are using the same DB without
transactions
>> getting in the way then ARMailer should work fine for what you want to
do.
>>     
>
> Is changing production infrastructure solely for the sake of making
> tests work a recommended practice?  To me it feels a bit backwards.
>   
Well, in this case the ARMailer is an arguably better alternative than 
waiting on an SMTP connection for ActionMailer in production settings.  
ARMailer was not created to solve testing woes.. it was created it solve 
production woes. :)  The fact that it helps in testing is just a side 
effect.

-Ben

On Wed, Feb 18, 2009 at 3:15 PM, Stephen Eley <sfeley at gmail.com>
wrote:
> On Wed, Feb 18, 2009 at 3:02 PM, Ben Mabey <ben at benmabey.com>
wrote:
>>
>> One option is to use ARMailer[1] to queue your mail.  As long as your
>> selenium process and test process are using the same DB without
transactions
>> getting in the way then ARMailer should work fine for what you want to
do.
>
> Is changing production infrastructure solely for the sake of making
> tests work a recommended practice?  To me it feels a bit backwards.
I''m not commenting on this particular situation, but if I want to use
some library, but I can''t figure out how to write tests for my code
that uses it, then I don''t use it.

Basically, if I have to change something to make it testable, I do.

Pat

On Wed, Feb 18, 2009 at 5:43 PM, Pat Maddox <pat.maddox at gmail.com>
wrote:
>
> Basically, if I have to change something to make it testable, I do.
Yeah. Testability is a positive attribute. If you''d change code to
make it more readable or more speedy, why not more testable?

///ark

On Wed, Feb 18, 2009 at 6:27 PM, Ben Mabey <ben at benmabey.com>
wrote:
>
> Well, in this case the ARMailer is an arguably better alternative than
> waiting on an SMTP connection for ActionMailer in production settings.
>  ARMailer was not created to solve testing woes.. it was created it solve
> production woes. :)  The fact that it helps in testing is just a side
> effect.
Oh, sure.  I wasn''t trying to be critical against ARMailer itself; I
do think that e-mail is the sort of side process that ought to be
brought out of the request critical path one way or another.  But
that''s unrelated here.  It just seems like the sort of process change
that might be overkill for ''I upgraded my gems and then my testing
toolchain broke.''  Testability may be a factor in initial tool
selection, but in this case the tool *was* testable until the tests
were broken by an unrelated change.  Once a production process is in
place, I think changes to it should be driven by business
requirements.

I just had a sympathetic twitch for the original poster.  I could be
on some other forum, asking about agility training for my dog or
something, and somebody will say "See, if you had a cat, you
wouldn''t
have that problem."

Yeah, maybe, but the fact that my dog is a dog wasn''t really the
problem.


-- 
Have Fun,
   Steve Eley (sfeley at gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org

On Wed, Feb 18, 2009 at 8:43 PM, Pat Maddox <pat.maddox at gmail.com>
wrote:
>
> I''m not commenting on this particular situation, but if I want to
use
> some library, but I can''t figure out how to write tests for my
code
> that uses it, then I don''t use it.
I take the opposite view.  My *goal* in coding isn''t to write good
tests, it''s to solve somebody''s problem.  My own or someone
else''s.
The right tool to help solve the problem is the right tool.  Testing
plays a supporting role, not a decision-making role.

Anyway, there are very few cases where I can''t figure out *some*
straightforward way to test a behavior.  The last time it happened,
ironically enough, was when I found a bug in Merb''s Autotest hooks for
Rspec.  There was a loop termination issue such that Rspec worked fine
until you tried to run Cucumber right after it with the merb-cucumber
gem.  Writing specs to prove that your specs terminate was...  Well,
after a couple hours I gave up and just submitted a patch with "I know
this is the wrong way to do things, but here.  My head hurts."


-- 
Have Fun,
   Steve Eley (sfeley at gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org

Stephen Eley wrote:
> On Wed, Feb 18, 2009 at 6:27 PM, Ben Mabey <ben at benmabey.com>
wrote:
>   
>> Well, in this case the ARMailer is an arguably better alternative than
>> waiting on an SMTP connection for ActionMailer in production settings.
>>  ARMailer was not created to solve testing woes.. it was created it
solve
>> production woes. :)  The fact that it helps in testing is just a side
>> effect.
>>     
>
> Oh, sure.  I wasn''t trying to be critical against ARMailer itself;
I
> do think that e-mail is the sort of side process that ought to be
> brought out of the request critical path one way or another.  But
> that''s unrelated here.  It just seems like the sort of process
change
> that might be overkill for ''I upgraded my gems and then my testing
> toolchain broke.''  Testability may be a factor in initial tool
> selection, but in this case the tool *was* testable until the tests
> were broken by an unrelated change.  Once a production process is in
> place, I think changes to it should be driven by business
> requirements.
>   
Yeah, and so I probably wouldn''t change to ARMailer in production if 
that was an issue or wasn''t really needed.  So I agree with you there.
However, I wouldn''t have a problem in using ARMailer in my tests but 
still use straight ActionMailer in production.  ActionMailer works and I 
don''t need to test that.  What I''m interested in testing is if
I am
sending the correct emails at the correct time.  If I really need to 
test this part of the app in conjunction with selenium or another 
secondary process then setting up ARMailer just for that purpose seems 
like a reasonable investment.  Once you have that tested the only point 
of failure in production would be your ActionMailer settings. I''ll
point
out that you will face the same risk when using ActionMailer in :test 
mode anyways.  So coming up with a method to test your settings would be 
needed in either case.  My method for doing this is clicking through the 
site once to fire off an email and never changing the working production 
settings again. :)

An alternative to using ARMailer would be to use something like 
Mailtrap[1] that acts as a dummy SMTP server to capture the emails from 
the app.  You would then need to provide a way for the tests to access 
these emails to verify them.

Both ways involve some extra work.  If it isn''t worth the investment 
then the only other option is to test the emails apart from the selenium 
tests.  It just depends on how important it is that you test everything 
at the same time.

-Ben


1. http://github.com/mmower/mailtrap/tree/master

On Thu, Feb 19, 2009 at 12:13 AM, Ben Mabey <ben at benmabey.com>
wrote:
>
> Both ways involve some extra work.  If it isn''t worth the
investment then
> the only other option is to test the emails apart from the selenium tests.
>  It just depends on how important it is that you test everything at the
same
> time.
Good point.  I have to confess, when I first read the message about
using the Selenium framework to test e-mail delivery, my first thought
was:  "Er...  Selenium?  Are you automating the browser to check your
Gmail or something...?"



-- 
Have Fun,
   Steve Eley (sfeley at gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org

Thanks guys, very interesting points.

As a good collegue of mine says, its all about trade offs :)

I agree with changing your production code to make it more testable, 
definitely, especially early on in the development process, once your 
site is up and running it would have to be done with extra care,
we don''t wanna be sending hundreds of emails accidentally, and I must 
confess I can unfortunately relate, even though it was a team effort to 
achieve it :) Thus I''m happy you mentioned Duncan Beevers blog post, 
Steve, cheers.  I''ll definitely recommend it as a precaution tool to
the
team.

For my particular situation ARMailer sounds just like what we need to 
get the build passing again, and will certainly shot out a couple of 
emails from the staging server to make sure emails are actually being 
send.
>"Er...  Selenium?  Are you automating the browser to check your
>Gmail or something...?"

Well this might sound like a crazy idea... but I admit that I''ve 
discussed this before with my team, and concluded that, without 
including speed limit constraints, it was the most thorough way of 
testing our signup and confirmation system :)

Thanks again for all your help!

Cheers,

Rai

P.S. I''ll follow up with a confirmation post just to let the people
know
how it worked out for us with ARMailer.
-- 
Posted via http://www.ruby-forum.com/.

>1.
>http://seattlerb.rubyforge.org/ar_mailer/classes/ActionMailer/ARMailer.html
> P.S. I''ll follow up with a confirmation post just to let the
people know
> how it worked out for us with ARMailer.
Yeap everything worked like a charm, once again checking email contents 
like there is no tomorrow ;-)

Rai
-- 
Posted via http://www.ruby-forum.com/.

> Yeap everything worked like a charm, once again checking email contents 
> like there is no tomorrow ;-)
> 
> Rai
> -- 
> Posted via http://www.ruby-forum.com/.
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
> 
> 

Hi Rai,

Good that you have found a solution.. 

but I still don''t understand why you want to test sending mails from
the
browser. In my project, we use Cucumber + Webrat for writing functional
features.
and Cucumber + selenium for acceptance features.

Functional tests can enable ruby code and test for the number of mails sent
and to whom, etc
Acdeptance only tests what the user would see..

Let me know if it makes sense.. and it would be great if you could elaborate
on your purpose of testing mails from the browser...

Thanks,
Manasi

-- 
View this message in context:
http://www.nabble.com/Cucumber-%2B-Webrat-%2B-Selenium-%2B-ActionMailer-tp22086863p22350981.html
Sent from the rspec-users mailing list archive at Nabble.com.

Manasi Vora wrote:
>> 
>> 
> 
> 
> Hi Rai,
> 
> Good that you have found a solution..
> 
> but I still don''t understand why you want to test sending mails
from the
> browser. In my project, we use Cucumber + Webrat for writing functional
> features.
> and Cucumber + selenium for acceptance features.
> 
> Functional tests can enable ruby code and test for the number of mails 
> sent
> and to whom, etc
> Acdeptance only tests what the user would see..
> 
> Let me know if it makes sense.. and it would be great if you could 
> elaborate
> on your purpose of testing mails from the browser...
> 
> Thanks,
> Manasi
Hi Manasi,

We have a bit of both, sometimes we can test email sending with Cucumber 
+ Webrat, but in this particular case, the feature that sent out emails 
also used Javascript.

We could have split the feature in two, one to test the Javascript with 
a Selenium feature and one to test the emails with a Webrat feature, but 
IMHO adding a couple of steps to the selenium feature to check for 
emails, is a good choice to help to clearly see the whole intent of the 
feature.

Rai
-- 
Posted via http://www.ruby-forum.com/.