rspec users - Oct 2008 - Should acceptance tests be run against a production environment?

When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment.  Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production.  Also, transactional fixtures should be turned
off.  This is so that the app runs as closely as possible to how it does
from a regular user''s perspective.  Models and pages get cached,
transactions commit and rollback as they''re defined, etc.  What do you
think?  Am I off base here?

Pat

On Tue, Oct 28, 2008 at 12:08 PM, Pat Maddox <pergesu at gmail.com>
wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it does
> from a regular user''s perspective.  Models and pages get cached,
> transactions commit and rollback as they''re defined, etc.  What do
you
> think?  Am I off base here?
I haven''t been doing this (generally running against the test env),
but I think this sounds right.

FWIW,
David
>
> Pat
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

I agree. I have seen way too many times selenium tests are OK but bugs
appear in production. Not only should we run selenium tests against
production environment, but also they should be run on a production like
environment, such as, same OS, same setting (behind Apache, or whatever HTTP
servers, etc.)

On Tue, Oct 28, 2008 at 12:08 PM, Pat Maddox <pergesu at gmail.com> wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it does
> from a regular user''s perspective.  Models and pages get cached,
> transactions commit and rollback as they''re defined, etc.  What do
you
> think?  Am I off base here?
>
> Pat
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20081028/55951340/attachment.html>

On Tue, Oct 28, 2008 at 1:08 PM, Pat Maddox <pergesu at gmail.com>
wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.
I believe that''s what the angels call a "staging
environment."

And yes.

-- 
Have Fun,
   Steve Eley (sfeley at gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org

On Oct 28, 2008, at 5:08 pm, Pat Maddox wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it  
> does
> from a regular user''s perspective.  Models and pages get cached,
> transactions commit and rollback as they''re defined, etc.  What do
you
> think?  Am I off base here?
Hi Pat

This makes sense to me.  I actually tend to run feature files against  
the development environment though (well, features, which is a cp of  
development).  The code reloading saves me from having to stop and  
start the server.  Transactions and page caching are the big ones to  
me - model caching should *reduce* bugs, surely.

A run against a production makes sense though, especially before  
committing.

Ashley

-- 
http://www.patchspace.co.uk/
http://aviewfromafar.net/

On Tue, Oct 28, 2008 at 6:08 PM, Pat Maddox <pergesu at gmail.com>
wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it does
> from a regular user''s perspective.  Models and pages get cached,
> transactions commit and rollback as they''re defined, etc.  What do
you
> think?  Am I off base here?
>
I recommend running against a production-LIKE environment. In Rails
you can create a production_test environment that you make as close to
your production environment as possible, for example by running
against a production_test database that contains a dump of your
production data from last night.

Aslak
> Pat
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

Pat Maddox wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it does
> from a regular user''s perspective.  Models and pages get cached,
> transactions commit and rollback as they''re defined, etc.  What do
you
> think?  Am I off base here?
>   
That is exactly what I do Pat. I want to get as close to testing the 
real users experience as possible. In the end those real user 
experiences are my greatest concern. When you move away from testing a 
production rails instances you leave yourself open to ''Users are 
complaining about X not working. Well it worked in testing'' :)

It however does mean you have to be very careful to isolate your 
acceptance tests from each other due to caching/session/cookie/database 
commits.

--
Joseph Wilk
http://www.joesniff.co.uk

One thing that bothers me about a ''staging'' or
''production_test'' environment
is simply the value of Rails.env. That should really ==
''production'', it
seems to me, but that''s not always practical.

///ark
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20081029/41f1d6ff/attachment.html>

aslak hellesoy wrote:
> I recommend running against a production-LIKE environment. In Rails
> you can create a production_test environment that you make as close to
> your production environment as possible, for example by running
> against a production_test database that contains a dump of your
> production data from last night.

I''m assuming that this is running on your development machine, right?

What about the idea of running it on machines that are as close to
production as possible? So if my production machine is a SliceHost VPS
with 256MB RAM, Nginx, and 3 Mongrels, then I should be running these
tests on a second SliceHost VPS with 256MB RAM, Nginx, and 3 Mongrels.

Is that just overkill?

On Wed, Oct 29, 2008 at 4:04 PM, DyingToLearn <phylae at gmail.com>
wrote:
>
> What about the idea of running it on machines that are as close to
> production as possible? So if my production machine is a SliceHost VPS
> with 256MB RAM, Nginx, and 3 Mongrels, then I should be running these
> tests on a second SliceHost VPS with 256MB RAM, Nginx, and 3 Mongrels.
>
> Is that just overkill?
It depends on what your risks and worries are.  If you have code that
has dependencies on a certain server configuration, database, etc., or
if you''re pushing resource boundaries in any way, then yeah, you
should test it against those constraints as best you can before
sending it to the wild.  If what you wrote is just a very vanilla
Rails app with no abstractions altered and you''re not anticipating a
traffic slam, it probably doesn''t matter if you match all resources
precisely.  You should at least match basic software (don''t put your
staging environment on Thin with Ruby 1.9 if your production is
Phusion Passenger with Ruby Enterprise) but having the same number of
boxes with the same RAM is not important unless there''s a reason.

That said, I doubt that there are many people who would call a
Slicehost VPS with 256 MB RAM overkill regardless.  >8->  If
you''re on
the low end like that, what the hell, spend another twenty bucks and
then clone your production slice.  You could even delete the staging
slice between test iterations.


-- 
Have Fun,
   Steve Eley (sfeley at gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org