Rails - Mar 2006 - Is Rails getting testing semantics wrong and making things confusing?

Yes, I believe it is and I think it is dangerous, and confusing to people
new to the concepts of testing.

I''ve mentioned a point similar to this on here before, but with the
announcement of Rails "integration" test, I''m worried that
somebody
somewhere is getting their types of tests worryingly mixed up.

Lets start with what we had in the beginning...unit tests (for models) and
"functional" tests (for controllers). This was wrong from the start,
because
Rails'' idea of functional tests were in fact just UNIT tests for
controllers
(or specifically, single actions in a controller). A unit test being, a test
that tests a small unit functionality at the code level - it makes sure that
that unit of code does what it is supposed to do. In a Rails app, a unit
might be a single Model or it might be a single action on a controller.

So what is functional testing? Functional testing is another name for
ACCEPTANCE testing. What is ACCEPTANCE testing? I will quote from this
article:

http://www-128.ibm.com/developerworks/java/library/wa-selenium-ajax/?ca=drs-

Acceptance testing, also known as black-box and functional testing, is a
way
> of testing and verifying that an application works according to functional,
> non-functional, and other important stakeholder requirements. Acceptance
> tests complement unit and integration tests, which are usually written
using
> xUnit frameworks. Acceptance tests can also be written using a programming
> language, but Selenium and similar tools like Fitnesse also support tests
> written using a tool-specific document format.
>
Further more, there are some important rules regarding acceptance tests:
> Acceptance tests differ from unit and integration tests in the following
> ways:
>
>    - The application is tested as a whole, end-to-end entity, not just
>    one class or a set of classes as it is with unit and integration tests.
>    - Acceptance tests are performed against the user interface using,
>    for example, a browser against a Web application interface.
>    - The person writing the test cases does not necessarily know about
>    the internals of the application, hence the designation of black-box
>    testing. Non-technical users can write acceptance tests.
>
> The key point here is that acceptance tests make sure that the system
works end-to-end and complies with its core requirements. In Extreme
Programming terms, this is often a "user story". A user story is
considered
complete when all acceptance tests pass. Unlike unit tests, which should
ideally be running with a 100% pass rate, acceptance tests start at 0% and
finish at 100% when the story is fully implemented. From the
extremeprogramming.org website:

Acceptance tests are black box system tests. Each acceptance test
represents
> some expected result from the system. Customers are responsible for
> verifying the correctness of the acceptance tests and reviewing test scores
> to decide which failed tests are of highest priority. Acceptance tests are
> also used as regression tests prior to a production release.
> A user story is not considered complete until it has passed its acceptance
> tests. This means that new acceptance tests must be created each iteration
> or the development team will report zero progress.
>
So now, we have the announcement of Integration Tests for Rails.
Integrations should be tests to ensure that multiple units work together
correctly. But Rails integration tests look just like acceptance/functional
tests though (much like FIT or Selenium in driven mode)! How confusing for
the testing newbie!

The reason I bring this up is because I really believe the Rails developers
should look at clarifying and renaming their tests to get the semantics
correct. How can you expect testing newbies to start testing their apps when
the framework they are using can''t even get the names of their tests
correct? So what do I propose:

   - Rails'' existing "functional" tests should be correctly
referred to
   as unit tests, which is all they are. For organisational purposes I can see
   the advantage of breaking up your model and controller unit tests as the
   unit testing strategy for these two different types of object are usually
   different.
   - Rails'' new "integration" tests are renamed to functional
or
   acceptance tests, which is what they really are. They should be seen as a
   built-in alternative to using a browser-based acceptance testing tool like
   Watir or Selenium. In fact, Rails'' new built-in Integration tests
could be
   compared to something like Selenium in "driven" mode instead of
test-runner
   mode. Personally, I feel that for web apps Selenium in test-runner mode is
   more appropriate, but thats just my preference.
   - Integration tests are left to the user to put together if they feel
   they need to use them. Their typical usage would be if there are several
   models that need to work together to produce results.

To support the above the tests folder would probably look something like
this:

unit/models
unit/controllers
acceptance/
integration/
fixtures/

I''d like to hear people''s thoughts on the above.

--
Cheers,
Luke Redpath
www.lukeredpath.co.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060305/f6b7b937/attachment.html

Luke Redpath wrote:
> The reason I bring this up is because I really believe the Rails 
> developers
> should look at clarifying and renaming their tests to get the semantics
> correct. How can you expect testing newbies to start testing their apps 
> when
> the framework they are using can''t even get the names of their
tests
> correct? So what do I propose:
> 
>    - Rails'' existing "functional" tests should be
correctly referred to
>    as unit tests, which is all they are. For organisational purposes I 
> can see
>    the advantage of breaking up your model and controller unit tests as 
> the
>    unit testing strategy for these two different types of object are 
> usually
>    different.
>    - Rails'' new "integration" tests are renamed to
functional or
>    acceptance tests, which is what they really are. They should be seen 
> as a
>    built-in alternative to using a browser-based acceptance testing tool 
> like
>    Watir or Selenium. In fact, Rails'' new built-in Integration
tests
> could be
>    compared to something like Selenium in "driven" mode instead
of
> test-runner
>    mode. Personally, I feel that for web apps Selenium in test-runner 
> mode is
>    more appropriate, but thats just my preference.
>    - Integration tests are left to the user to put together if they feel
>    they need to use them. Their typical usage would be if there are 
> several
>    models that need to work together to produce results.
> 
> To support the above the tests folder would probably look something like
> this:
> 
> unit/models
> unit/controllers
> acceptance/
> integration/
> fixtures/
> 
> I''d like to hear people''s thoughts on the above.
Hi,

As for your suggestion to stop referring to controller unit tests as
"functional" tests I couldn''t agree more. However, renaming
the new
integration tests to "acceptance" would not be a good idea as far as
I''m
concerned. As you wrote, acceptance tests complement unit and 
integration tests - they check if stakekolder requirements are met. Unit 
test check how
components work in isolation, integration tests check cooperation
between components. So I suppose even if "integration" might not be
the
best word it seems better than "acceptance", since what the new tests
actually bring in is the ability to test cooperation between
controllers. The fact that this makes it possible to test scenarios does
not make it "acceptance tests" I guess :)

--
Agnieszka Figiel

-- 
Posted via http://www.ruby-forum.com/.

I agree with what you said, Luke, although I don''t yet have a good
understanding of Watir and Selenium, which I''ve been meaning to check
out.

I don''t mean to hijack this thread, but I have to share this as an
example
of what Luke was touching upon. Speaking of testing vocabulary; today at
work our group of developers were overruled as unit testing is now to be
known as developer testing in our organization. We watched in amazement as
management declared unit testing to mean non-automated scripted testing,
otherwise known as functional testing. Talk about going against standard
industry concepts!

Peter

On 3/5/06, Luke Redpath <contact@lukeredpath.co.uk>
wrote:
>
> Yes, I believe it is and I think it is dangerous, and confusing to people
> new to the concepts of testing.
>
> I''ve mentioned a point similar to this on here before, but with
the
> announcement of Rails "integration" test, I''m worried
that somebody
> somewhere is getting their types of tests worryingly mixed up.
>
> Lets start with what we had in the beginning...unit tests (for models) and
> "functional" tests (for controllers). This was wrong from the
start, because
> Rails'' idea of functional tests were in fact just UNIT tests for
controllers
> (or specifically, single actions in a controller). A unit test being, a
test
> that tests a small unit functionality at the code level - it makes sure
that
> that unit of code does what it is supposed to do. In a Rails app, a unit
> might be a single Model or it might be a single action on a controller.
>
> So what is functional testing? Functional testing is another name for
> ACCEPTANCE testing. What is ACCEPTANCE testing? I will quote from this
> article:
>
>
>
http://www-128.ibm.com/developerworks/java/library/wa-selenium-ajax/?ca=drs-
>
> Acceptance testing, also known as black-box and functional testing, is a
> > way of testing and verifying that an application works according to
> > functional, non-functional, and other important stakeholder
requirements.
> > Acceptance tests complement unit and integration tests, which are
usually
> > written using xUnit frameworks. Acceptance tests can also be written
using a
> > programming language, but Selenium and similar tools like Fitnesse
also
> > support tests written using a tool-specific document format.
> >
>
> Further more, there are some important rules regarding acceptance tests:
>
> > Acceptance tests differ from unit and integration tests in the
following
> > ways:
> >
> >    - The application is tested as a whole, end-to-end entity, not
> >    just one class or a set of classes as it is with unit and
integration tests.
> >    - Acceptance tests are performed against the user interface using,
> >    for example, a browser against a Web application interface.
> >    - The person writing the test cases does not necessarily know
> >    about the internals of the application, hence the designation of
black-box
> >    testing. Non-technical users can write acceptance tests.
> >
> > The key point here is that acceptance tests make sure that the system
> works end-to-end and complies with its core requirements. In Extreme
> Programming terms, this is often a "user story". A user story is
considered
> complete when all acceptance tests pass. Unlike unit tests, which should
> ideally be running with a 100% pass rate, acceptance tests start at 0% and
> finish at 100% when the story is fully implemented. From the
> extremeprogramming.org website:
>
> Acceptance tests are black box system tests. Each acceptance test
> > represents some expected result from the system. Customers are
responsible
> > for verifying the correctness of the acceptance tests and reviewing
test
> > scores to decide which failed tests are of highest priority.
Acceptance
> > tests are also used as regression tests prior to a production release.
> > A user story is not considered complete until it has passed its
> > acceptance tests. This means that new acceptance tests must be created
each
> > iteration or the development team will report zero progress.
> >
>
> So now, we have the announcement of Integration Tests for Rails.
> Integrations should be tests to ensure that multiple units work together
> correctly. But Rails integration tests look just like acceptance/functional
> tests though (much like FIT or Selenium in driven mode)! How confusing for
> the testing newbie!
>
> The reason I bring this up is because I really believe the Rails
> developers should look at clarifying and renaming their tests to get the
> semantics correct. How can you expect testing newbies to start testing
their
> apps when the framework they are using can''t even get the names of
their
> tests correct? So what do I propose:
>
>    - Rails'' existing "functional" tests should be
correctly referred to
>    as unit tests, which is all they are. For organisational purposes I can
see
>    the advantage of breaking up your model and controller unit tests as the
>    unit testing strategy for these two different types of object are
usually
>    different.
>    - Rails'' new "integration" tests are renamed to
functional or
>    acceptance tests, which is what they really are. They should be seen as
a
>    built-in alternative to using a browser-based acceptance testing tool
like
>    Watir or Selenium. In fact, Rails'' new built-in Integration
tests could be
>    compared to something like Selenium in "driven" mode instead
of test-runner
>    mode. Personally, I feel that for web apps Selenium in test-runner mode
is
>    more appropriate, but thats just my preference.
>    - Integration tests are left to the user to put together if they
>    feel they need to use them. Their typical usage would be if there are
>    several models that need to work together to produce results.
>
> To support the above the tests folder would probably look something like
> this:
>
> unit/models
> unit/controllers
> acceptance/
> integration/
> fixtures/
>
> I''d like to hear people''s thoughts on the above.
>
> --
> Cheers,
> Luke Redpath
> www.lukeredpath.co.uk
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060308/f7df485e/attachment-0001.html