rspec users - Jun 2008 - MVC testing vs. models & Webrat+RailsStory?

I just read David Chelimsky''s slides from RailsConf.  I was at
RailsConf, but was unable to attend the session.  The slides are
excellent, and I''ve recently become VERY interested in stories.  In
particular I''m a bit unhappy with regular view testing (even with the
easier notion of it in RSpec).  I was wondering what folks think about
testing the full stack these days, and any recommendations you have?

Specifically, I''m wondering, or contemplating, if I do unit tests for
my models, and then I use WebRat plus RailsStory, do I even need to
then do functional testing of my controllers and views?  I can see
that I might want to do some particular view testing that ensured
certain elements were on a page or something, but in terms of actually
testing the workings of my app/site, it seems model unit tests
combined with RailsStory+Webrat will do a pretty darn thorough job.  I
would then consider going to Selenium to test some of the detailed
JavaScript as needed.

Thoughts?  What am I missing, or what are the downsides of such an approach?

--
Christopher Bailey
Cobalt Edge LLC
http://cobaltedge.com

On Thu, Jun 19, 2008 at 9:31 AM, Christopher Bailey
<chris at cobaltedge.com> wrote:
> Specifically, I''m wondering, or contemplating, if I do unit tests
for
> my models, and then I use WebRat plus RailsStory, do I even need to
> then do functional testing of my controllers and views?  I can see
> that I might want to do some particular view testing that ensured
> certain elements were on a page or something, but in terms of actually
> testing the workings of my app/site, it seems model unit tests
> combined with RailsStory+Webrat will do a pretty darn thorough job.  I
> would then consider going to Selenium to test some of the detailed
> JavaScript as needed.
>
> Thoughts?  What am I missing, or what are the downsides of such an
approach?
In the project I''m working on, we lean pretty heavily on model/helper
specs and webrat-driven plain text stories.

We still do *some* controller specs, but only in those instances where
we deviate from the standard REST design, or to test authentication
requirements. With a custom it_should_... example generator, that can
be as simple as:

describe FooController do
  describe "GET :index" do
    it_should_require_admin_access(:get, :index)
  end
end

We currently have no view specs at all, though I am beginning to
regret that decision somewhat. Initially we avoided them because the
effort required to mock everything out wasn''t jiving well with rapid
prototyping, but I''m considering going back and pushing some things
into view specs using stub_model(). There are a lot of simple
requirements that are a good fit for view specs which we are currently
doing in the user stories, solely because the infrastructure was
already there.

As far as controllers go, I wouldn''t say we''ve seen any
downside to
relying on stories instead of isolated specs; I''m sure you''ve
seen how
90% of every controller spec ends up looking the same, and they start
to be a headache to maintain. Maybe the shoulda-esque rspec plugins
out there would help, but when I can easily just exercise the full
stack, I haven''t bothered to try. I saw one big *win* of not testing
controllers in isolation just this week, though, when I had to shift a
lot of an application''s view paths around; with standard controller
specs, I would''ve been spending a lot of typing updating the render
expectations, but with stories it wasn''t an issue: the right markup
either showed up or didn''t.

The primary downside for us has been speed; it takes a few minutes to
run the full story suite. The ability to selectively run just a few
stories, and then a good CI server, alleviates that for the most part.

I always ramble too much so I''ll stop here. HTH,

k

Kyle, thanks much for sharing your experience.  You mention the speed
and so on.  I''ve read that it is slow.  Question: does Autotest work
the same way with stories, or have a way to detect what file(s)
changed and run the appropriate stories, or because they''re
integration tests, would it just re-run the entire story suite as it
really doesn''t have a way to correlate source files to stories?  I
presume the latter.

I guess what I''d wonder, in terms of dealing with the speed issue is:

- Can I run a single story via TextMate like I do with specs?  That
would allow me to concentrate on one particular thing if I needed
quick turn around time.
- I just always have autotest running, and most of the time, I just
let it do its thing, listening for any failures (I use Mac''s speech to
alert me :)  So, the speed isn''t so bad as I typically just keep
working until I hear a problem, then go fix that.
- I am a huge fan of CI servers, and have one setup for every project
I control.  I do like to know my suite passes before I push my changes
up, but the CI server is a great way to ensure everything passes on a
clean system.  I suppose if the story runner was really taking very
long times, that I''d be ok pushing and seeing if the CI server caught
a problem.

I''d like to hear other''s thoughts as well, but your experience
seems
consistent with my theory of the direction I want to go: unit test
models and helpers, drop controller and view tests in favor of Webrat
+ RailsStory, and then, as needed drop in RSpec view tests for things
that can''t be as efficiently/effectively tested in stories.

Final bit/question (for now ;-)...  With Webrat, does it pay attention
to whether or not div''s and such are hidden/displayed in terms of
whether it lets you enter values in them?  In this particular project
I use a fair bit of disclosure style UI, where some fields are shown
after a use takes some other action or presses a button, etc.  But
they''re all there, just not visible.  So, wondering if I can simply
tell Webrat to put data into them, and not worry about having to
effect a button press that does its work via JavaScript.  I can move
to Selenium to handle that, but that''s actually something I''m
not
wanting to do yet (I''ve used it in the past, and it''s just a
ton of
overhead, truly slow, etc.).

On Thu, Jun 19, 2008 at 8:41 AM, Kyle Hargraves <philodespotos at
gmail.com> wrote:
> On Thu, Jun 19, 2008 at 9:31 AM, Christopher Bailey
> <chris at cobaltedge.com> wrote:
>> Specifically, I''m wondering, or contemplating, if I do unit
tests for
>> my models, and then I use WebRat plus RailsStory, do I even need to
>> then do functional testing of my controllers and views?  I can see
>> that I might want to do some particular view testing that ensured
>> certain elements were on a page or something, but in terms of actually
>> testing the workings of my app/site, it seems model unit tests
>> combined with RailsStory+Webrat will do a pretty darn thorough job.  I
>> would then consider going to Selenium to test some of the detailed
>> JavaScript as needed.
>>
>> Thoughts?  What am I missing, or what are the downsides of such an
approach?
>
> In the project I''m working on, we lean pretty heavily on
model/helper
> specs and webrat-driven plain text stories.
>
> We still do *some* controller specs, but only in those instances where
> we deviate from the standard REST design, or to test authentication
> requirements. With a custom it_should_... example generator, that can
> be as simple as:
>
> describe FooController do
>  describe "GET :index" do
>    it_should_require_admin_access(:get, :index)
>  end
> end
>
> We currently have no view specs at all, though I am beginning to
> regret that decision somewhat. Initially we avoided them because the
> effort required to mock everything out wasn''t jiving well with
rapid
> prototyping, but I''m considering going back and pushing some
things
> into view specs using stub_model(). There are a lot of simple
> requirements that are a good fit for view specs which we are currently
> doing in the user stories, solely because the infrastructure was
> already there.
>
> As far as controllers go, I wouldn''t say we''ve seen any
downside to
> relying on stories instead of isolated specs; I''m sure
you''ve seen how
> 90% of every controller spec ends up looking the same, and they start
> to be a headache to maintain. Maybe the shoulda-esque rspec plugins
> out there would help, but when I can easily just exercise the full
> stack, I haven''t bothered to try. I saw one big *win* of not
testing
> controllers in isolation just this week, though, when I had to shift a
> lot of an application''s view paths around; with standard
controller
> specs, I would''ve been spending a lot of typing updating the
render
> expectations, but with stories it wasn''t an issue: the right
markup
> either showed up or didn''t.
>
> The primary downside for us has been speed; it takes a few minutes to
> run the full story suite. The ability to selectively run just a few
> stories, and then a good CI server, alleviates that for the most part.
>
> I always ramble too much so I''ll stop here. HTH,
>
> k
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>


-- 
Christopher Bailey
Cobalt Edge LLC
http://cobaltedge.com

Just found answer to one of my questions below on being able to run
stories in TextMate.  The new bundle for stories looks cool:
http://github.com/bmabey/rspec-story-tmbundle/tree/master

On Thu, Jun 19, 2008 at 9:56 AM, Christopher Bailey
<chris at cobaltedge.com> wrote:
> Kyle, thanks much for sharing your experience.  You mention the speed
> and so on.  I''ve read that it is slow.  Question: does Autotest
work
> the same way with stories, or have a way to detect what file(s)
> changed and run the appropriate stories, or because they''re
> integration tests, would it just re-run the entire story suite as it
> really doesn''t have a way to correlate source files to stories?  I
> presume the latter.
>
> I guess what I''d wonder, in terms of dealing with the speed issue
is:
>
> - Can I run a single story via TextMate like I do with specs?  That
> would allow me to concentrate on one particular thing if I needed
> quick turn around time.
> - I just always have autotest running, and most of the time, I just
> let it do its thing, listening for any failures (I use Mac''s
speech to
> alert me :)  So, the speed isn''t so bad as I typically just keep
> working until I hear a problem, then go fix that.
> - I am a huge fan of CI servers, and have one setup for every project
> I control.  I do like to know my suite passes before I push my changes
> up, but the CI server is a great way to ensure everything passes on a
> clean system.  I suppose if the story runner was really taking very
> long times, that I''d be ok pushing and seeing if the CI server
caught
> a problem.
>
> I''d like to hear other''s thoughts as well, but your
experience seems
> consistent with my theory of the direction I want to go: unit test
> models and helpers, drop controller and view tests in favor of Webrat
> + RailsStory, and then, as needed drop in RSpec view tests for things
> that can''t be as efficiently/effectively tested in stories.
>
> Final bit/question (for now ;-)...  With Webrat, does it pay attention
> to whether or not div''s and such are hidden/displayed in terms of
> whether it lets you enter values in them?  In this particular project
> I use a fair bit of disclosure style UI, where some fields are shown
> after a use takes some other action or presses a button, etc.  But
> they''re all there, just not visible.  So, wondering if I can
simply
> tell Webrat to put data into them, and not worry about having to
> effect a button press that does its work via JavaScript.  I can move
> to Selenium to handle that, but that''s actually something
I''m not
> wanting to do yet (I''ve used it in the past, and it''s
just a ton of
> overhead, truly slow, etc.).
>
> On Thu, Jun 19, 2008 at 8:41 AM, Kyle Hargraves <philodespotos at
gmail.com> wrote:
>> On Thu, Jun 19, 2008 at 9:31 AM, Christopher Bailey
>> <chris at cobaltedge.com> wrote:
>>> Specifically, I''m wondering, or contemplating, if I do
unit tests for
>>> my models, and then I use WebRat plus RailsStory, do I even need to
>>> then do functional testing of my controllers and views?  I can see
>>> that I might want to do some particular view testing that ensured
>>> certain elements were on a page or something, but in terms of
actually
>>> testing the workings of my app/site, it seems model unit tests
>>> combined with RailsStory+Webrat will do a pretty darn thorough job.
I
>>> would then consider going to Selenium to test some of the detailed
>>> JavaScript as needed.
>>>
>>> Thoughts?  What am I missing, or what are the downsides of such an
approach?
>>
>> In the project I''m working on, we lean pretty heavily on
model/helper
>> specs and webrat-driven plain text stories.
>>
>> We still do *some* controller specs, but only in those instances where
>> we deviate from the standard REST design, or to test authentication
>> requirements. With a custom it_should_... example generator, that can
>> be as simple as:
>>
>> describe FooController do
>>  describe "GET :index" do
>>    it_should_require_admin_access(:get, :index)
>>  end
>> end
>>
>> We currently have no view specs at all, though I am beginning to
>> regret that decision somewhat. Initially we avoided them because the
>> effort required to mock everything out wasn''t jiving well with
rapid
>> prototyping, but I''m considering going back and pushing some
things
>> into view specs using stub_model(). There are a lot of simple
>> requirements that are a good fit for view specs which we are currently
>> doing in the user stories, solely because the infrastructure was
>> already there.
>>
>> As far as controllers go, I wouldn''t say we''ve seen
any downside to
>> relying on stories instead of isolated specs; I''m sure
you''ve seen how
>> 90% of every controller spec ends up looking the same, and they start
>> to be a headache to maintain. Maybe the shoulda-esque rspec plugins
>> out there would help, but when I can easily just exercise the full
>> stack, I haven''t bothered to try. I saw one big *win* of not
testing
>> controllers in isolation just this week, though, when I had to shift a
>> lot of an application''s view paths around; with standard
controller
>> specs, I would''ve been spending a lot of typing updating the
render
>> expectations, but with stories it wasn''t an issue: the right
markup
>> either showed up or didn''t.
>>
>> The primary downside for us has been speed; it takes a few minutes to
>> run the full story suite. The ability to selectively run just a few
>> stories, and then a good CI server, alleviates that for the most part.
>>
>> I always ramble too much so I''ll stop here. HTH,
>>
>> k
>> _______________________________________________
>> rspec-users mailing list
>> rspec-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/rspec-users
>>
>
>
>
> --
> Christopher Bailey
> Cobalt Edge LLC
> http://cobaltedge.com
>


-- 
Christopher Bailey
Cobalt Edge LLC
http://cobaltedge.com

Christopher Bailey wrote:
> Just found answer to one of my questions below on being able to run
> stories in TextMate.  The new bundle for stories looks cool:
> http://github.com/bmabey/rspec-story-tmbundle/tree/master
>
>   
Hey Chris,
There is actually a bug for running the stories that accidentally got 
introduced recently... We''ll be pushing the fix out to github soon.
We are also adding more flexibility to how you can structure your 
stories directories. Anyways, look for that as the running is currently 
broken.

-Ben

On 6/19/08, Christopher Bailey <chris at cobaltedge.com>
wrote:
>  Specifically, I''m wondering, or contemplating, if I do unit tests
for
>  my models, and then I use WebRat plus RailsStory, do I even need to
>  then do functional testing of my controllers and views?  I can see
>  that I might want to do some particular view testing that ensured
>  certain elements were on a page or something, but in terms of actually
>  testing the workings of my app/site, it seems model unit tests
>  combined with RailsStory+Webrat will do a pretty darn thorough job.  I
>  would then consider going to Selenium to test some of the detailed
>  JavaScript as needed.
>
>  Thoughts?  What am I missing, or what are the downsides of such an
approach?
Having spent the last 3 months working on a project which heavily
relied on Story Driven Development (currently 78 story files which
encompass 401 scenarios) one thing that I have found is that the build
has become quite slow.  We made a decision early on that we wouldn''t
bother with view specs, as they felt redundant with our stories.
Looking back now I think that was a mistake.  If I were to do it over
I would leverage stories for the happy path, as well as to catch
expected error conditions, but I would also rely heavily on
Controller/View/Helper specs to test all of the branches and abnormal
use cases.

Basically I think that you should test as much stuff at the unit level
as possible, and leverage the integration tools where appropriate, but
don''t overuse without expecting some decrease in performance.

-- 
Josh Knowles
phone: 509-979-1593
email:  joshknowles at gmail.com
web:    http://joshknowles.com

On Thu, Jun 19, 2008 at 11:56 AM, Christopher Bailey
<chris at cobaltedge.com> wrote:
> Kyle, thanks much for sharing your experience.  You mention the speed
> and so on.  I''ve read that it is slow.  Question: does Autotest
work
> the same way with stories, or have a way to detect what file(s)
> changed and run the appropriate stories, or because they''re
> integration tests, would it just re-run the entire story suite as it
> really doesn''t have a way to correlate source files to stories?  I
> presume the latter.
>
> - Can I run a single story via TextMate like I do with specs?  That
> would allow me to concentrate on one particular thing if I needed
> quick turn around time.
> - I just always have autotest running, and most of the time, I just
> let it do its thing, listening for any failures (I use Mac''s
speech to
> alert me :)  So, the speed isn''t so bad as I typically just keep
> working until I hear a problem, then go fix that.
I have not attempted to set up autotest + stories, but it should be
possible to set up autotest to watch stories/**/*.story and rerun the
corresponding story when *that* changes; like you say, it''d be
difficult to map implementation files to their stories.

I do miss autotest at times, but up+enter on your console to rerun
some focused stories is not bad at all. I use a command-line runner
without the maintenance requirements of the *.rb runner files
(http://github.com/pd/story), so I can pretty easily select the
individual foo.story, a full feature, or any mix I want, then just
rerun those when I''m ready. Actually, I save files so often that
waiting on the stories to catch up would probably be annoying.
> I''d like to hear other''s thoughts as well, but your
experience seems
> consistent with my theory of the direction I want to go: unit test
> models and helpers, drop controller and view tests in favor of Webrat
> + RailsStory, and then, as needed drop in RSpec view tests for things
> that can''t be as efficiently/effectively tested in stories.
As Josh Knowles also said, dropping the unit level specs has turned
out to be a mistake. I wouldn''t say it''s a BIG mistake, and
the main
issue is just waiting too damned long before you are comfortable
pushing your code (which leads to saying "ah screw it" and then
"ah
crap it failed on CI" and then ...). But a mistake nonetheless.

If I started a fresh project today, I would still probably ignore view
specs early on; they''re cumbersome and take a lot of tweaking when
you''re just barely figuring out what it is you want to accomplish.
Once your ideas solidify a bit, tho, view specs using stub_model()
seem perfectly good these days. For controller specs, I think I''d
continue to do exactly what I''m already doing: spec the deviations
from the pretty-much-scaffolding REST controllers, and wrap up
authentication/authorization specs in a tidy DSL to generate that
stuff for me. Thin controllers deserve thin controller specs! =)
> Final bit/question (for now ;-)...  With Webrat, does it pay attention
> to whether or not div''s and such are hidden/displayed in terms of
> whether it lets you enter values in them?  In this particular project
> I use a fair bit of disclosure style UI, where some fields are shown
> after a use takes some other action or presses a button, etc.  But
> they''re all there, just not visible.  So, wondering if I can
simply
> tell Webrat to put data into them, and not worry about having to
> effect a button press that does its work via JavaScript.  I can move
> to Selenium to handle that, but that''s actually something
I''m not
> wanting to do yet (I''ve used it in the past, and it''s
just a ton of
> overhead, truly slow, etc.).
Yes, I take extensive advantage of that. So you can click
''edit'' next
to something on the screen, which unhides a form, but the form is
already on the page; webrat is entirely unaware of the CSS and will
just fill it out and submit it. It''s a minor headache when there are
10 ''edit'' links and 10 corresponding forms, cuz
you''ll need to dig
through and be sure to fill out the right fields. Other than that,
tho, nothin but sunshine.

k