rspec users - Jan 2009 - [Cucumber, BDD] When not to use Cucumber?

Hi all,
Just wanted to get some feedback and opinions on something...  I have 
been using Cucumber/Story Runner since plaintext support was introduced. 
(So, since October/November of 2007.)  Throughout this time I have used 
it on various projects.  Some projects had customer involvement in help 
writing the scenarios, some I mostly wrote the scenarios after gathering 
requirements and then got verification from the customer, and others 
involved no customer at all (i.e. I was the customer).  In all three 
cases I have gotten a lot of benefit from the upfront analysis that 
writing the feature forces you to do and the outside-in development that 
follows.  That said, the business analysis and outside-in development 
can be accomplished with any tool (i.e. rspec with webrat for webapps).  
In the past I have always defaulted on using the GWT (Given, When, Then) 
syntax for most of my acceptance tests, but that may be because I am so 
accustomed to that workflow by now.

So... I''m curious what people''s thoughts are.  When writing
acceptance
tests how do you choose which tool is best for the job?  I often hear 
people complaining about the GWT syntax and they see no benefit of it 
over the frameworks that provide contexts (i.e rspec, shoulda, etc).[1] 
In what cases, if any, do you think the GWT gets in the way and how?  In 
particular, if you are writing an app or lib whose customers/users are 
all developers do you still think there is value in the GWT syntax that 
Cucumber provides?

I have my own thoughts on the matter, but in playing devil''s advocate I
want to get other people''s thoughts before I skew you with mine. :)

Thanks,
Ben

[1] http://www.dcmanges.com/blog/practicality-of-the-rspec-story-runner

Ben Mabey wrote:
> 
> So... I''m curious what people''s thoughts are.  When
writing acceptance
> tests how do you choose which tool is best for the job?  I often hear
> people complaining about the GWT syntax and they see no benefit of it
> over the frameworks that provide contexts (i.e rspec, shoulda, etc).[1]
> In what cases, if any, do you think the GWT gets in the way and how?  In
> particular, if you are writing an app or lib whose customers/users are
> all developers do you still think there is value in the GWT syntax that
> Cucumber provides?
> 
My own experience is quite limited, but I did start out trying to get 
what I desired done through TestUnit, then RSpec, then RSpec Story 
Runner and finally Cucumber. In the end it was Cucumber that clarified 
my approach for me, mainly by clearing away all (or at least most) of 
the nitty gritty details of configuring and using the tool itself.  I 
found that, coming from a non OO programming background, that the degree 
of mastery required of the other tools mentioned, together with the 
effort I had to make to get the tool to behave as I wished (desires that 
were often malformed by lack of understanding in any case), distracted 
considerably from the task at hand, programming the application.

I have progressed on my current project far, far more since discovering 
Cucumber this past November than in all the time before.  The 
bifurcation between what and how seems far more natural to me with the 
Cucumber features/step definitions file split than with the unified 
expectation method/test steps of RSpec and the test method/assert 
statements of TestUnit.  Cucumber feature files just seem to permit one 
to concentrate solely on "what" in the one case and then, with the 
separate step definitions file, deal with the "how" in the other, 
without crossing the line between the two.

Given, When and Then seemed contrived and far too stiff to me to begin 
with.  However, I discovered that much of my initial discomfort arose 
from too vague a conception of what is was that was desired.  The 
benefit of GWT stylistic conventions is that it really does make one 
consider deeply where one is starting from, what one is actually trying 
to accomplish, and how, exactly, does one tell if it has been 
accomplished.

If this sort of self-analysis is so ingrained that one need not think of 
it then GWT probably does not provide a measurable benefit.  So, perhaps 
deeply experienced developers and programmers might not get as much from 
it as I have.

On the other hand, I have frequently discovered in myself a lamentable 
tendency to believe that I understood what a problem was simply because 
I wanted to believe and not because of any demonstrable evidence to that 
effect. Sitting down and writing out what you believe to be the truth 
often reveals the flaws in ones assumptions.  The GWT litany seems to 
provide a very useful, if formulaic, examination of exactly what one is 
setting out to accomplish thereby sometimes raising the question of why 
it needs to be done at all.  I find that it thereby clears away much 
that is irrelevant to the question at hand.

I also hold to the opinion that when writing an end user application the 
only testing that can be consider definitive is what is described as 
integration or acceptance testing.  I accept that there are pragmatic 
reasons why, because of limited time and equipment resources, unit and 
functional tests are viewed as attractive alternatives.  But, that 
should not blind one the the fact that they really are alternatives, 
chosen because of environmental considerations and not because of 
inherent superiority.  The only test that means anything to an end user 
is one that shows the application works as intended in the environment 
it is deployed to.

Libraries and utilities have a different audience and their testing is 
perhaps better handled solely in unit and functional tests, but I cannot 
say so with any conviction.
-- 
Posted via http://www.ruby-forum.com/.

> My own experience is quite limited, [...]
Hi James, are you saying that you only use Cucumber and not RSpec at 
all?
-- 
Posted via http://www.ruby-forum.com/.

Fernando Perez wrote:
>> My own experience is quite limited, [...]
> 
> Hi James, are you saying that you only use Cucumber and not RSpec at 
> all?
No, I do use RSpec and TestUnit for that matter, but not to any extent 
at the present time.  I find that cucumber features are much more 
conducive to driving out the design than RSpec expectations.  Once the 
essence of the application is laid out and working then, if necessary, I 
will consider adding more technical type expectations via RSpec.

Frankly, at the moment, I do not see a compelling case to move from step 
definitions into formal RSpec expectations.  Again, my experience in 
these matters is limited and perhaps as it grows my appreciations will 
change as well.

I do note however, that once one finds a tool that suits ones needs then 
it is somewhat inefficient to use another without some compelling 
advantage to be gained.  What advantage does formal RSpec constructions 
provide over cucumber step definitions using RSpec and RSpec-Rails 
methods?
-- 
Posted via http://www.ruby-forum.com/.

On 23/01/2009, at 12:04 PM, Ben Mabey wrote:
> So... I''m curious what people''s thoughts are.  When
writing
> acceptance tests how do you choose which tool is best for the job?   
> I often hear people complaining about the GWT syntax and they see no  
> benefit of it over the frameworks that provide contexts (i.e rspec,  
> shoulda, etc).[1] In what cases, if any, do you think the GWT gets  
> in the way and how?  In particular, if you are writing an app or lib  
> whose customers/users are all developers do you still think there is  
> value in the GWT syntax that Cucumber provides?
I''ve been using RSpec since July, and love it. For the last few  
months, I''ve been reading most of the Cucumber posts on this list.  
Last week, I thought I''d give Cucumber a try. It feels like a really  
fantastic tool for determining business requirements, functionality,  
etc. From what I can tell though, there seems to be a lot of overlap  
between RSpec specs, and Cucumber features and stories. For that  
reason, I haven''t delved much further into Cucumber, because I
don''t
want to be repeating myself, and having to update two sets of files  
whenever I make a change.

However, if that''s not the case, please correct me!
Nick

I''ve been using story runner for the first time over the past 4 months
or so
and I''ve found the invaluable part to be the plain text support. Having
a
shared artifact that can be used by the customer, developers and testers is
a huge boost towards achieving ubiquitous language. We use the story kickoff
discussion to refine the language used in each scenario together, ensuring
that the language makes sense to everyone and is consistent with other
stories.

In the sense that we want a way to store the feature/story in an executable
format that we can all understand, a tool that supports plain text stories
is the best choice for us. You can implement a scenario just as easily using
plain rspec but then it becomes less accessible to other people in the team.


If I was choosing a testing approach for a project that had technical people
as customers I would still use the GWT format. I think it helps you
continually focus on what you are really trying to achieve, otherwise it can
be too easy to burrow down into the technical details and lose sight of
that. No matter what the nature of project, every thing that you do is
driven by the goal of adding some sort of value. Continually keeping that
goal in mind helps you make better choices along the way.

I agree with James that having that separation between the "what" and
"how"
is very useful. This approach is amenable to a more black box style of
testing appropriate for acceptance tests. You should theoretically be able
to completely change how your system works without touching the
story/feature text. Your mileage would of course vary with this depending on
whether or not you use a more declarative or imperative approach (as Ben
describes in
http://www.benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories/
).

My current recipe for testing spread is to automate the acceptance scenarios
using a tool like story runner/cucumber and then use specs for targeted
integration and unit level testing. The automated acceptance scenarios cover
the main scenarios that the customer cares about while more specialised
cases (or more technical aspects) are covered at the spec level. The specs
will typically gravitate towards the white box end of the scale (describing
the internals in more detail).

- Brent
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20090124/cf69431b/attachment-0001.html>

Brent Snook wrote:
> 
> My current recipe for testing spread is to automate the acceptance
> scenarios
> using a tool like story runner/cucumber and then use specs for targeted
> integration and unit level testing. The automated acceptance scenarios
> cover
> the main scenarios that the customer cares about while more specialised
> cases (or more technical aspects) are covered at the spec level. The specs
> will typically gravitate towards the white box end of the scale
> (describing
> the internals in more detail).
> 

Hi, I think this is the most used approach nowadays, high level testing with
features and more specific testing, that end users don''t care about at
the
spec level.  However, I prefer the distinction between, declarative stories
and imperative stories, which I think can be applied to these two levels
also.

For example, the end user or the business person, most probably gets bored
of reading a very long imperative story of clicking links and filling out
forms, plus other setup steps that might seem unnecessary to him, but are
needed sometimes due to a not fully correct design. Whilst a developer will
appreciate an imperative story as it tells him/her exactly what to do.

In my opinion, even at the spec level we can use the philosophy behind steps
(re-usability) to reduce the redundancies inherent to spec definitions. 
Trivial cases of you should see this in the page (view specs), or this
object should have this attribute set to true (model specs), happen all the
time, and seem like a great place to use variables inside step definitions.  

Other cases are more complex and the use of regular expressions to
encapsulate them all, can certainly be very difficult, maybe the type of an
attribute is a datetime and needs to be parsed, thus not being completely
re-usable to say a boolean field. 

Still we could account for this variations by checking the type of the
attribute, at the expense of increasing the complexity of the step.  This is
where the argument continues, people support the view of writing repetitive
specs considering that they are clearer.  I still have doubts about this. 
In my opinion this will certainly be a complex problem to solve, but once we
tackle it a number of times, it will become trivial, we''ll figure out
an
elegant step_helper to deal with this situations, and move on to the next
hard problem.

The hardest problem seems to be agreeing on a language to write the stories
including minimal variations in the syntax and grammar of our steps, to be
able to come up with a re-usable set of step definitions out-of-the-box, a
set of steps that will allow us to write stories and jump straight to the
failing nature of a step, due to it not being implemented, once its
implemented the step will be passing, without further need to write trivial
step definitions.  This would be a great time saver and allow us to
concentrate on harder problems very specific to our application that are not
accounted for in the general step definitions.

In conclusion, when not to use cucumber? never.  Always use cucumber,
everyone always use cucumber, lets find this hard problems to re-use steps
and tackle them together, find a solution and move on to the next
road-blocker.  Use Rspec but as the underlying language to make cucumber
work, not to divide your application into model view controller specs,
separate your tests into user and developer features/steps.  
Distributed testing is a must to use cucumber effectively, and the use of
mocks is also unrefutable[1], but only as an alternative to an inability to
test something in a classical way[2] 

Rai

[1]http://rubyconf2008.confreaks.com/testing-heresies.html
[2]http://martinfowler.com/articles/mocksArentStubs.html


-- 
View this message in context:
http://www.nabble.com/-Cucumber%2C-BDD--When-not-to-use-Cucumber--tp21628693p21639914.html
Sent from the rspec-users mailing list archive at Nabble.com.

See comments inline...

On 1/23/09 1:46 PM, James Byrne wrote:
> Ben Mabey wrote:
>
>    
>> So... I''m curious what people''s thoughts are.  When
writing acceptance
>> tests how do you choose which tool is best for the job?  I often hear
>> people complaining about the GWT syntax and they see no benefit of it
>> over the frameworks that provide contexts (i.e rspec, shoulda, etc).[1]
>> In what cases, if any, do you think the GWT gets in the way and how? 
In
>> particular, if you are writing an app or lib whose customers/users are
>> all developers do you still think there is value in the GWT syntax that
>> Cucumber provides?
>>
>>      
>
> My own experience is quite limited, but I did start out trying to get
> what I desired done through TestUnit, then RSpec, then RSpec Story
> Runner and finally Cucumber. In the end it was Cucumber that clarified
> my approach for me, mainly by clearing away all (or at least most) of
> the nitty gritty details of configuring and using the tool itself.  I
> found that, coming from a non OO programming background, that the degree
> of mastery required of the other tools mentioned, together with the
> effort I had to make to get the tool to behave as I wished (desires that
> were often malformed by lack of understanding in any case), distracted
> considerably from the task at hand, programming the application.
>    
Interesting, thanks for sharing what your background is.  I have seen 
other developers who don''t have a strong OO background struggle with
the
concepts of isolation have a really hard time understanding how and why 
to use rspec.  However, they seem to love Cucumber and it makes total 
sense to them.
> I have progressed on my current project far, far more since discovering
> Cucumber this past November than in all the time before.  The
> bifurcation between what and how seems far more natural to me with the
> Cucumber features/step definitions file split than with the unified
> expectation method/test steps of RSpec and the test method/assert
> statements of TestUnit.  Cucumber feature files just seem to permit one
> to concentrate solely on "what" in the one case and then, with
the
> separate step definitions file, deal with the "how" in the other,
> without crossing the line between the two.
>    
Ahh.. that is a very good observation.  When I write examples in rspec I 
start out with the example description first (obviously) but then I like 
to jump straight to my expectation.  Meaning, I will say something like 
"dog.should be_happy" before I have even created the dog object.  I
like
to do this for the same reason- I first write what I want and then I go 
a line above and create the how/given.  A lot of people complain about 
the separation of code form story in plaintext features, but as you have 
pointed out that separation can be a very good thing.
> Given, When and Then seemed contrived and far too stiff to me to begin
> with.  However, I discovered that much of my initial discomfort arose
> from too vague a conception of what is was that was desired.  The
> benefit of GWT stylistic conventions is that it really does make one
> consider deeply where one is starting from, what one is actually trying
> to accomplish, and how, exactly, does one tell if it has been
> accomplished.
>    
> If this sort of self-analysis is so ingrained that one need not think of
> it then GWT probably does not provide a measurable benefit.  So, perhaps
> deeply experienced developers and programmers might not get as much from
> it as I have.
>
> On the other hand, I have frequently discovered in myself a lamentable
> tendency to believe that I understood what a problem was simply because
> I wanted to believe and not because of any demonstrable evidence to that
> effect. Sitting down and writing out what you believe to be the truth
> often reveals the flaws in ones assumptions.  The GWT litany seems to
> provide a very useful, if formulaic, examination of exactly what one is
> setting out to accomplish thereby sometimes raising the question of why
> it needs to be done at all.  I find that it thereby clears away much
> that is irrelevant to the question at hand.
>    
+1.  I totally agree with this.  When it is difficult for me to write 
scenarios I find that the reason is that I fail to really understand 
what I am trying to accomplish.  By flushing the acceptance criteria out 
in the GWT language I come out with a much better understanding of the 
problem and know exactly where to begin.
> I also hold to the opinion that when writing an end user application the
> only testing that can be consider definitive is what is described as
> integration or acceptance testing.  I accept that there are pragmatic
> reasons why, because of limited time and equipment resources, unit and
> functional tests are viewed as attractive alternatives.  But, that
> should not blind one the the fact that they really are alternatives,
> chosen because of environmental considerations and not because of
> inherent superiority.  The only test that means anything to an end user
> is one that shows the application works as intended in the environment
> it is deployed to.
>    
So here is where I disagree with you. :)  I do agree that all the user 
cares about in the end is that the desired outcome is reached and the 
business value is provided.  So with that mindset features are the only 
kind of tests a customer really cares about.  Unit tests however are 
more for the developer.  They aid in design and provide much more 
detailed documentation than the customer testes (acceptance tests).  I 
don''t think that cucumber features help me design my objects that much 
better unless I have focused object level examples telling me where the 
pain points are in my design.  Eventually the customer will be greatful 
that you have unit tests (even if they don''t know about them) because 
they will help lower the maintenace of the project by encouraging better 
design and providing tighter and more focused feedback on the smaller 
parts of the system.  IMHO, of course.


Thanks for the feedback,
Ben

On 1/23/09 10:04 PM, Brent Snook wrote:

*snip*
>
> If I was choosing a testing approach for a project that had technical 
> people as customers I would still use the GWT format. I think it helps 
> you continually focus on what you are really trying to achieve, 
> otherwise it can be too easy to burrow down into the technical details 
> and lose sight of that. No matter what the nature of project, every 
> thing that you do is driven by the goal of adding some sort of value. 
> Continually keeping that goal in mind helps you make better choices 
> along the way.
This is at the herat of what I was asking.  Even if every "customer"
of
the project is fluent in ruby would you still bother with the plaintext 
GWT features of Cucumber?  I think you have nailed it.. Or at least, 
have confirmed my thoughts on the matter. :)  Business value is business 
value no matter who is the customer.  While other tools can be used to 
automate acceptance criteria Cucumber provides the huge advantage of 
helping you focus and analyze on what the desired business value is.  I 
have talked to other people who have claimed that the GWT gets in the 
way though, so YMMV.  So, I wouldn''t be surprised that in some
technical
domains the overhead of adding features just doesn''t make sense.

Thanks for your thoughts,
Ben

Hi,

I''ve been using cucumber, and story runner before that since plain  
text stories were introduced 1.5 years ago. I also use rspec for  
model and controller specing/testing. I''ll answer the opposite  
question: when do I use cucumber?

I am a team of one, and I am my own client, so the way I use cucumber  
may be different than other people''s experience. As outlined in my  
recent presentation (http://www.slideshare.net/linoj/cucumber-how-i- 
slice-it-presentation-924254 ) I''ve found myself using cucumber for:

* Talking to myself (planning)
* Keeping track of what I''m working on (outlining)
* Automating my manual testing (scripting)
* View specs
* Regression testing (confidence)

Plain text stories let me plan out my app from the end-user point of  
view. When I''m my own client, Given/When/Then helps me structure  
conversations with myself. Scenarios let me explore variations.  
However, I do not shoot for full coverage or many edge cases, I leave  
that for the controller and model specs. These tend to be short  
atomic G/W/T scenarios.

When taking the BDD approach (which TBH is sometimes), it can take  
days to implement a single scenario, drilling down from feature to  
controller spec to model spec, etc, at which point the story provides  
a valuable context for what to work on next as i "pop the to-do  
stack" and work my way through the steps.

However, in reality, sometimes I jump in and implement stuff first,  
and test it as I go by clicking in the browser (oh my!). At a certain  
point I''m repeating the same set of clicks so I write a cucumber  
story to more easily script the scenario. In this case there may be a  
high level Given step(s), and a sequence of multiple When''s, with a  
few Then''s inbetween (to validate where I landed, flash messages, or  
whatever).

I''ve also started using cucumber in place of view specs. These  
scenarios will often be a simple Give/When with a whole bunch of  
Then''s that validate elements on the page.

By the way, I''ve written utility steps for viewing the response in a  
browser (http://www.vaporbase.com/postings/Show_me_the_response ),  
and I often use rdebug to work through my code.

Finally, I love to run my suite of scenarios as regression tests,  
especially at stabilization points in my development, to make sure  
everything is still operational. I even have a "cap deploy:features"  
to run them on my staging server.

linoj

--------------------------------------
               wtf <=> ftw
--------------------------------------

On 1/24/09 5:02 AM, voodoorai2000 wrote:
> In conclusion, when not to use cucumber? never.  Always use cucumber,
> everyone always use cucumber, lets find this hard problems to re-use steps
> and tackle them together, find a solution and move on to the next
> road-blocker.  Use Rspec but as the underlying language to make cucumber
> work, not to divide your application into model view controller specs,
> separate your tests into user and developer features/steps.
> Distributed testing is a must to use cucumber effectively, and the use of
> mocks is also unrefutable[1], but only as an alternative to an inability to
> test something in a classical way[2]
>
> Rai
>
> [1]http://rubyconf2008.confreaks.com/testing-heresies.html
> [2]http://martinfowler.com/articles/mocksArentStubs.html
>
>
>    
Well, I have learned never to say never. ;)  As far as the mocking 
goes...  I always use real objects in features unless I need to make an 
external network call to another system (i.e. a web services call, in 
which case I like to use fakeweb[1]).  That said, one of the things I 
really like about having features is that it allows me to use mock 
objects in my object level examples without any fear of integration 
point failures.  I find it very hard to test objects under total 
isolation without using mocks to remove dependencies and I gain a lot of 
design benefit from using mocks.  I really don''t want to get into the 
whole classic vs mockist debate though as it was just beat to death on 
the TDD mailing list.[2] :)

-Ben


[1] http://github.com/chrisk/fakeweb/tree/master
[2] http://tech.groups.yahoo.com/group/testdrivendevelopment/message/29674