I''m fairly new to cucumber and rspec but so far am falling in love with
both. I''ve read up on several different articles concerning these
testing tools including the ''beta'' BDD rspec/cucumber book.
I saw this thread here: http://www.ruby-forum.com/topic/183428
Which concerns how small or detailed specs and scenarios should be - and
the basic gist was to keep scenarios at a higher level and leave the
detailed form
fields etc to rspec specs.
This was all wonderfully insightful but it brings me to another
question. How detailed (or should they be included in cucumber tests at
all?) should the error path be? "Happy paths" are great to test, but
it''s also necessary to test error paths so that users aren''t
encountering rails stack traces, empty feedback, etc.
Should there be one scenario per "empty" field? Should there be one
scenario per validation check per field? Should it be condensed to a
single scenario that attempts to encompass all error paths?
I have one specific, seemingly overly complicated scenario that attempts
to go the one scenario per validation check per field route:
Scenario Outline: Add account with invalid fields
Given I am logged in as BudgetTest
And I am on the New Account page
When I fill in the form for account "<account_name>" with
valid data
And I submit the data "<value1>" for the
"<field_type1>" field,
"<field_name1>"
And I submit the data "<value2>" for the
"<field_type2>" field,
"<field_name2>"
And I press "Add Account"
Then I should see the rendered template for "the new account page"
And I should see an error indicating "<field_name>" was
"<error>"
I''ve removed the Scenarios: blocks because they would wordwrap and
look
terrible/undreadable. Following this is two sets of scenarios:
Scenarios: missing required fields
Scenarios: submitting bad data
Some of the fields compare data with each other to determine validity
which is why there''s two data entries in the scenario outline. If the
second is left blank then the defaults that were set in "When I fill in
the form..." are used for it. Each "Scenarios" block contains a
table
with allll of the fields defined by <> in the outline. As you can see,
it seems to me to be overly complicated, overly verbose, and perhaps
doing more than it should be.
I think maybe this test goes overboard...but what level of detail is
good for error-path testing?
--
Posted via http://www.ruby-forum.com/.
Nathan Benes wrote:> I''m fairly new to cucumber and rspec but so far am falling in love with > both. I''ve read up on several different articles concerning these > testing tools including the ''beta'' BDD rspec/cucumber book. > > I saw this thread here: http://www.ruby-forum.com/topic/183428 > > Which concerns how small or detailed specs and scenarios should be - and > the basic gist was to keep scenarios at a higher level and leave the > detailed form > fields etc to rspec specs. > > This was all wonderfully insightful but it brings me to another > question. How detailed (or should they be included in cucumber tests at > all?) should the error path be? "Happy paths" are great to test, but > it''s also necessary to test error paths so that users aren''t > encountering rails stack traces, empty feedback, etc. > > Should there be one scenario per "empty" field? Should there be one > scenario per validation check per field? Should it be condensed to a > single scenario that attempts to encompass all error paths? > > I have one specific, seemingly overly complicated scenario that attempts > to go the one scenario per validation check per field route: > > Scenario Outline: Add account with invalid fields > Given I am logged in as BudgetTest > And I am on the New Account page > When I fill in the form for account "<account_name>" with valid data > And I submit the data "<value1>" for the "<field_type1>" field, > "<field_name1>" > And I submit the data "<value2>" for the "<field_type2>" field, > "<field_name2>" > And I press "Add Account" > Then I should see the rendered template for "the new account page" > And I should see an error indicating "<field_name>" was "<error>" > > > I''ve removed the Scenarios: blocks because they would wordwrap and look > terrible/undreadable. Following this is two sets of scenarios: > > Scenarios: missing required fields > > Scenarios: submitting bad data > > Some of the fields compare data with each other to determine validity > which is why there''s two data entries in the scenario outline. If the > second is left blank then the defaults that were set in "When I fill in > the form..." are used for it. Each "Scenarios" block contains a table > with allll of the fields defined by <> in the outline. As you can see, > it seems to me to be overly complicated, overly verbose, and perhaps > doing more than it should be. > > I think maybe this test goes overboard...but what level of detail is > good for error-path testing? >Hi Nathan, I think testing all of the validations from Cucumber is going overboard *unless* the customer wants to see them there. I have done something like that but it was when I was writing my own validation logic (in this case I wasn''t even using AR or another standard library). It was a nice way of showing the customer what were and were not valid values. However, if you are using ActiveRecord to do standard validations then I see little value in checking each validation separately. It will take a long time to run and it seems like it would really not add much value in terms of regression catching beyond what the fast model-level code examples would provide. So, like I said, the only value I would see in that would be for a customer to see that it is working as they want it to. In general, I like to keep happy paths in Cucumber features along with some special error cases. Driving though the entire stack is a double-edged sword. It provides great regression testing and is an excellent way to frame scenarios for customers. However, the number of execution paths you can execute from such a high level is inherently limited... the number of objects and all of there potential states in concert with one another results in a combinatorial explosion of number of test cases required to test everything. That is why isolated specs on a lower level are still very valuable and for me provide a great deal of confidence when used along side some happy-path full-stack tests. (And even then you can never test everything...) Thats my current take on things at least... HTH, Ben
On 24 Jul 2009, at 05:59, Nathan Benes wrote:> I''m fairly new to cucumber and rspec but so far am falling in love > with > both. I''ve read up on several different articles concerning these > testing tools including the ''beta'' BDD rspec/cucumber book. > > I saw this thread here: http://www.ruby-forum.com/topic/183428 > > Which concerns how small or detailed specs and scenarios should be - > and > the basic gist was to keep scenarios at a higher level and leave the > detailed form > fields etc to rspec specs. > > This was all wonderfully insightful but it brings me to another > question. How detailed (or should they be included in cucumber > tests at > all?) should the error path be? "Happy paths" are great to test, but > it''s also necessary to test error paths so that users aren''t > encountering rails stack traces, empty feedback, etc. > > Should there be one scenario per "empty" field? Should there be one > scenario per validation check per field? Should it be condensed to a > single scenario that attempts to encompass all error paths? > > I have one specific, seemingly overly complicated scenario that > attempts > to go the one scenario per validation check per field route: > > Scenario Outline: Add account with invalid fields > Given I am logged in as BudgetTest > And I am on the New Account page > When I fill in the form for account "<account_name>" with valid > data > And I submit the data "<value1>" for the "<field_type1>" field, > "<field_name1>" > And I submit the data "<value2>" for the "<field_type2>" field, > "<field_name2>" > And I press "Add Account" > Then I should see the rendered template for "the new account page" > And I should see an error indicating "<field_name>" was "<error>" > > > I''ve removed the Scenarios: blocks because they would wordwrap and > look > terrible/undreadable. Following this is two sets of scenarios: > > Scenarios: missing required fields > > Scenarios: submitting bad data > > Some of the fields compare data with each other to determine validity > which is why there''s two data entries in the scenario outline. If the > second is left blank then the defaults that were set in "When I fill > in > the form..." are used for it. Each "Scenarios" block contains a table > with allll of the fields defined by <> in the outline. As you can > see, > it seems to me to be overly complicated, overly verbose, and perhaps > doing more than it should be. > > I think maybe this test goes overboard...but what level of detail is > good for error-path testing?The question is difficult to answer as you''ll eventually find your own balance with this from experience. As Ben said, Cucumber scenarios are best applied for ''broad brush'' scenarios that stakeholders care about. Although the stakeholders obviously care that this form renders error messages when invalid values are submitted, you may well start to bore them when you get into the level of detail of specifying each field and the error message that goes along with it. So in this instance, it might well be OK to have a cuke that tells you whether or not validation is being invoked at all, and then rely on specs to detail the way that validation is working. Equally, your stakeholder / customer may be the type of person who will trust you better if they can see all the edge cases being explored in the cukes. So it depends a lot on the context you''re working in. If so you can use some of the features of Cucumber like scenario outlines and step tables to make these kind of repetitive cukes more readable. I''d advise you do go and check out those features on the wiki. Finding and testing for failure cases is something that most teams are pretty bad at - unless we have someone of that ''constructive destruction'' mindset on the team, we often miss these scenarios the first time we build a feature. Steve McConnell''s ''Code Complete'' reckons on something like a 5:1 ratio of sad:happy path tests being produced by mature test teams, so you''re on the right lines by thinking about them! What you might find is that writing high-level specs (i.e. cukes) for the more obvious failure cases will mean you are distracted from seeing other ones. What if someone tries to type some javascript or HTML into one of these boxes, for example? cheers, Matt +447974 430184 matt at mattwynne.net http://mattwynne.net
Ben Mabey wrote:> Nathan Benes wrote: >> I''m fairly new to cucumber and rspec but so far am falling in love with >> both. I''ve read up on several different articles concerning these >> testing tools including the ''beta'' BDD rspec/cucumber book. >> >> I saw this thread here: http://www.ruby-forum.com/topic/183428 >> >> Which concerns how small or detailed specs and scenarios should be - and >> the basic gist was to keep scenarios at a higher level and leave the >> detailed form >> fields etc to rspec specs. >> >> This was all wonderfully insightful but it brings me to another >> question. How detailed (or should they be included in cucumber tests at >> all?) should the error path be? "Happy paths" are great to test, but >> it''s also necessary to test error paths so that users aren''t >> encountering rails stack traces, empty feedback, etc. >> >> Should there be one scenario per "empty" field? Should there be one >> scenario per validation check per field? Should it be condensed to a >> single scenario that attempts to encompass all error paths? >> >> I have one specific, seemingly overly complicated scenario that attempts >> to go the one scenario per validation check per field route: >> >> Scenario Outline: Add account with invalid fields >> Given I am logged in as BudgetTest >> And I am on the New Account page >> When I fill in the form for account "<account_name>" with valid data >> And I submit the data "<value1>" for the "<field_type1>" field, >> "<field_name1>" >> And I submit the data "<value2>" for the "<field_type2>" field, >> "<field_name2>" >> And I press "Add Account" >> Then I should see the rendered template for "the new account page" >> And I should see an error indicating "<field_name>" was "<error>" >> >> >> I''ve removed the Scenarios: blocks because they would wordwrap and look >> terrible/undreadable. Following this is two sets of scenarios: >> >> Scenarios: missing required fields >> >> Scenarios: submitting bad data >> >> Some of the fields compare data with each other to determine validity >> which is why there''s two data entries in the scenario outline. If the >> second is left blank then the defaults that were set in "When I fill in >> the form..." are used for it. Each "Scenarios" block contains a table >> with allll of the fields defined by <> in the outline. As you can see, >> it seems to me to be overly complicated, overly verbose, and perhaps >> doing more than it should be. >> >> I think maybe this test goes overboard...but what level of detail is >> good for error-path testing? >> > > Hi Nathan, > I think testing all of the validations from Cucumber is going > overboard *unless* the customer wants to see them there. I have done > something like that but it was when I was writing my own validation > logic (in this case I wasn''t even using AR or another standard > library). It was a nice way of showing the customer what were and were > not valid values. However, if you are using ActiveRecord to do > standard validations then I see little value in checking each > validation separately. It will take a long time to run and it seems > like it would really not add much value in terms of regression > catching beyond what the fast model-level code examples would provide. > So, like I said, the only value I would see in that would be for a > customer to see that it is working as they want it to. > > In general, I like to keep happy paths in Cucumber features along with > some special error cases. Driving though the entire stack is a > double-edged sword. It provides great regression testing and is an > excellent way to frame scenarios for customers. However, the number > of execution paths you can execute from such a high level is > inherently limited... the number of objects and all of there potential > states in concert with one another results in a combinatorial > explosion of number of test cases required to test everything. That > is why isolated specs on a lower level are still very valuable and for > me provide a great deal of confidence when used along side some > happy-path full-stack tests. (And even then you can never test > everything...) > > Thats my current take on things at least... HTH, > > Ben >BTW, please use the cucumber mailing list for cucumber related questions: http://groups.google.com/group/cukes -Ben