I''ve been thinking about the role that stories played compared to specs, and that "should we use should" thread brought those thoughts up again. First I want to discuss whether or not specs are authoritative regarding the desired behavior of a system. I would say they''re a good approximation of how the system currently runs according to the developer''s understanding. I say they''re an approximation because if you use mocks everywhere (as I believe you ought to) then you''re not performing integrations and the specs are necessarily incomplete. Why does that idea matter? Traditionally, we''ve leaned heavily on a suite of unit tests to perform refactorings. In fact, Fowler says the first thing you have to do in order to refactor is grow a set of comprehensive unit tests. Refactoring, by definition, is changing the structure without changing the observable behavior. Implicit in using tests to enable refactoring is the idea that those tests precisely define what the behavior of a system should be. If we say that our specs are not actual specifications of desired behavior, then specs aren''t sufficient safety nets for refactoring. This mostly comes from my experience that refactoring is often a PITA when using mocks. Mocks isolate dependencies, which is great, and makes it easy to change code without affecting parts of the system that really shouldn''t be affected. However they make simple, standard refactorings like "move method" more difficult. If I say that stories are authoritative and specs aren''t, then that means I ought to rely on stories to provide the safety net for my refactoring. When taking those baby steps, it''s important to keep the stories green rather than the specs. Another result of this thinking is that more specification should be done in stories than I originally thought. Edge cases are my prime example. I used to think I should write a story that expresses the general desired behavior, and then cover any edge cases in specs. However edge cases should be covered by stories because stories define the desired behavior of the system. Also, edge cases presumably represent some business value to the customer. Finally, it''s better to keep all of the desired behavior specs in one place. Stories are naturally more verbose than specs though. That paragraph feels awkward to me, because I think I''m missing some vocabulary. It suggests to me that the collection of all user stories is the app specification - "spec" for short. What is the collection of all descriptions, the object-level specs? I''ve always just said "specs," but I don''t think that really fits anymore with my current thinking. Not to mention it''s just confusing if I also use "spec" to mean the collection of all user stories. I know that an individual describe block is a "description," but maybe the whole suite should be called "descriptions" instead of "specs?" What do you guys think? Is it useful to draw a line and say that stories are authoritative and specs aren''t? Do you think it becomes less crucial to use object-level specs to enable refactoring, and instead lean on stories more? Pat
On Nov 13, 2007 1:10 PM, Pat Maddox <pergesu at gmail.com> wrote:> I''ve been thinking about the role that stories played compared to > specs, and that "should we use should" thread brought those thoughts > up again. > > First I want to discuss whether or not specs are authoritative > regarding the desired behavior of a system. I would say they''re a > good approximation of how the system currently runs according to the > developer''s understanding. I say they''re an approximation because if > you use mocks everywhere (as I believe you ought to) then you''re not > performing integrations and the specs are necessarily incomplete. > > Why does that idea matter? Traditionally, we''ve leaned heavily on a > suite of unit tests to perform refactorings. In fact, Fowler says the > first thing you have to do in order to refactor is grow a set of > comprehensive unit tests. > > Refactoring, by definition, is changing the structure without changing > the observable behavior.IMO, that definition refers to a system, not individual objects. When you change objects, you usually have to change the examples. Keep in mind that that goes nearly unnoticed in java or .NET because of the refactoring tools available on those platforms. Change a method name once and it changes it everywhere, *including the examples*. Move a method from one object to another and you get a dialog that let''s you opt to leave the existing method in place delegating to the new location, or change all the consumers of that method in the system, *including the examples*, to use the new location.> Implicit in using tests to enable > refactoring is the idea that those tests precisely define what the > behavior of a system should be.Not so much about precision as it is about comprehensiveness.> If we say that our specs are not > actual specifications of desired behavior, then specs aren''t > sufficient safety nets for refactoring.Again - this word "specification" is causing trouble. Examples have three different roles. When you first write them, they are examples of how you want an object to behave. After you get them to pass, they become both executable documents of the system as it is and regression tests. In their role as regression tests, they serve the same role as unit tests do in refactoring.> This mostly comes from my experience that refactoring is often a PITA > when using mocks. Mocks isolate dependencies, which is great, and > makes it easy to change code without affecting parts of the system > that really shouldn''t be affected. However they make simple, standard > refactorings like "move method" more difficult.Again - this is a solved problem in other platforms and will eventually be solved in ours.> If I say that stories are authoritative and specs aren''t, then that > means I ought to rely on stories to provide the safety net for my > refactoring. When taking those baby steps, it''s important to keep the > stories green rather than the specs.On some level that makes sense - but it''s going to slow you down quite a bit when you start building up a significant suite of stories. The way I''ve always dealt with this was to rely on the examples to get me through a refactoring, and then run the stories before committing. That allows me to stay focused on small problems and progress steadily. By the way, when I say "always dealt with this", I''m talking about some years of experience using FitNesse for stories and xUnit for examples. So although the Story Runner is new, the idea of having two separate environments - one for system-level examples and one for object-level examples - has been around for a while.> Another result of this thinking is that more specification should be > done in stories than I originally thought. Edge cases are my prime > example. I used to think I should write a story that expresses the > general desired behavior, and then cover any edge cases in specs.I think they should be covered in both stories and examples. The thing is that they get covered in different ways. For one, the language in which they are expressed is different - because the stories are about the system and the examples are about the objects in the system. And there will be some duplicated effort between the stories and the examples - and that is a GOOD thing. Additionally, when dealing with stories, you may specify that a particular error message shows up on the screen. However, when you get down to the individual objects, you would likely have a single example showing that the screen will show whatever error message you throw at it, and then move down to the model to generate this particular edge case message.> However edge cases should be covered by stories because stories define > the desired behavior of the system.Definitely!> Also, edge cases presumably > represent some business value to the customer.Definitely!> Finally, it''s better > to keep all of the desired behavior specs in one place.Here''s where we disagree. I think that it''s a matter of who the audience is for the different tools.> Stories are > naturally more verbose than specs though. > > That paragraph feels awkward to me, because I think I''m missing some > vocabulary. It suggests to me that the collection of all user stories > is the app specification - "spec" for short. What is the collection > of all descriptions, the object-level specs? I''ve always just said > "specs," but I don''t think that really fits anymore with my current > thinking. Not to mention it''s just confusing if I also use "spec" to > mean the collection of all user stories. I know that an individual > describe block is a "description," but maybe the whole suite should be > called "descriptions" instead of "specs?"The vocabulary that we seem to be moving to in the trunk is this: Story Framework - Stories - Scenarios - Steps Example Framework - ExampleGroups (what you referred to as descriptions) - Examples I''m not completely comfortable with this yet in terms of being able to talk about it, so suggestions are welcome. The problem with using Spec or Example is that either can easily refer to the coarse grained stories or the fine grained object level examples. Maybe we should call them Coarse Grained Examples and Fine Grained Examples :) Seriously - I welcome other thoughts on this.> > What do you guys think? Is it useful to draw a line and say that > stories are authoritative and specs aren''t?I don''t find this useful. To me they are two completely separate animals that work together to describe the behaviour of a system. They are equally authoritative (or not). They both have a life cycle in which they move from examples of what we''d like to examples of what we have.> Do you think it becomes > less crucial to use object-level specs to enable refactoring, and > instead lean on stories more?I think it''s useful to use the stories as a periodic sanity check (as I mentioned above), but I think that if you start using stories as your safety net while you''re in the rapid flow of refactoring, you''re begging for a significant drop in productivity. They''re just going to take to long to run and won''t help you to isolate problems as they arise. Thanks for the thought provoking post! Cheers, David
On Nov 13, 2007 11:51 AM, David Chelimsky <dchelimsky at gmail.com> wrote:> On Nov 13, 2007 1:10 PM, Pat Maddox <pergesu at gmail.com> wrote: > > I''ve been thinking about the role that stories played compared to > > specs, and that "should we use should" thread brought those thoughts > > up again. > > > > First I want to discuss whether or not specs are authoritative > > regarding the desired behavior of a system. I would say they''re a > > good approximation of how the system currently runs according to the > > developer''s understanding. I say they''re an approximation because if > > you use mocks everywhere (as I believe you ought to) then you''re not > > performing integrations and the specs are necessarily incomplete. > > > > Why does that idea matter? Traditionally, we''ve leaned heavily on a > > suite of unit tests to perform refactorings. In fact, Fowler says the > > first thing you have to do in order to refactor is grow a set of > > comprehensive unit tests. > > > > Refactoring, by definition, is changing the structure without changing > > the observable behavior. > > IMO, that definition refers to a system, not individual objects. When > you change objects, you usually have to change the examples.I agree completely. That opinion is what spurred this line of thought.> Keep in mind that that goes nearly unnoticed in java or .NET because > of the refactoring tools available on those platforms. Change a method > name once and it changes it everywhere, *including the examples*. Move > a method from one object to another and you get a dialog that let''s > you opt to leave the existing method in place delegating to the new > location, or change all the consumers of that method in the system, > *including the examples*, to use the new location.Which is fantastic, but what strategies should Ruby dudes use in the mean time?> > If I say that stories are authoritative and specs aren''t, then that > > means I ought to rely on stories to provide the safety net for my > > refactoring. When taking those baby steps, it''s important to keep the > > stories green rather than the specs. > > On some level that makes sense - but it''s going to slow you down quite > a bit when you start building up a significant suite of stories.That was my main concern. You can get 1200 examples running in under 5 seconds. Probably can''t do that with stories.> The way I''ve always dealt with this was to rely on the examples to get > me through a refactoring, and then run the stories before committing. > That allows me to stay focused on small problems and progress > steadily.Herein lies the (my) problem. I find that when I refactor the examples break. You''ve already said that it''s okay, because tools ought to change the examples appropriately. I don''t personally have an issue with breaking specs as I refactor...but that''s because I''ve learned to deal with it, and know what kind of breaking changes I''m comfortable with. My main point is that if I have to go in and manually change specs, then in general I don''t consider the specs to be my comfortable safety net. They''ll get there when we have better tool support, but I have to do stuff today. I suppose the *real* problem is that I can''t quantify what impact that all has on my velocity. So I just end up focusing on the pain that it causes me. But, I think getting rid of the pain is a very worthwhile endeavor on its own.> > Another result of this thinking is that more specification should be > > done in stories than I originally thought. Edge cases are my prime > > example. I used to think I should write a story that expresses the > > general desired behavior, and then cover any edge cases in specs. > > I think they should be covered in both stories and examples. The thing > is that they get covered in different ways. > > For one, the language in which they are expressed is different - > because the stories are about the system and the examples are about > the objects in the system. And there will be some duplicated effort > between the stories and the examples - and that is a GOOD thing.Agreed. The more-than-occasional-less-than-frequent duplication between stories and examples doesn''t bother me. In my mind they''re completely separate systems, it''s just that sometimes I happen to be working on such a simple level that they appear to be the same.> Additionally, when dealing with stories, you may specify that a > particular error message shows up on the screen. However, when you get > down to the individual objects, you would likely have a single example > showing that the screen will show whatever error message you throw at > it, and then move down to the model to generate this particular edge > case message. > > > Finally, it''s better > > to keep all of the desired behavior specs in one place. > > Here''s where we disagree. I think that it''s a matter of who the > audience is for the different tools.Let''s say in some banking software we have a transfer screen, and there are three possible errors: insufficient funds, source account frozen, target account frozen. Would you write a scenario for each of the possible errors, and an example for each? This probably isn''t a good example because I think you should :) but it''s easy to see how it would lead to more duplication than I''m comfortable with. Every time you need to specify business value you have to stick it in a story and an example.> The vocabulary that we seem to be moving to in the trunk is this: > > Story Framework > - Stories > - Scenarios > - Steps > > Example Framework > - ExampleGroups (what you referred to as descriptions) > - Examples > > I''m not completely comfortable with this yet in terms of being able to > talk about it, so suggestions are welcome. The problem with using Spec > or Example is that either can easily refer to the coarse grained > stories or the fine grained object level examples. Maybe we should > call them Coarse Grained Examples and Fine Grained Examples :) > Seriously - I welcome other thoughts on this.There''s another part missing, which is the set of all stories, and the set of all example groups.> > What do you guys think? Is it useful to draw a line and say that > > stories are authoritative and specs aren''t? > > I don''t find this useful. To me they are two completely separate > animals that work together to describe the behaviour of a system. They > are equally authoritative (or not). They both have a life cycle in > which they move from examples of what we''d like to examples of what we > have.I agree that they''re completely separate beasts. I have a feeling we''ll disagree on the point of what is authoritative. I think that stories clearly define how much of the system is done, and examples don''t do that at all. However, I''m not sure yet if it''s an important distinction to make, and hopefully over the next few months I''ll gain deeper insight.> Thanks for the thought provoking post!Thanks so much for your reply. Cheers, Pat
On Nov 13, 2007 5:17 PM, Pat Maddox <pergesu at gmail.com> wrote:> On Nov 13, 2007 11:51 AM, David Chelimsky <dchelimsky at gmail.com> wrote:<snip/>> > Keep in mind that that goes nearly unnoticed in java or .NET because > > of the refactoring tools available on those platforms. Change a method > > name once and it changes it everywhere, *including the examples*. Move > > a method from one object to another and you get a dialog that let''s > > you opt to leave the existing method in place delegating to the new > > location, or change all the consumers of that method in the system, > > *including the examples*, to use the new location. > > Which is fantastic, but what strategies should Ruby dudes use in the mean time?I won''t speak for all, or say what one should use. But what I do is I change things one step at a time with autotest running in the background. When autotest tells me something failed, I look at it. When it tells me nothing failed and I expected something to, I look at that. Whether or not you have tool support, refactoring is its own world of study - taking steps that move you in the right direction and keep the bar green as much as possible.> > > If I say that stories are authoritative and specs aren''t, then that > > > means I ought to rely on stories to provide the safety net for my > > > refactoring. When taking those baby steps, it''s important to keep the > > > stories green rather than the specs. > > > > On some level that makes sense - but it''s going to slow you down quite > > a bit when you start building up a significant suite of stories. > > That was my main concern. You can get 1200 examples running in under > 5 seconds. Probably can''t do that with stories.Exactly> > The way I''ve always dealt with this was to rely on the examples to get > > me through a refactoring, and then run the stories before committing. > > That allows me to stay focused on small problems and progress > > steadily. > > Herein lies the (my) problem. I find that when I refactor the > examples break. You''ve already said that it''s okay, because tools > ought to change the examples appropriately. I don''t personally have > an issue with breaking specs as I refactor...but that''s because I''ve > learned to deal with it, and know what kind of breaking changes I''m > comfortable with. My main point is that if I have to go in and > manually change specs, then in general I don''t consider the specs to > be my comfortable safety net. They''ll get there when we have better > tool support, but I have to do stuff today.Actually, if I change something and it causes an example to fail, that suggests to me that the examples ARE a good safety net! Just as in the TDD cycle, the failures actually give you confidence that you''re touching the code you think you''re touching.> I suppose the *real* problem is that I can''t quantify what impact that > all has on my velocity. So I just end up focusing on the pain that it > causes me. But, I think getting rid of the pain is a very worthwhile > endeavor on its own.You may just have to will it away. I find that this does not slow me down. I''m slower on a step by step basis than I am with powerful tools, but since I can solve the same problems with so much less code in Ruby, it sort of balances itself out. <snip/>> Let''s say in some banking software we have a transfer screen, and > there are three possible errors: insufficient funds, source account > frozen, target account frozen. Would you write a scenario for each of > the possible errors, and an example for each?Without even batting an eye! The scenario is going to reflect how things look from the outside. The example is going to be targeted right at the part of the system that is responsible for producing the correct error message.> This probably isn''t a > good example because I think you should :) but it''s easy to see how > it would lead to more duplication than I''m comfortable with. Every > time you need to specify business value you have to stick it in a > story and an example.This used to bother me too, but that was a long time ago. The thing is that you want the examples to be comprehensive so you can refactor. You also want the stories to be comprehensive, so you know what the system does. And so you can refactor ;)> > The vocabulary that we seem to be moving to in the trunk is this: > > > > Story Framework > > - Stories > > - Scenarios > > - Steps > > > > Example Framework > > - ExampleGroups (what you referred to as descriptions) > > - Examples > > > > I''m not completely comfortable with this yet in terms of being able to > > talk about it, so suggestions are welcome. The problem with using Spec > > or Example is that either can easily refer to the coarse grained > > stories or the fine grained object level examples. Maybe we should > > call them Coarse Grained Examples and Fine Grained Examples :) > > Seriously - I welcome other thoughts on this. > > There''s another part missing, which is the set of all stories, and the > set of all example groups.Anybody got any ideas? <snip/> Cheers, David
On Nov 13, 2007 4:00 PM, David Chelimsky <dchelimsky at gmail.com> wrote:> On Nov 13, 2007 5:17 PM, Pat Maddox <pergesu at gmail.com> wrote: > > Let''s say in some banking software we have a transfer screen, and > > there are three possible errors: insufficient funds, source account > > frozen, target account frozen. Would you write a scenario for each of > > the possible errors, and an example for each? > > Without even batting an eye! > > The scenario is going to reflect how things look from the outside. The > example is going to be targeted right at the part of the system that > is responsible for producing the correct error message.When I gave that example I thought it sucked, because I would write a story and example for each too. But I guess I can say it served as a bit of a sanity check. Anyway, a more gray-area example (imo) is Rails validations. We''ve got a user registration page, and it requires the user to fill in their name, email address, and a password. Would you write a scenario where nothing is filled in? Where everything but the name is filled in? And so on for each attribute. How about combinations of missing stuff? It seems to me like that story is on a VERY high level and all we really care about is verifying that the plumbing works (so maybe I need to think of this as a story for the UI domain instead of the business domain?). If the user fills in everything, make sure there''s a new record. If they don''t fill it in, make sure they''re shown the registration page again. It''s cumbersome to try to cover every possible scenario even though you "should." Would you write a story at a lower level, which actually tries to save the objects to the db and verifies that nothing changes? That *definitely* duplicates what''s going on in the model-level specs. Hopefully this thread hasn''t died... Cheers, Pat
On Nov 14, 2007 5:50 PM, Pat Maddox <pergesu at gmail.com> wrote:> On Nov 13, 2007 4:00 PM, David Chelimsky <dchelimsky at gmail.com> wrote: > > On Nov 13, 2007 5:17 PM, Pat Maddox <pergesu at gmail.com> wrote: > > > Let''s say in some banking software we have a transfer screen, and > > > there are three possible errors: insufficient funds, source account > > > frozen, target account frozen. Would you write a scenario for each of > > > the possible errors, and an example for each? > > > > Without even batting an eye! > > > > The scenario is going to reflect how things look from the outside. The > > example is going to be targeted right at the part of the system that > > is responsible for producing the correct error message. > > When I gave that example I thought it sucked, because I would write a > story and example for each too. But I guess I can say it served as a > bit of a sanity check. > > Anyway, a more gray-area example (imo) is Rails validations. We''ve > got a user registration page, and it requires the user to fill in > their name, email address, and a password. > > Would you write a scenario where nothing is filled in? Where > everything but the name is filled in? And so on for each attribute. > How about combinations of missing stuff? > > It seems to me like that story is on a VERY high level and all we > really care about is verifying that the plumbing works (so maybe I > need to think of this as a story for the UI domain instead of the > business domain?). If the user fills in everything, make sure there''s > a new record. If they don''t fill it in, make sure they''re shown the > registration page again. It''s cumbersome to try to cover every > possible scenario even though you "should." > > Would you write a story at a lower level, which actually tries to save > the objects to the db and verifies that nothing changes? That > *definitely* duplicates what''s going on in the model-level specs.Keep in mind that stories/scenarios and specs each play multiple roles. Scenarios start off as a collaboration tool for the whole team to understand what''s about to be developed. Then code is developed, the scenarios pass, and now they serve two new purposes. Assuming they are well maintained (they are kept well organized and passing), scenarios document the system as it actually is and they are system-level regression tests. Similarly, examples start as a means of designing subsets of the system at a very granular level. Then the become documentation of your objects as they really are, and object-level regression tests. Also consider that both levels have a red/green/refactor life cycle, though less apparent at the higher levels. So they are both constantly subject to change in response to new and changing requirements. But they have subtly different forces of change. At a high level, they are both changing because the business rules grow or change. But specs change much more frequently as the design changes. In the end, it''s about finding a process that works for you and your team throughout the lifecycle of a project. For me, the duplication between stories and specs helps me to stay productive all the time. It is faith in combination of the two alerting me of behaviour-breaking changes that keeps me moving. If on your team you find it more pragmatic to express things in one place or the other, and you find that doing so keeps you agile (remember - agility is not only about speed - it''s about responding to change) throughout the project, then that''s OK. My experience tells me that I''m a happier person when I''ve got robust sets of stories and specs that live independently and both cover the entire system from their unique perspective. FWIW.> Hopefully this thread hasn''t died...No - it was just on vacation. Though I would welcome more voices in it if anybody else has an opinion to express. Cheers, David> > > Cheers, > Pat > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >