Hi, One of the problems with mocks, as far as I can tell, is that they might go out of sync with the real object they''re mocking. Is it possible and sane to detect this by running each spec against its corresponding mock? Does anyone already do this? For example: your Account object has a particular behavioural specification; you perhaps have some convenient helper method for creating a mock Account which conforms to that specification with various stubbed responses; and the specifications of other objects use that mock Account when describing behaviours which involve interaction with Accounts. What happens when you want to change the behaviour of Account? Naively, you update the specification and implementation of Account, your specs all pass, and you think you''re done, except you''re not: the Account mock is now misrepresenting the behaviour of Account all over the system, with the result that you''ve got failures that the specs don''t reveal. Of course you have integration tests that will show these failures at a higher level but it doesn''t feel fundamentally like a problem that RSpec has no business solving. Why not establish a base case for RSpec''s inductive demonstration of correctness by running the Account spec against the standard Account mock? That way you''ll automatically be told, by RSpec, whenever the actual behaviour of your mock doesn''t match the specified behaviour of the mocked object, and you can let your integration tests concentrate on the hard stuff (stateful interactions between many objects) instead of checking that you haven''t forgotten to keep your specs up-to-date. This assumes a priori that it makes engineering sense to have a globally-available DRY mock helper for each object (i.e. class) rather than building up your mocks piecemeal in every spec; I''ve found that I always start off a project by doing the latter but end up refactoring into the former once I get sick of mocking the same stuff over and over again. Maybe that''s the wrong way to do things altogether, or maybe this is a process issue that I''ve misunderstood (i.e. the answer is always to use integration tests to detect this kind of failure), so stop me if I''m being stupid. Cheers, -Tom
Hi Tom, On Fri, May 29, 2009 at 4:11 AM, Tom Stuart <tom at experthuman.com> wrote:> Hi, > > One of the problems with mocks, as far as I can tell, is that they might go > out of sync with the real object they''re mocking.Have you read "Mock Roles, not Objects"? http://mockobjects.com/files/mockrolesnotobjects.pdf> Is it possible and sane to > detect this by running each spec against its corresponding mock? Does anyone > already do this?"it''s corresponding mock" suggests a single mock object for each real implementation. There''s nothing stopping you from writing your own mock objects to play this role. IMO, this is not what a dynamic mock objects framework is for.> For example: your Account object has a particular behavioural specification; > you perhaps have some convenient helper method for creating a mock Account > which conforms to that specification with various stubbed responses; and the > specifications of other objects use that mock Account when describing > behaviours which involve interaction with Accounts. What happens when you > want to change the behaviour of Account? Naively, you update the > specification and implementation of Account, your specs all pass, and you > think you''re done, except you''re not: the Account mock is now > misrepresenting the behaviour of Account all over the system, with the > result that you''ve got failures that the specs don''t reveal.First of all, let''s make a distinction between behaviour and method signatures. Just because two objects sport the same methods doesn''t mean they behave the same way. In order to verify that a mock Account honors the same contract as a real Account, you''d have to have some very general specs like "Account balance should be an instance of the Money class." This would not really get you very far in terms of specifying the real account, and it''s exactly the opposite of what we''re getting at with BDD - focus on what an object does (behaviour), not what it is (structure). So really, what we''re talking about is concern over method signatures straying. Theoretically, this could be solved with some audit mechanism, but then we''re tying mocks to specific objects. What about when we use mocks in their most powerful way, to specify an object''s contract with polymorphic collaborators? Consider the case of an Account in a banking application. Checking accounts and savings accounts have some significant differences in terms of their behaviour, yet a Transfer will probably only use the parts that are common to both: def transfer(source_account, target_account, amount) source_account.debit(amount) target_account.credit(amount) end an admittedly naive implementation, but it demonstrates the point. The source account may be a checking account. It may be a savings account. It may be a checking account with overdraw privileges. It may be a checking account with overdraw privileges that draw on the target account. Etc, etc. What mocks/stubs let us do here is set up the context for each example exactly how we want it to be, with objects that respond how we program them to, but aren''t necessarily of a specific class.> Of course you have integration tests that will show these failures at a > higher level but it doesn''t feel fundamentally like a problem that RSpec has > no business solving.My personal opinion is that RSpec has no business solving this :) Personal opinion aside, let''s say we set out to solve this. We''d need some auditing mechanism that says the object being mocked has all the same APIs as the mock object. For RSpec, or any other Ruby framework, we have a serious obstacle to being able to do this universally and reliably: Ruby We''re dealing with a language that lets us modify an object''s behaviour on the fly. This means that it is entirely possible (and quite common) for a given method not to exist after a class definition is loaded, but to be added later in the process via a dynamic method definition or a mixin. If the point at which we do the audit, that method has not yet been added to the object, we''re screwed. Would that be a problem all the time? Certainly not. But it means we''d be depending on something that is not very reliable and/or restricting our use of Ruby''s most powerful features.> Why not establish a base case for RSpec''s inductive > demonstration of correctness by running the Account spec against the > standard Account mock? That way you''ll automatically be told, by RSpec, > whenever the actual behaviour of your mock doesn''t match the specified > behaviour of the mocked object, and you can let your integration tests > concentrate on the hard stuff (stateful interactions between many objects) > instead of checking that you haven''t forgotten to keep your specs > up-to-date. > > This assumes a priori that it makes engineering sense to have a > globally-available DRY mock helper for each object (i.e. class) rather than > building up your mocks piecemeal in every spec; I''ve found that I always > start off a project by doing the latter but end up refactoring into the > former once I get sick of mocking the same stuff over and over again. Maybe > that''s the wrong way to do things altogether, or maybe this is a process > issue that I''ve misunderstood (i.e. the answer is always to use integration > tests to detect this kind of failure), so stop me if I''m being stupid.The most powerful use of mocks is as a design tool, allowing you to focus on the object at hand and invent its collaborators as you go, without having to go out and develop them just yet. As you suggest above, even in your own process, you tend to build up mocks piecemeal and then refactor towards a single, global helper to create a mock Account (for example). This lets you keep focus on the task at hand, and refactor to eliminate duplication as the duplication appears. Seems like a perfectly viable approach to me. Of course that doesn''t solve the auditing problem I describe above, and it also pushes you towards a one to one mapping between mocked object and mock object, which reduces the power of mocks in terms of polymorphism. So that''s my 1.8 cents (recession, and all). Looking forward to some other opinions. Cheers, David> Cheers, > -Tom
Hi David, On 29 May 2009, at 12:10, David Chelimsky wrote:> Have you read "Mock Roles, not Objects"? > "its corresponding mock" suggests a single mock object for each real > implementation. There''s nothing stopping you from writing your own > mock objects to play this role. IMO, this is not what a dynamic mock > objects framework is for.Agreed -- I entirely concur that the main benefit of such a framework is the ability to speculatively mock collaborators (while thinking in terms of their role) before you have implemented them or perhaps even know what they are -- but in practice I find that these collaborating roles end up falling into one-one correspondence with actual classes, which doesn''t seem to contradict the core message of "Mock Roles, not Objects". Maybe that just indicates I''m not working on designs that are sufficiently sophisticated, sufficiently polymorphic or sufficiently aspect-oriented to feel the burn of the divergence of concepts. Ultimately I think I''m agreeing with you, but perhaps just using the wrong word: I might better have described Account as a role rather than necessarily an object/class. The original point applies, namely "how can I be sure that my mock (for a given role) is in sync with my specification (of that role in the object(s) which perform it)"; instead of suggesting a single mock object for each real implementation, I''m suggesting a single mock object for each role. And speaking of using the wrong words: maybe what I actually mean is "specifying stubs" instead of "specifying mocks", in as much as those global helpers I talked about don''t really return a "mock" at all, just a stub which has been prepared for use as a mock in any spec which requires it (cf Spec::Rails'' mock_model). But any such spec will exercise those stubbed methods which are important for the role''s behaviour, assuming one expectation per example, so it''s important that the stubbed behaviour accurately reflects reality, and unless you are rigorous about keeping the stubbed behaviour updated to reflect the specified behaviour they''ll drift out of sync and the specs won''t be making the right assumptions about collaborators any more.> Just because two objects sport the same methods doesn''t > mean they behave the same way. In order to verify that a mock Account > honors the same contract as a real Account, you''d have to have some > very general specs like "Account balance should be an instance of the > Money class."Why? I can''t quite get my head around the issues of stubbing stateful interactions, which is preventing me from thinking of a good example, but what''s (abstractly) wrong with: class Account def credit(cents) self.balance = self.balance + cents return self.balance end end describe ''Account#credit'' do before(:each) do @account = Account.new(:balance => 100) end specify { @account.credit(10).should == 110 } end def mock_account account = mock(''Account'') account.stub!(:credit).with(10).and_return(110) return account end It seems like there''s nothing stopping you stubbing the detailed behaviour of Account in a way that will allow the stub to pass the specs -- it''s just that the stub is only good for the canned responses, necessarily a strict subset of the real implementation''s behaviour over all inputs. Given the above, I could easily change spec and implementation to class Account def credit(cents) self.balance = self.balance - cents return self.balance end end describe ''Account#credit'' do before(:each) do @account = Account.new(:balance => 100) end specify { @account.credit(10).should == 90 } end and the spec will pass, and so will every spec that''s using my Account stub for mocking Account behaviour, but the system as a whole is screwed, right? Because the behaviour''s changed but the stub (plus the behaviour that is mocked on it by other specs) has stayed the same.> So really, what we''re talking about is concern over method > signatures straying.Sorry, that''s not what I meant at all: I intended to talk specifically about cases where the signature stays the same but the underlying behaviour changes.> What about when we use mocks in their most powerful way, to specify > an object''s > contract with polymorphic collaborators?In that case there''s a role (an object fulfilling that contract) for which you might end up creating a shared stub that''s used in the specifications of all objects which act as the client in that specific collaboration. Individual examples in those specifications will set individual expectations on that stub in order to specify the interaction, presumably clobbering a stubbed method in the process, but the remaining stubbed methods will be exercised during each example, which gives you a way of knowing that the mocked behaviour lines up with what''s been canned in the stub. You can''t do much about the detailed, piecemeal mocking that happens inside such specs -- aside from the aforementioned reliance upon the parts of the stub''s canned behaviour for which an expectation hasn''t explicitly been set, there''s no way to check that the expectations match up with the specification(s) of the object(s) which performs the role you''re mocking, so you have to rely on integration specs -- but if all of these specs are starting with the same stub, you can at least run the role''s spec over the stub to make sure it''s accurate. Why wouldn''t you?> Personal opinion aside, let''s say we set out to solve this. We''d need > some auditing mechanism that says the object being mocked has all the > same APIs as the mock object.As I say, that''s not what I meant, and it''s probably my fault for saying "mock". What I meant is that a stub can provide a snapshot of all of the behaviour of an object (or role), and I often end up in the situation where the same stub is being used as a starting point for mocking interactions in lots of specs, but right now there''s no mechanism for spotting when that stub is spreading misinformation because it''s become desynchronised from the specification of the object (or role) which it represents. For example: Spec::Rails'' mock_model. (Should be called stub_model but isn''t.) It''s very convenient because it stubs out a bunch of "behaviour" that you''d otherwise have to stub out yourself in all of your model specs before you started adding expectations. It conforms to a tiny subset of ActiveRecord::Base''s notional specification: @record.should_receive(:new_record?).and_return(false) etc. If "the ActiveRecord::Base spec" changes, mock_model needs updating, but we can''t discover that automatically because we never run "the ActiveRecord::Base spec" over the object that''s returned by mock_model. (This is a stupid example because there is no ActiveRecord::Base spec, but you get the idea.) So is my fundamental mistake that I''m being lazy by trying to concentrate all of this stub setup in one helper method instead of doing piecemeal stubbing in every individual spec that might care? It seems superficially like good practice to concentrate all of the detail in one place like this, so that when a role''s behaviour changes you just update one piece of stub setup to reflect the change rather than chase around the specs of every single collaborator, but maybe that''s just serving to obscure the problem that all of the actual mocking has to happen in the individual specs and that this is the stuff you really care about, not the behaviour of the shared stub? Cheers, -Tom
On 29 May 2009, at 13:51, Tom Stuart wrote:> Hi David, > > On 29 May 2009, at 12:10, David Chelimsky wrote: >> Have you read "Mock Roles, not Objects"? >> "its corresponding mock" suggests a single mock object for each real >> implementation. There''s nothing stopping you from writing your own >> mock objects to play this role. IMO, this is not what a dynamic mock >> objects framework is for. > > Agreed -- I entirely concur that the main benefit of such a > framework is the ability to speculatively mock collaborators (while > thinking in terms of their role) before you have implemented them or > perhaps even know what they are -- but in practice I find that these > collaborating roles end up falling into one-one correspondence with > actual classes, which doesn''t seem to contradict the core message of > "Mock Roles, not Objects". Maybe that just indicates I''m not working > on designs that are sufficiently sophisticated, sufficiently > polymorphic or sufficiently aspect-oriented to feel the burn of the > divergence of concepts. > > Ultimately I think I''m agreeing with you, but perhaps just using the > wrong word: I might better have described Account as a role rather > than necessarily an object/class. The original point applies, namely > "how can I be sure that my mock (for a given role) is in sync with > my specification (of that role in the object(s) which perform it)"; > instead of suggesting a single mock object for each real > implementation, I''m suggesting a single mock object for each role. > > And speaking of using the wrong words: maybe what I actually mean is > "specifying stubs" instead of "specifying mocks", in as much as > those global helpers I talked about don''t really return a "mock" at > all, just a stub which has been prepared for use as a mock in any > spec which requires it (cf Spec::Rails'' mock_model). But any such > spec will exercise those stubbed methods which are important for the > role''s behaviour, assuming one expectation per example, so it''s > important that the stubbed behaviour accurately reflects reality, > and unless you are rigorous about keeping the stubbed behaviour > updated to reflect the specified behaviour they''ll drift out of sync > and the specs won''t be making the right assumptions about > collaborators any more. > >> Just because two objects sport the same methods doesn''t >> mean they behave the same way. In order to verify that a mock Account >> honors the same contract as a real Account, you''d have to have some >> very general specs like "Account balance should be an instance of the >> Money class." > > Why? I can''t quite get my head around the issues of stubbing > stateful interactions, which is preventing me from thinking of a > good example, but what''s (abstractly) wrong with: > > class Account > def credit(cents) > self.balance = self.balance + cents > return self.balance > end > end > > describe ''Account#credit'' do > before(:each) do > @account = Account.new(:balance => 100) > end > specify { @account.credit(10).should == 110 } > end > > def mock_account > account = mock(''Account'') > account.stub!(:credit).with(10).and_return(110) > return account > end > > It seems like there''s nothing stopping you stubbing the detailed > behaviour of Account in a way that will allow the stub to pass the > specs -- it''s just that the stub is only good for the canned > responses, necessarily a strict subset of the real implementation''s > behaviour over all inputs. > > Given the above, I could easily change spec and implementation to > > class Account > def credit(cents) > self.balance = self.balance - cents > return self.balance > end > end > > describe ''Account#credit'' do > before(:each) do > @account = Account.new(:balance => 100) > end > specify { @account.credit(10).should == 90 } > end > > and the spec will pass, and so will every spec that''s using my > Account stub for mocking Account behaviour, but the system as a > whole is screwed, right? Because the behaviour''s changed but the > stub (plus the behaviour that is mocked on it by other specs) has > stayed the same. > >> So really, what we''re talking about is concern over method >> signatures straying. > > Sorry, that''s not what I meant at all: I intended to talk > specifically about cases where the signature stays the same but the > underlying behaviour changes. > >> What about when we use mocks in their most powerful way, to specify >> an object''s >> contract with polymorphic collaborators? > > In that case there''s a role (an object fulfilling that contract) for > which you might end up creating a shared stub that''s used in the > specifications of all objects which act as the client in that > specific collaboration. Individual examples in those specifications > will set individual expectations on that stub in order to specify > the interaction, presumably clobbering a stubbed method in the > process, but the remaining stubbed methods will be exercised during > each example, which gives you a way of knowing that the mocked > behaviour lines up with what''s been canned in the stub. > > You can''t do much about the detailed, piecemeal mocking that happens > inside such specs -- aside from the aforementioned reliance upon the > parts of the stub''s canned behaviour for which an expectation hasn''t > explicitly been set, there''s no way to check that the expectations > match up with the specification(s) of the object(s) which performs > the role you''re mocking, so you have to rely on integration specs -- > but if all of these specs are starting with the same stub, you can > at least run the role''s spec over the stub to make sure it''s > accurate. Why wouldn''t you? > >> Personal opinion aside, let''s say we set out to solve this. We''d need >> some auditing mechanism that says the object being mocked has all the >> same APIs as the mock object. > > As I say, that''s not what I meant, and it''s probably my fault for > saying "mock". What I meant is that a stub can provide a snapshot of > all of the behaviour of an object (or role), and I often end up in > the situation where the same stub is being used as a starting point > for mocking interactions in lots of specs, but right now there''s no > mechanism for spotting when that stub is spreading misinformation > because it''s become desynchronised from the specification of the > object (or role) which it represents. > > For example: Spec::Rails'' mock_model. (Should be called stub_model > but isn''t.) It''s very convenient because it stubs out a bunch of > "behaviour" that you''d otherwise have to stub out yourself in all of > your model specs before you started adding expectations. It conforms > to a tiny subset of ActiveRecord::Base''s notional specification: > @record.should_receive(:new_record?).and_return(false) etc. If "the > ActiveRecord::Base spec" changes, mock_model needs updating, but we > can''t discover that automatically because we never run "the > ActiveRecord::Base spec" over the object that''s returned by > mock_model. (This is a stupid example because there is no > ActiveRecord::Base spec, but you get the idea.) > > So is my fundamental mistake that I''m being lazy by trying to > concentrate all of this stub setup in one helper method instead of > doing piecemeal stubbing in every individual spec that might care? > It seems superficially like good practice to concentrate all of the > detail in one place like this, so that when a role''s behaviour > changes you just update one piece of stub setup to reflect the > change rather than chase around the specs of every single > collaborator, but maybe that''s just serving to obscure the problem > that all of the actual mocking has to happen in the individual specs > and that this is the stuff you really care about, not the behaviour > of the shared stub?On my team, we''ve built up a StubFactory which works a lot like the FactoryGirl plug-in and creates ''stock'' stubbed objects which we use when we need a generic stub of a given role. This is a useful tool, and it saves a lot of noise in our tests when we just need something that quacks enough like an Account or whatever in order to let a test run through, so that you can focus on the detail of the *specific* collaboration behaviour you''re concerned about in that test. So that''s one part of this - I definitely think it''s pragmatic and valuable to factor out common mocking setup into a single place if that''s what you seem to be duplicating a lot in your tests. The auditing though... I''ve been around the loop with this one myself - before I really discovered the value of acceptance tests I was really keen on the idea, but to be honest this just isn''t a problem I come across very often now. By the time my acceptance tests have pointed me down to the unit tests / classes I need to work on, I''m (especially with a pair to work with) sufficiently focussed to remember enough the of collaborations between objects and keep them in step with the mocks / stubs. If I do forget something, I get pretty rapid feedback when I step back up a level and run the acceptance tests. I appreciate that the argument ''I can keep all the collaboration in my head'' is a bit wooly - but equally if the collaborations you''re mocking are so complex that you can''t, then perhaps that in itself is a whiff that something is wrong with your design? Matt Wynne http://beta.songkick.com http://blog.mattwynne.net