I have some high-level acceptance tests that read a bunch of files, process them, and then confirm that the processed output matches the expected output. That is, for each file "a.input" I have a corresponding file "a.expected" and I basically want to churn through all of them producing "a.processed" and making sure that "a.processed" equals "a.expected" The way I currently have this is: context ''transforming files'' do specify ''actual output should match expected output'' do @files.each do |file| file.input.process.should == file.expected end end end The problem with this is that a failure on one file prevents all the others from being tested. What would be the "best practice" way to overcome this shortcoming? The solution I used with Test::Unit was to add an "ErrorCollector" as described here: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/138320 Basically it catches exceptions as they are raised, sending messages to the Test::Unit machinery so that the problems can be counted and reported, but continues processing. Would a similar technique be a good idea for RSpec? Or should I just restructure my specs? By that I mean something like the following (although not sure if it will work): context ''transforming files'' do files.each do |file| specify ''actual output should match expected output'' do file.input.process.should == file.expected end end end
David Chelimsky
2007-Feb-26 20:22 UTC
[rspec-users] Continuing a "specify" block after failure
On 2/26/07, leenoori <leenoori at quepica.com> wrote:> I have some high-level acceptance tests that read a bunch of files, > process them, and then confirm that the processed output matches the > expected output. That is, for each file "a.input" I have a > corresponding file "a.expected" and I basically want to churn through > all of them producing "a.processed" and making sure that > "a.processed" equals "a.expected" > > The way I currently have this is: > > context ''transforming files'' do > specify ''actual output should match expected output'' do > @files.each do |file| > file.input.process.should == file.expected > end > end > end > > The problem with this is that a failure on one file prevents all the > others from being tested. What would be the "best practice" way to > overcome this shortcoming? > > The solution I used with Test::Unit was to add an "ErrorCollector" as > described here: > > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/138320 > > Basically it catches exceptions as they are raised, sending messages > to the Test::Unit machinery so that the problems can be counted and > reported, but continues processing. > > Would a similar technique be a good idea for RSpec? Or should I just > restructure my specs? By that I mean something like the following > (although not sure if it will work): > > context ''transforming files'' do > files.each do |file| > specify ''actual output should match expected output'' do > file.input.process.should == file.expected > end > end > endIMO, since statements about each file could fail independently, then each file wants its own specify block. So I would do it this way, with the addition of something to separate out the spec names. For example (possibly to verbose, but you get the idea): context ''transforming files'' do files.each do |file| specify "actual output should match expected output (#{file.inspect})" do file.input.process.should == file.expected end end end Cheers, David> > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >
aslak hellesoy
2007-Feb-26 21:43 UTC
[rspec-users] Continuing a "specify" block after failure
On 2/26/07, leenoori <leenoori at quepica.com> wrote:> I have some high-level acceptance tests that read a bunch of files, > process them, and then confirm that the processed output matches the > expected output. That is, for each file "a.input" I have a > corresponding file "a.expected" and I basically want to churn through > all of them producing "a.processed" and making sure that > "a.processed" equals "a.expected" > > The way I currently have this is: > > context ''transforming files'' do > specify ''actual output should match expected output'' do > @files.each do |file| > file.input.process.should == file.expected > end > end > end > > The problem with this is that a failure on one file prevents all the > others from being tested. What would be the "best practice" way to > overcome this shortcoming? > > The solution I used with Test::Unit was to add an "ErrorCollector" as > described here: > > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/138320 > > Basically it catches exceptions as they are raised, sending messages > to the Test::Unit machinery so that the problems can be counted and > reported, but continues processing. > > Would a similar technique be a good idea for RSpec?No, because it''s already supported by what you suggest below.> Or should I just > restructure my specs?Yes> By that I mean something like the following > (although not sure if it will work): >It will, with a minor modification - you should make sure the name of each specify block is different, otherwise you''ll have a hard time knowing shich one failed.> context ''transforming files'' do > files.each do |file| > specify ''actual output should match expected output'' do > file.input.process.should == file.expected > end > end > end >I have just checked in an example that illustrates this: http://rubyforge.org/viewvc/trunk/rspec/examples/dynamic_spec.rb?revision=1537&root=rspec Aslak> _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >
El 26/2/2007, a las 21:22, David Chelimsky escribi?:> IMO, since statements about each file could fail independently, then > each file wants its own specify block. So I would do it this way, with > the addition of something to separate out the spec names. For example > (possibly to verbose, but you get the idea): > > context ''transforming files'' do > files.each do |file| > specify "actual output should match expected output (# > {file.inspect})" do > file.input.process.should == file.expected > end > end > end > > Cheers, > DavidMany thanks for your advice, David! I will go ahead and do it this way. Cheers!
El 26/2/2007, a las 22:43, aslak hellesoy escribi?:> I have just checked in an example that illustrates this: > > http://rubyforge.org/viewvc/trunk/rspec/examples/dynamic_spec.rb? > revision=1537&root=rspec > > AslakI''ve discovered one minor problem with this approach; if you have a "context_setup" block inside your context then it will be run *after* the dynamic specify block. For example, look below where I''ve added a "context_setup" block to your example. Intuition might lead one to believe that it would be executed *before* the specs because it''s labelled as a "setup" block and also because it appears physically before the "specify" block. But it actually runs after: context "The month march" do context_setup do @start = 1 @max = 10 end (@start.. at max).each do |n| specify "The root of #{n} square should be #{n}" do Math.sqrt(n*n).should == n end end end No big deal, of course, but it can cause breakage (and does in the case of the contrived example). The workaround is to forget using "context_setup" and do the setup in the same scope as the following "specify" block: context "The month march" do @start = 1 @max = 10 (@start.. at max).each do |n| specify "The root of #{n} square should be #{n}" do Math.sqrt(n*n).should == n end end end I suspect a robust fix would involve evaluating context_setup blocks as soon as they are encountered rather than deferring them until later. Should I file a bug report/enhance request for this? Cheers :-)
David Chelimsky
2007-Feb-27 14:53 UTC
[rspec-users] Continuing a "specify" block after failure
On 2/27/07, leenoori <leenoori at quepica.com> wrote:> El 26/2/2007, a las 22:43, aslak hellesoy escribi?: > > > I have just checked in an example that illustrates this: > > > > http://rubyforge.org/viewvc/trunk/rspec/examples/dynamic_spec.rb? > > revision=1537&root=rspec > > > > Aslak > > I''ve discovered one minor problem with this approach; if you have a > "context_setup" block inside your context then it will be run *after* > the dynamic specify block. For example, look below where I''ve added a > "context_setup" block to your example. Intuition might lead one to > believe that it would be executed *before* the specs because it''s > labelled as a "setup" block and also because it appears physically > before the "specify" block. But it actually runs after: > > context "The month march" do > context_setup do > @start = 1 > @max = 10 > end > (@start.. at max).each do |n| > specify "The root of #{n} square should be #{n}" do > Math.sqrt(n*n).should == n > end > end > end > > No big deal, of course, but it can cause breakage (and does in the > case of the contrived example). The workaround is to forget using > "context_setup" and do the setup in the same scope as the following > "specify" block: > > context "The month march" do > @start = 1 > @max = 10 > (@start.. at max).each do |n| > specify "The root of #{n} square should be #{n}" do > Math.sqrt(n*n).should == n > end > end > end > > I suspect a robust fix would involve evaluating context_setup blocks > as soon as they are encountered rather than deferring them until > later. Should I file a bug report/enhance request for this?This is the only list I''ve seen where everyone is so polite that they ask whether they should ask for what they want ;) Yes, please, if you want it addressed the best way to ensure that it gets on and stays on the radar is a feature request. Even if we reject it, it''s there for posterity and can always be resurrected if appropriate. Cheers, David> > Cheers :-) > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >