We ran into a situation today where running ''rake spec'' on one machine produced different results than when running on a different machine. My reasoned hunch is that the tests were run in a different order, due to the file-globbing producing differently ordered file lists. Because some tests used fixtures and some didn''t, that resulted in different test db contents, depending on the order the tests were run (clearly, these tests need fixing). I haven''t really looked into this at all, for now. But it occurs to me that if my suspicion is correct, this situation would be fixed by running tests in alphabetical order. Presumably this would be a change to the spec Rake task. Does this make any kind of sense? If so, I (or someone better suited to the task) could look into it further. (Coincidentally, a similar situation resulted in a contribution I made to Visual FoxPro!) ///ark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/rspec-users/attachments/20080821/78c8583c/attachment.html>
On Thu, Aug 21, 2008 at 2:25 PM, Mark Wilden <mark at mwilden.com> wrote:> We ran into a situation today where running ''rake spec'' on one machine > produced different results than when running on a different machine. My > reasoned hunch is that the tests were run in a different order, due to the > file-globbing producing differently ordered file lists. Because some tests > used fixtures and some didn''t, that resulted in different test db contents, > depending on the order the tests were run (clearly, these tests need > fixing). > > I haven''t really looked into this at all, for now. But it occurs to me that > if my suspicion is correct, this situation would be fixed by running tests > in alphabetical order. Presumably this would be a change to the spec Rake > task. > > Does this make any kind of sense? If so, I (or someone better suited to the > task) could look into it further. (Coincidentally, a similar situation > resulted in a contribution I made to Visual FoxPro!)My personal feeling is that the code examples should be able to run in any arbitrary order and still pass (no dependencies between examples), so I would recommend you find the dependency rather than working around the problem by enforcing order. Cheers, David> ///ark > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >
On Thu, Aug 21, 2008 at 12:51 PM, David Chelimsky <dchelimsky at gmail.com>wrote:> > My personal feeling is that the code examples should be able to run in > any arbitrary order and still pass (no dependencies between examples)I agree, which is why I said "clearly these tests need fixing." Maybe a better suggestion would be to execute the tests in random order each time... However, the point was that we didn''t -know- there was an order dependency, and neither would the guy whose specs all passed before he went on vacation. ///ark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/rspec-users/attachments/20080821/12cfd703/attachment.html>
On Thu, Aug 21, 2008 at 4:29 PM, Mark Wilden <mark at mwilden.com> wrote:> On Thu, Aug 21, 2008 at 12:51 PM, David Chelimsky <dchelimsky at gmail.com> > wrote: >> >> My personal feeling is that the code examples should be able to run in >> any arbitrary order and still pass (no dependencies between examples) > > I agree, which is why I said "clearly these tests need fixing."Got it.> Maybe a > better suggestion would be to execute the tests in random order each time...There''s been some discussion of this but it never got completed.> > However, the point was that we didn''t -know- there was an order dependency, > and neither would the guy whose specs all passed before he went on vacation.Any chance the different machines are windows and otherwise? By default the files are loaded alphabetically, but that means different things on windows machines (which treat alpha without regard to case) and pretty much anything else.> > ///ark > > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >
On Fri, Aug 22, 2008 at 6:18 AM, David Chelimsky <dchelimsky at gmail.com>wrote:> > Any chance the different machines are windows and otherwise? By > default the files are loaded alphabetically, but that means different > things on windows machines (which treat alpha without regard to case) > and pretty much anything else. >The specs ran fine a Free-BSD machine, but failed on Windows and OS X machines. The guy who "initiated" the failures was on Windows but doesn''t tend to run rake spec locally (this was Bruce, Pat). Maybe I''ll have a look at the random order thing... ///ark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/rspec-users/attachments/20080822/822071d2/attachment-0001.html>
Mark Wilden wrote:> On Fri, Aug 22, 2008 at 6:18 AM, David Chelimsky <dchelimsky at gmail.com > <mailto:dchelimsky at gmail.com>> wrote: > > > Any chance the different machines are windows and otherwise? By > default the files are loaded alphabetically, but that means different > things on windows machines (which treat alpha without regard to case) > and pretty much anything else. > > > The specs ran fine a Free-BSD machine, but failed on Windows and OS X > machines. The guy who "initiated" the failures was on Windows but > doesn''t tend to run rake spec locally (this was Bruce, Pat). > > Maybe I''ll have a look at the random order thing... >I think autotest does this already... It will at least mess with the order of your specs/tests so that brittle ones are exposed. -Ben> ///ark > ------------------------------------------------------------------------ > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users