Bill Wendling <isanbard at gmail.com> writes: [snip]> All four of the above should be run on at least a nightly basis (more > frequently for some, like the regression tests). Each of these are > automated, making that easy. If there are no regressions from the > above four, we could tag that revision as being potentially "valid".If a new test case is created (coming from a bug report or a code review, not from adding a new feature) and it fails for a previously "valid" revision, is the tag removed? -- Oscar
On Nov 11, 2008, at 11:00 PM, Óscar Fuentes wrote:> Bill Wendling <isanbard at gmail.com> writes: > > [snip] > >> All four of the above should be run on at least a nightly basis (more >> frequently for some, like the regression tests). Each of these are >> automated, making that easy. If there are no regressions from the >> above four, we could tag that revision as being potentially "valid". > > If a new test case is created (coming from a bug report or a code > review, not from adding a new feature) and it fails for a previously > "valid" revision, is the tag removed? >There would probably be some sort of concept of "last known good". In the above case, either the tag could be removed and/or a new valid revision tagged. It was "good" up until that point, at least. :-) -bw
On Wednesday 12 November 2008 02:20, Bill Wendling wrote:> On Nov 11, 2008, at 11:00 PM, Óscar Fuentes wrote: > > Bill Wendling <isanbard at gmail.com> writes: > > > > [snip] > > > >> All four of the above should be run on at least a nightly basis (more > >> frequently for some, like the regression tests). Each of these are > >> automated, making that easy. If there are no regressions from the > >> above four, we could tag that revision as being potentially "valid". > > > > If a new test case is created (coming from a bug report or a code > > review, not from adding a new feature) and it fails for a previously > > "valid" revision, is the tag removed? > > There would probably be some sort of concept of "last known good". In > the above case, either the tag could be removed and/or a new valid > revision tagged. It was "good" up until that point, at least. :-)I would say that once a validation tag is created, it stays. We don't want to be in the business of going back through lots of tag history and checking against new tests that didn't exist when the tags were originally created. Perhaps for a "last known good" tag we could re-check but I'm even hesitent to do that. It seems like a lot of extra work for little real gain. -Dave