Gluster users - Oct 2012 - Automated regression tests now part of development work flow

Hello all,

The recent ongoing changes to the development work flow are now complete.
The highlight of the rework is the automated regression tests.

So far we have only had fixed smoke tests that were run against every patch
pre-commit (dbench, POSIX compliance). Here after, every change submitted
to Gerrit must be accompanied with test case scripts which (attempts to)
prove the correctness of the code change - as part of the new regression
test in the work flow.

These test cases will be much more concentrated and focused on the changes
brought in by the patch itself. The test cases, once committed will be part
of every future pre-commit test. This will make sure no commit in the
future will accidentally break or change the current commit either directly
or with side effects. If a future change must change the behavior of the
current patch, then the future patch must include corresponding changes to
the test cases as well. This policy will keep code and test cases in sync.

The wiring and framework for supporting this are already available in
glusterfs.git and Jenkins. Currently the infrastructure supports test cases
limited to single node. Most of the test cases can actually be implemented
in single node by having multiple instances of bricks and client mounts.
Support of multi node test cases will be added soon.

The automated smoke test's voting power in Gerrit is now downgraded to +0
Verified for PASS and -1 Verified for FAILURE. Regression test's voting
powers are now +1 Verified and -1 Verified. This means just passing the
smoke test can no more be a qualifier for verification. For those patches
for which a regression test is required but currently not automatable into
a test script, must be tested and voted manually for the Verification vote.

As the above described changes are already in effect, most of the currently
submitted patches under review will have to be resubmitted with test cases.
This is necessary to make the changes start being effective immediately. If
you have currently

Note that this opens up a new avenue for contributors in the community. You
can now contribute test cases. These contributions need not include source
code changes but just extend our regression test set. These test cases can
either be for coverage of old code and functionality, or for coverage of
your use cases which could be applicable to others as well.

The workflow document is also updated at
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow.
Specifically:
Sec 1.4
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Commit_policy
Sec 1.9
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Regression_tests_and_test_cases

More details on how to write test cases with examples and how the framework
works can be found at -

1. "tests: pre-commit regression tests"
https://github.com/gluster/glusterfs/commit/bb41c8ab88f1a3d8c54b635674d0a72133623496
2. tests/README in glusterfs.git
3. Example new change - http://review.gluster.org/4114

Please offer feedback on how the framework can be improved (inclusion of
more tools?) to capture test cases better.

Happy Hacking!
Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121020/e907a2c7/attachment.html>

On 10/21/2012 02:53 AM, Anand Avati wrote:
> More details on how to write test cases with examples and how the
> framework works can be found at -
>
> 1. "tests: pre-commit regression tests"
>
https://github.com/gluster/glusterfs/commit/bb41c8ab88f1a3d8c54b635674d0a72133623496
> 2. tests/README in glusterfs.git
> 3. Example new change - http://review.gluster.org/4114
>
> Please offer feedback on how the framework can be improved (inclusion of
> more tools?) to capture test cases better.
>
This is a simple question on guide lines of writing a test case, and has 
raised due to failure of tests for patch http://review.gluster.org/4567 
in particular.

Looks like the failure of the tests for this patch (logs here 
http://build.gluster.org/job/regression/729/consoleFull ) are due to 
missing 'cleanup;' at the end of the script added in this patch. But the
question is, should the test script always 'assume' it starts from a 
clean slate, or should it call a 'cleanup;' at the beginning all the
time?

It would help me to review the patchsets by having clear answer to this.

Regards,
Amar

On Sat, 2012-10-20 at 14:23 -0700, Anand Avati wrote:
> Hello all,
> 
> The recent ongoing changes to the development work flow are now complete.
> The highlight of the rework is the automated regression tests.
This is awesome!
> 
> So far we have only had fixed smoke tests that were run against every patch
> pre-commit (dbench, POSIX compliance). Here after, every change submitted
> to Gerrit must be accompanied with test case scripts which (attempts to)
> prove the correctness of the code change - as part of the new regression
> test in the work flow.
> 
> These test cases will be much more concentrated and focused on the changes
> brought in by the patch itself. The test cases, once committed will be part
> of every future pre-commit test. This will make sure no commit in the
> future will accidentally break or change the current commit either directly
> or with side effects. If a future change must change the behavior of the
> current patch, then the future patch must include corresponding changes to
> the test cases as well. This policy will keep code and test cases in sync.
I don't suppose someone knows offhand of a way to automatically
"watch"
for proposals to certain test cases? This could be a useful way for a
programmer to get a chance to speak up about an upcoming incompatible
change. Anyways, low priority, if someone has already hacked this
together with git, or if you'd be interested in this sort of thing, let
me know.
> 
> The wiring and framework for supporting this are already available in
> glusterfs.git and Jenkins. Currently the infrastructure supports test cases
> limited to single node. Most of the test cases can actually be implemented
> in single node by having multiple instances of bricks and client mounts.
> Support of multi node test cases will be added soon.
> 
> The automated smoke test's voting power in Gerrit is now downgraded to
+0
> Verified for PASS and -1 Verified for FAILURE. Regression test's voting
> powers are now +1 Verified and -1 Verified. This means just passing the
> smoke test can no more be a qualifier for verification. For those patches
> for which a regression test is required but currently not automatable into
> a test script, must be tested and voted manually for the Verification vote.
> 
> As the above described changes are already in effect, most of the currently
> submitted patches under review will have to be resubmitted with test cases.
> This is necessary to make the changes start being effective immediately. If
> you have currently
> 
> Note that this opens up a new avenue for contributors in the community. You
> can now contribute test cases. These contributions need not include source
> code changes but just extend our regression test set. These test cases can
> either be for coverage of old code and functionality, or for coverage of
> your use cases which could be applicable to others as well.
I will put it onto my TODO list to write some test cases that "watch"
the functionality used by my puppet-gluster module.
> 
> The workflow document is also updated at
>
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow.
> Specifically:
> Sec 1.4
>
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Commit_policy
> Sec 1.9
>
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Regression_tests_and_test_cases
> 
> More details on how to write test cases with examples and how the framework
> works can be found at -
> 
> 1. "tests: pre-commit regression tests"
>
https://github.com/gluster/glusterfs/commit/bb41c8ab88f1a3d8c54b635674d0a72133623496
> 2. tests/README in glusterfs.git
> 3. Example new change - http://review.gluster.org/4114
> 
> Please offer feedback on how the framework can be improved (inclusion of
> more tools?) to capture test cases better.
> 
> Happy Hacking!
> Avati
Thanks!!
James
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130225/385f0f4d/attachment.sig>