It would help with rsync quality if we had an automatic test suite
that could run on the build farm machines to validate functionality.
Writing a test suite is hard when there is no requirements document.
The best thing to do is probably to start with the functionality that
is both commonly used and easy to test, and work up from there. So in
the case of rsync this is copying from one directory to another, and
then the straightforward options like --exclude and --delete.
On the other hand you should test where you expect to find bugs. So,
at the moment, this seems to be in the networking code rather than
--exclude.
Here are a few ideas:
* The test must run unattended by 'make check'. It should display
some kind of progress indication.
* It should be possible to run the tests without having root on the
box, and without requiring any changes to configuration.
* Running the tests should not interfere with other services on the
system (such as an rsyncd) and should not open any security holes.
So, if we listen on a TCP port we should choose a high-numbered one
that is dynamically checked not to be in use.
* Testing should not impose an undue load on the test machine. So,
on today's machines, using 10GB of disk to run the test suite will
discourage many people from running it. Similarly for VM and CPU
usage. Tests should probably run maximally nice. Similarly the
suite probably ought to run in less than 15minutes on typical
hardware. On the other hand there may be some bugs that only arise
on very large transfers, so it should be possible to run them if
desired.
* If the test fails, the error message should show all the
information that the developers are likely to want to know. Tests
should be as reproducible as possible, including reproduction
across platforms. So for example rather than random numbers, we
should use pseudorandom numbers from a predictable sequence.
* Testing across a network would be nice, but it is harder to
coordinate. Testing across a loopback TCP socket to localhost is
adequate.
* If facilities like running as root or having sshd are available on
a particular box it would be good to take advantage of them. We
need to distinguish "not possible to test this" from
"failed".
* The test must terminate itself and fail if it does not complete in
a reasonable amount of time.
* It's acceptable to use a scripting language like Perl, Python or
Tcl. Tcl is probably less desirable than the other two because the
language is a little ugly and not so many people know it these
days. Python is better. Since rsync is more-or-less
noninteractive using expect is not a big benefit as far as I can
see.
* Using only POSIX sh would be better because it will work on all
build farm contributed machines. I don't know if you can get
Python or Perl on Cray UNICOS, for example. You can imagine a
hybrid system like autoconf where a high-level Python program
generates a more longwinded sh script. (And we all know how
friendly autoconf is :-)
* You can use DejaGNU or some other framework if you want. Or not.
The framework generated by automake is interesting, though we
probably don't want to adopt automake.
* Automated testing Nirvana is to be able to add a test that
precisely fails on any reported bug, and that then passes when the
bug is fixed.
I don't know if I'll get around to this soon, but if anybody feels
like trying they're very welcome.
--
Martin
