Xapian devel - Apr 2007 - Re: [Xapian-commits] 8181: trunk/xapian-bindings/ trunk/xapian-bindings/python/

On Wed, Apr 11, 2007 at 10:31:39AM +0100, richard wrote:
> python/smoketest.py: Replace cut-and-pasted test checking code
> with functions, to simplify tests.
It might be simpler, but mostly because functionality has gone - you've
thrown away all my carefully written failure messages!

Now if a test fails, the message is very generic and you have to read
the code to find out what's wrong.  This makes handling user reports of
test failures harder, since the line numbers reported may be different
in the version they're using (and the exact version is frequently not
mentioned in initial problem reports).

It'll also be easier to keep the different language versions of the
smoketest in step if we resist the urge to mercilessly refactor them
individually...

Cheers,
    Olly

Olly Betts wrote:
> It might be simpler, but mostly because functionality has gone - you've
> thrown away all my carefully written failure messages!
Ah.  It wouldn't be that hard to put the error messages back (just by 
adding an extra argument to each checking function).

Alternatively, I could improve the error reporting, and make it just 
display the exception message and quote the lines of code around each 
the error failure, instead of displaying a raw traceback.  I could also 
make it report the xapian version number.

It seems a good plan to me to lower the barrier to writing tests, so 
avoiding having to think of a clear explanatory message for each new 
test seems a win.
> Now if a test fails, the message is very generic and you have to read
> the code to find out what's wrong.  This makes handling user reports of
> test failures harder, since the line numbers reported may be different
> in the version they're using (and the exact version is frequently not
> mentioned in initial problem reports).
Hmm.  Fair points there.
> It'll also be easier to keep the different language versions of the
> smoketest in step if we resist the urge to mercilessly refactor them
> individually...
Ah.  The reason I did all this is that I'm working on the Pythonic 
iterators, and since their implementation is relatively complicated 
they'll need quite a lot of extra checking - particularly since I'm 
trying to support the current interfaces to iterators as well as the new 
ones.  I was going to add the checks for these to the smoketest.

Perhaps the solution is to revert the changes I made to the smoketest, 
and to add a Python specific testsuite for features specific to Python.

I thought that the smoketests for each language were completely 
out-of-step already - I didn't realise I was breaking this.  I confess I 
haven't looked at the smoketests for other languages at all, lately.  Sorry.

-- 
Richard