Xapian devel - Mar 2007 - stemtest failing with romanian

On Tuesday, I replaced the romanian1 and romanian2 stemmers in 
Xapian-core with Martin's new romanian stemmer.  At the time, I also 
updated the stemming test data (by re-generating the output file using 
snowball's "stemwords" utility), and I clearly remember re-running
the
testsuite at the time and checking that all tests passed.

Now, when I run make check, stemtest fails with the romanian stemmer on 
the word "acela?I".  This should stem to "acel" according to
the output
file generated by the stemwords utility from snowball, but xapian stems 
it to "acela?i".  The only change I can see to the stemming algorithms
since then is a change to the snowball code generator made on wednesday 
morning by Olly, but reverting this change doesn't seem to fix the 
problem.  So - does anyone else see this error, or is it just something 
on my local machine which has changed (possibly a character set thing, 
or something like that)?

For reference, the output from stemtest is as follows:

$ ./tests/runtest tests/stemtest stemdict -l romanian -v
Running test 'tests/stemtest stemdict -l romanian -v' under valgrind
The random seed is 42
Please report the seed when reporting a test failure.
Running tests with romanian stemmer...
Running test: stemdict...
Testing romanian with fixed dictionary...
/home/richard/private/Working/xapian/xapian-core/tests/stemtest.cc:146: 
((stem) == (expect))
Expected `stem' and `expect' to be equal: were acela?i and acel

  FAILED
/home/richard/private/Working/xapian/build/xapian-core/tests/.libs/lt-stemtest 
completed test run: 0 tests passed, 1 failed.


-- 
Richard

On Thu, Mar 29, 2007 at 05:18:11PM +0100, Richard Boulton
wrote:
> Now, when I run make check, stemtest fails with the romanian stemmer on 
> the word "acela??I".  This should stem to "acel"
according to the output
> file generated by the stemwords utility from snowball, but xapian stems 
> it to "acela??i".  The only change I can see to the stemming
algorithms
> since then is a change to the snowball code generator made on wednesday 
> morning by Olly, but reverting this change doesn't seem to fix the 
> problem.
I did run stemtest before committing that change, but it looks like
xapian-data wasn't updated then so it didn't test romanian.  I do now
see the same problem as you.

However, I don't think the code generator change is to blame.  It looks
to me like romanian test data just isn't in step with that from snowball
for some reason:

snowball:
-rw-r--r-- 1 olly olly 171550 2007-03-27 13:59 output.txt
-rw-r--r-- 1 olly olly 202066 2007-03-27 13:59 voc.txt

xapian-data:
-rw-r--r-- 1 olly olly 533334 2007-03-27 14:46 romanian.st
-rw-r--r-- 1 olly olly 690463 2007-03-27 14:46 romanian.voc

Cheers,
    Olly