Xapian devel - May 2007 - Last minute feature for 1.0.0

I'd like to draw people's attention to bug report #143 that I've
just
submitted.  This is a proposal (and patch) to add the ability to store 
arbitrary metadata associated with a database (rather than with an 
individual document in the database).  The rationale for this feature is 
explained more fully in the bug report, but briefly I've come across 
several situations where I need to store information about how a 
database has been indexed, or which version of an external database 
corresponds to a xapian database, and it needs to be possible to ensure 
that modifications to this information are transactionally modified 
together with the Xapian database contents.

If at all possible, I'd like to get this into version 1.0.0, although I 
appreciate that it's very late in the day for such changes.  Partly, my 
desire to do this is because I want to use it now, but it's also because 
the feature requires a minor change to the flint database format, so now 
seems a good time to apply it (since people are going to have to rebuild 
their databases to use 1.0.0 anyway).

The patch only adds the feature to the flint and inmemory databases, but 
I think that adding it to remote databases wouldn't be hard at all.  I 
don't think it's worth adding it to quartz databases.

It's probably best to add further comments to the bug report at:
http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=143

-- 
Richard

On Fri, May 04, 2007 at 01:35:27PM +0100, Richard Boulton
wrote:
> If at all possible, I'd like to get this into version 1.0.0, although I
> appreciate that it's very late in the day for such changes.  Partly, my
> desire to do this is because I want to use it now, but it's also
because
> the feature requires a minor change to the flint database format, so now 
> seems a good time to apply it (since people are going to have to rebuild 
> their databases to use 1.0.0 anyway).
I understand your motivation for wanting this in 1.0.0, but I'm afraid
the time really has passed for crowbarring random new features into the
release.  I've already said "not until after 1.0.0" to patches
submitted
weeks or months ago!

We were hoping to release 1.0.0 in April, and it's already May 4th!
Each new feature diverts us from addressing the issues we've already
identified as release blockers, and has the risk of introducing new
bugs which we'll have to address before the release.  Or identify the
problematic change and back it out, which also takes time.

In fact, after a quick look at the previous change in this direction
(rev 8426) I'm a little dubious about including even that in the
release.  It's mostly a step in the right direction, but it doesn't
appear to fix any actual bugs, and for a WritableDatabase, you're now
carefully setting the total_length and lastdocid members of the
database_ro member when the database is opened, but AFAICS they'll never
get updated.  I hope they never get used, but if they do that's a new
bug!

Also, we now always retrieve and parse the metainfo record, even if we
never use the information - if you're using a weighting scheme which
ignores document length, I think that means that the first search on a
cold database will now be slower.  At least that's not the default
for BM25Weight.

And all this has distacted me from working on TermGenerator for 30
minutes or so.  And that's delayed the release of 1.0.0!

Fixes for actual bugs that affect users are still generally OK, but
otherwise I think we need to stick to the features which have already
been marked as blockers.

Cheers,
    Olly