Paul Legato
2006-Apr-28 02:22 UTC
[Xapian-discuss] Multiple simultaneous WritableDatabases?
Hi Xapian users, Is there any way that one can write to more than one database at the same time in parallel? I'd like to keep a redundant backup copy of my index. If I can't write directly to multiple databases simultaneously, I'm thinking of rsync'ing the database directory to multiple backup servers (which could also serve as read-only slave replicators). Are there any caveats involved with copying a live database around while some process may still be potentially writing to it? Is it necessary to pause the writing processes before copying, or to lock the database in some way? Best, Paul -- -------------------------------------------------- -- Paul Legato, Senior Software Engineer -- --- Networked Knowledge Systems --- ---- P.O. Box 20772 Tampa, FL. 33622-0772 ---- ----- (813)594-0064 Voice (813)594-0045 FAX ----- ------ plegato@nks.net ------ -------------------------------------------------- -------------------------------------------------- ----- This email bound by the following: ----- ---- http://www.nks.net/email_disclaimer.html ---- --------------------------------------------------
Olly Betts
2006-Apr-28 11:57 UTC
[Xapian-discuss] Multiple simultaneous WritableDatabases?
On Thu, Apr 27, 2006 at 10:21:28PM -0300, Paul Legato wrote:> Is there any way that one can write to more than one database at the > same time in parallel? I'd like to keep a redundant backup copy of my index.There's no built-in support, but you could just create multiple WritableDatabase objects and apply the same operations to each. There's no reason why you can't call add_document with the same Document object on more than one WritableDatabase object.> If I can't write directly to multiple databases simultaneously, I'm > thinking of rsync'ing the database directory to multiple backup servers > (which could also serve as read-only slave replicators). Are there any > caveats involved with copying a live database around while some process > may still be potentially writing to it?If two or more updates get committed during the copy, you may end up with a broken database (depending which versions of which blocks get copied). One update should be OK.> Is it necessary to pause the > writing processes before copying, or to lock the database in some way?That's safest. Just opening the database as a WritableDatabase is an easy way to prevent it being updated (for quartz you'll have to manually remove the lock file if you want to update a back-up version, since the lock file will also then get backed up). If you're using rsync, you could call rsync repeatedly until no updates happened during an rsync (if you're backing up nightly, then the first rsync has a lot to do, but the subsequent ones will just be copying over the blocks changed by any committed changes since the previous rsync. Another approach is to use a filing system that allows you to take an instant snapshot of a file, and then backup the snapshot - that should always get you a valid database. Cheers, Olly