Xapian discuss - Dec 2007 - backing up a database by locking while backup program runs

Hi.

I want to back up a Xapian database.  If I copy the database directory
and its contents using (for instance) rsync, and I know that throughout
the duration of the rsync process another process has had the database
open for writing, but has not actually made any modifications, can I
be sure that, on completion of the rsync process, I have a copy of the
database which is consistent and fully up-to-date?

The document at [1] makes it fairly clear that the answer is "Yes",
but I'm a little doubtful about the status of that document, since there
seem to be no links to it from the Xapian website.  (The following Google
query finds nothing: "link:www.xapian.org/docs/admin_notes.html".)

[1] http://www.xapian.org/docs/admin_notes.html

I propose to use a simple PHP script like the following in order to
obtain the write lock:

  <?php 
  $h = new XapianWritableDatabase("/my/Xapian/database", DB_OPEN);
  sleep(99999);

Regards,

Ronan.

On Thu, Dec 13, 2007 at 05:52:44PM +0000, Ronan wrote:
> I want to back up a Xapian database.  If I copy the database directory
> and its contents using (for instance) rsync, and I know that throughout
> the duration of the rsync process another process has had the database
> open for writing, but has not actually made any modifications, can I
> be sure that, on completion of the rsync process, I have a copy of the
> database which is consistent and fully up-to-date?
Providing all changes were flushed before you started the rsync, and
no changes have happened since then, you're fine.
> The document at [1] makes it fairly clear that the answer is
"Yes",
> but I'm a little doubtful about the status of that document, since
there
> seem to be no links to it from the Xapian website.  (The following Google
> query finds nothing:
"link:www.xapian.org/docs/admin_notes.html".)
> 
> [1] http://www.xapian.org/docs/admin_notes.html
We should link that in somewhere, but all those docs are distributed
as part of the core library, so it's an official document you can
depend on. (And similarly, if you find a problem, please let us know! :-)
> I propose to use a simple PHP script like the following in order to
> obtain the write lock:
> 
>   <?php 
>   $h = new XapianWritableDatabase("/my/Xapian/database",
DB_OPEN);
>   sleep(99999);
Better would be to spawn the rsync from within the PHP script, if you
can manage it (might require some ssh funkiness). That way you can't
eg background the rsync and have the lock released while it's still
running.

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james@tartarus.org                               uncertaintydivision.org

On Thu, Dec 13, 2007 at 05:52:44PM +0000, Ronan wrote:
> I want to back up a Xapian database.  If I copy the database directory
> and its contents using (for instance) rsync, and I know that throughout
> the duration of the rsync process another process has had the database
> open for writing, but has not actually made any modifications, can I
> be sure that, on completion of the rsync process, I have a copy of the
> database which is consistent and fully up-to-date?
Yes, that will work.
> The document at [1] makes it fairly clear that the answer is
"Yes",
> but I'm a little doubtful about the status of that document, since
there
> seem to be no links to it from the Xapian website.
There's a link from here ("sysadmins" at the top):

http://www.xapian.org/docs/
> (The following Google
> query finds nothing:
"link:www.xapian.org/docs/admin_notes.html".)
It's not my job to explain Google's inability to find things!

Cheers,
    Olly