I've been doing work to keep a xapian DB in-step with a mysql DB. I spent quite some time freaking about with mysqlbinlog replication, custom perl tail-ers, using perl mysql parsers and all kinds of magic to detect which rows in which tables have changed and therefore which dbi2omega'd records have been updated/removed. In the end various issues including buggy sql parsers made me give up, the dataset isn't very big anyway, so I an afford to rebuild every hour or so, but what would not be nice is for some poor user to get no data at the point its rebuilt. I think I can't rely on scriptindex not flushing as it will have to if the dataset gets too large, and in anycase I want to start by deleting everything. But if I use symlinks (and I do) in /var/lib/omega/data to point to the db - well as we know, its impossible to atomically update a symlink, so I can't update another DB and then symlink to it once done. Any tips for me? Otherise, as I'm parsing the XML results in php so I can just wait a second and try again if I get no data, and presume it is the not-so-atomic update. Sam
On Mon, Oct 04, 2004 at 05:05:44PM +0100, Sam Liddicott wrote:> But if I use symlinks (and I do) in /var/lib/omega/data to point to the > db - well as we know, its impossible to atomically update a symlink, so > I can't update another DB and then symlink to it once done.Do we know that? You can certainly create a new symlink with a temporary name, then use rename() to replace the old symlink with the old one. And according to "man 2 rename": If newpath already exists it will be atomically replaced (subject to a few conditions - see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. Nothing in the ERRORS section seems to say that symlinks aren't replaced atomically. Cheers, Olly
Richard Boulton wrote:>On Tue, 2004-10-05 at 09:22 +0100, Sam Liddicott wrote: > > >>Does omega chdir to the database dir before opening all the files? >> >> > >No. > > > >> Or is there a race condition as the symlink changes if a DB is being >>opened? >> >> > >I think there is. Possibly this could be solved by using readlink(2) to >read the actual path pointed to by the symlink, and constructing the >database path based on the real path. > >Repeatedly calling readlink as I now have a couple of levels of symlink redirection. Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.tartarus.org/pipermail/xapian-discuss/attachments/20041005/bb29c85d/attachment.htm