Hello all, we are running tests with 2.0.3 currently. After 24h the only critics left from our side is the timestamps of self-healed directories are still not synced correctly. Running bonnie on 2 clients for the last 20 hours revealed no problems so far. -- Regards, Stephan
----- "Stephan von Krawczynski" <skraw at ithnet.com> wrote:> Hello all, > > we are running tests with 2.0.3 currently. After 24h the only critics > left from our side is the timestamps of self-healed directories are still > not synced correctly.This bug has been fixed in the repository and is also in the 2.0.4 release. Vikas
----- "Stephan von Krawczynski" <skraw at ithnet.com> wrote:> Hello, > > this bug is _not_ fixed in 2.0.4. We just tried and the problem stays > the same. > All you have to do to reproduce is: > - take 2 servers with replicate > - copy data (with directories) onto first servers glusterfs exported > dir. > - do ls -lR on client, self healing on second server starts. > - when self-healing is done look at second servers exported dir. > find all healed directories with current timestamp from healing and > not with timestamp from original on first server.If you look closely, you'll see that the mtime is consistent, while the atime and ctime might have changed. This is because: * atime -- This is the access time. This will change with every access ("ls" or read), and hence even though it is synchronized during self-heal, it will obviously change the next time you do any access operation. * ctime -- This is the inode change time. This is entirely under the control of the kernel and there is no system call in Unix that allows us to change it. Hence we cannot synchronize this. There was indeed a bug in the previous versions which would leave mtime inconsistent too, and that has been fixed. Vikas
----- "Stephan von Krawczynski" <skraw at ithnet.com> wrote:> Sorry to say that. But if I rsync the trees they end up with _correct_ > (i.e. identical) timestamp displayed at standard "ls". > Whereas using self-heal shows different timestamps on "ls". > This at least proves that the stamps I mean are settable by user-space > tools (like rsync).Please clarify what you mean by "timestamp". As you know, there are three timestamp values, not one. And by "different", do you mean the timestamps are different on different servers, or that mtime and ctime are different on a single server? Vikas
----- "Stephan von Krawczynski" <skraw at ithnet.com> wrote:> Maybe I should more precisely explain what I am doing, in case there > are related problems. > > I am using a copy of the opensuse 11.1 DVD for that test. I copy the > whole DVD files into a directory called "suse" on the first server. The > directory "suse" is located on the top level of the gluster-exported "/p3". Then I > start "ls -lR" on one of two connected clients and watch it flow. In the > meantime you can see the directories and files being created on the second server.> After some minutes the "ls" exits correctly. Then looking at the second > servers' "suse" directory reveals it has incorrect mtime at least on the "suse" > dir. My personal opinion is that this derives from the files being added > _inside_ the directory during the healing process. Nevertheless glusterfs should > handle that case like rsync does, too.> Another thing you can see in this example: if you repeat "ls -lR" you > will notice quite some directories inside the healed "suse" tree have wrong > mtime (from the healing) displayed during ls. This proves that mtime > displayed on the client does not always come from the first server. And it proves > that this really is a production no-go, because your mtimes are indeed flapping > after this healing process.Ah, finally! Thanks for the detailed explanation. I wasn't reproducing the problem because I just tried it with empty directories, and not with directories which contained files inside them. The problem in a nutshell is this: Whenever replicate self-heal creates an entry (file or directory), it does not sync the parent directory's mtime. The fix is to sync it after every create. However, we've chosen to defer the fix, since there are some changes planned for 2.1 that will make it very easy to fix this (it is planned for all create operations to return stat info for the parent directory). Fixing it now on the other hand will involve significant work. You can track the progress of this issue at: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=137 Vikas