I was under the impression that file ctime was consistent across nodes in a Lustre instance, but that does not seem to be the case. Please don''t be confused by the output here, the file names are also dates: align@h001:results/standard$ date && ls -l --sort=time --time=ctime | head Wed Apr 13 19:06:22 PDT 2005 total 103740128 -rw-r--r-- 1 align users 46198006 2005-04-13 19:06 2005-04-03.h001 -rw-r--r-- 1 align users 242947406 2005-04-13 19:06 2005-04-02.h001 -rw-r--r-- 1 align users 369042155 2005-04-13 19:04 2005-04-01.h001 -rw-r--r-- 1 align users 417227936 2005-04-13 19:02 2005-03-31.h001 -rw-r--r-- 1 align users 442690042 2005-04-13 19:01 2005-03-28.h001 -rw-r--r-- 1 align users 443345410 2005-04-13 19:01 2005-03-29.h001 -rw-r--r-- 1 align users 475632378 2005-04-13 18:55 2005-03-27.h001 -rw-r--r-- 1 align users 752312718 2005-04-13 18:55 2005-02-13.h001 -rw-r--r-- 1 align users 426716117 2005-04-13 18:52 2005-03-24.h001 align@h004:results/standard$ date && ls -l --sort=time --time=ctime | head Wed Apr 13 19:06:10 PDT 2005 total 103749024 -rw-r--r-- 1 align users 108 2005-04-13 18:42 2005-04-25.h001 -rw-r--r-- 1 align users 110 2005-04-13 17:48 2005-04-31.h001 -rw-r--r-- 1 align users 108 2005-04-13 17:33 2005-03-01.h001 -rw-r--r-- 1 align users 220428193 2005-04-13 16:36 2005-02-28 -rw-r--r-- 1 align users 268118093 2005-04-13 16:35 2005-02-27 -rw-r--r-- 1 align users 195607095 2005-04-13 16:35 2005-02-26 -rw-r--r-- 1 align users 178738028 2005-04-13 16:34 2005-02-25 -rw-r--r-- 1 align users 200693502 2005-04-13 16:34 2005-02-24 -rw-r--r-- 1 align users 190933631 2005-04-13 16:34 2005-02-23 align@h001:results/standard$ ls -l 2005-04-03.h001 --time=ctime -rw-r--r-- 1 align users 79010868 2005-04-13 19:13 2005-04-03.h001 align@h004:results/standard$ ls -l --time=ctime 2005-04-03.h001 -rw-r--r-- 1 align users 82309032 2005-03-25 20:23 2005-04-03.h001 The ctime as seen on h001 is today, but the ctime as seen on all other hosts is March 25. The correct ctime is March 25. This is making find -newer $given and other types of scripts work improperly. I can understand why atime might not be consistent, but ctime and mtime really must be before you can do anything of use with the FS. Maybe there''s a mount option I need? -jwb
On Apr 13, 2005 19:18 -0700, Jeffrey Baker wrote:> I was under the impression that file ctime was consistent across nodes > in a Lustre instance, but that does not seem to be the case. Please > don''t be confused by the output here, the file names are also dates: > > align@h001:results/standard$ date && ls -l --sort=time --time=ctime | > Wed Apr 13 19:06:22 PDT 2005 > -rw-r--r-- 1 align users 46198006 2005-04-13 19:06 2005-04-03.h001 > > align@h001:results/standard$ ls -l 2005-04-03.h001 --time=ctime > -rw-r--r-- 1 align users 79010868 2005-04-13 19:13 2005-04-03.h001 > > align@h004:results/standard$ ls -l --time=ctime 2005-04-03.h001 > -rw-r--r-- 1 align users 82309032 2005-03-25 20:23 2005-04-03.h001 > > The ctime as seen on h001 is today, but the ctime as seen on all other > hosts is March 25. The correct ctime is March 25.It would appear from these outputs that the 2005-04-03.h001 file is being written to during these tests, so in fact the current ctime is correct.> This is making find > -newer $given and other types of scripts work improperly. I can > understand why atime might not be consistent, but ctime and mtime really > must be before you can do anything of use with the FS.What version of Lustre are you running? There were some bugs (4829 shows in the ChangeLog for 1.2.8) wherein files currently being written to had the size updated on other clients, but the timestamps were not. I believe if the clients lose these inodes or locks from cache they will get the updated timestamps. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Hi, I''m not sure exactly what semantics Lustre aims for, but I think the POSIX (IEEE/ANSI 1003.1) spec does not specify the ctime to be sequentially consistent. I believe it says something like upon change the file''s ctime shall be marked for update, meaning that the ctime should be updated at some point in the future, but not necessarily in a way that makes it appear that all processes across nodes share a single inode cache. The relaxed consistency of ctime was likely meant, originally, to avoid disk operations on a local file system (so that the inode would not have to be committed to disk with every write). In a distributed environment, this relaxed consistency would allow for greater system scalability. Lex On Wed, 13 Apr 2005, Jeffrey Baker wrote:> I was under the impression that file ctime was consistent across nodes > in a Lustre instance, but that does not seem to be the case. Please > don''t be confused by the output here, the file names are also dates: > > align@h001:results/standard$ date && ls -l --sort=time --time=ctime | > head > Wed Apr 13 19:06:22 PDT 2005 > total 103740128 > -rw-r--r-- 1 align users 46198006 2005-04-13 19:06 2005-04-03.h001 > -rw-r--r-- 1 align users 242947406 2005-04-13 19:06 2005-04-02.h001 > -rw-r--r-- 1 align users 369042155 2005-04-13 19:04 2005-04-01.h001 > -rw-r--r-- 1 align users 417227936 2005-04-13 19:02 2005-03-31.h001 > -rw-r--r-- 1 align users 442690042 2005-04-13 19:01 2005-03-28.h001 > -rw-r--r-- 1 align users 443345410 2005-04-13 19:01 2005-03-29.h001 > -rw-r--r-- 1 align users 475632378 2005-04-13 18:55 2005-03-27.h001 > -rw-r--r-- 1 align users 752312718 2005-04-13 18:55 2005-02-13.h001 > -rw-r--r-- 1 align users 426716117 2005-04-13 18:52 2005-03-24.h001 > > align@h004:results/standard$ date && ls -l --sort=time --time=ctime | head > Wed Apr 13 19:06:10 PDT 2005 > total 103749024 > -rw-r--r-- 1 align users 108 2005-04-13 18:42 2005-04-25.h001 > -rw-r--r-- 1 align users 110 2005-04-13 17:48 2005-04-31.h001 > -rw-r--r-- 1 align users 108 2005-04-13 17:33 2005-03-01.h001 > -rw-r--r-- 1 align users 220428193 2005-04-13 16:36 2005-02-28 > -rw-r--r-- 1 align users 268118093 2005-04-13 16:35 2005-02-27 > -rw-r--r-- 1 align users 195607095 2005-04-13 16:35 2005-02-26 > -rw-r--r-- 1 align users 178738028 2005-04-13 16:34 2005-02-25 > -rw-r--r-- 1 align users 200693502 2005-04-13 16:34 2005-02-24 > -rw-r--r-- 1 align users 190933631 2005-04-13 16:34 2005-02-23 > > align@h001:results/standard$ ls -l 2005-04-03.h001 --time=ctime > -rw-r--r-- 1 align users 79010868 2005-04-13 19:13 2005-04-03.h001 > > align@h004:results/standard$ ls -l --time=ctime 2005-04-03.h001 > -rw-r--r-- 1 align users 82309032 2005-03-25 20:23 2005-04-03.h001 > > The ctime as seen on h001 is today, but the ctime as seen on all other > hosts is March 25. The correct ctime is March 25. This is making find > -newer $given and other types of scripts work improperly. I can > understand why atime might not be consistent, but ctime and mtime really > must be before you can do anything of use with the FS. > > Maybe there''s a mount option I need? > > -jwb > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss >