Afternoon Following my previous question about atime, we have noticed that atime is not consistent across lustre? Is this expected? We wrote a few line piece of code which reads a byte from a file, waits 10s and reads another byte etc. Locally, the atime gets updated after each byte read. However, on other nodes, the atime is not updated. Even after the file has been closed, the atime on other nodes is still at ctime. We notice it can take quite a long time for the atime''s to be consistent across the cluster. Is this expected? The reason I ask is that we have an application which is sensitive to atime changes and it is running really really slowely. Thanks Stu. -- Dr Stuart Midgley sdm900@gmail.com
On Jul 16, 2007 16:00 +0800, Stuart Midgley wrote:> Following my previous question about atime, we have noticed that atime is > not consistent across lustre? Is this expected?There are no guarantees that the atime is kept coherent across the cluster. The overhead of doing this far outweighs the benefits for the majority of cases. Unfortunately, the POSIX "stat" API does not tell the kernel whether a specific field will be used, and atime is seldom used (hence proliferation of noatime, nodiratime, lazyatime mount options to remove the overhead of maintaining it).> We wrote a few line piece of code which reads a byte from a file, waits > 10s and reads another byte etc. Locally, the atime gets updated after > each byte read. However, on other nodes, the atime is not updated. Even > after the file has been closed, the atime on other nodes is still at > ctime. We notice it can take quite a long time for the atime''s to be > consistent across the cluster.This is likely because these clients have cached the MDS+OST locks for the file and the read on the client will not invalidate those locks. If there was other activity on those clients (e.g. lustre locks causing old entries in the lock LRU to be expired, or memory pressure causing the inodes to be evicted) then they would get the right atime on the next access.> Is this expected? The reason I ask is that we have an application which > is sensitive to atime changes and it is running really really slowely.It sounds like a relatively fragile application model? Could you describe in a nutshell what the application is trying to do? Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Right, that''s useful to know. The application that is sensitive to atime (and mtime) is using the combination to (dodgily) tell if a file is up-to-date with its index (which is in another file). If it isn''t, the index is rebuilt. The exact behaviour of the code isn''t critical, we can change that to match how the file system works. We just needed to know how Lustre works. Anyway, thanks for the explanation.> This is likely because these clients have cached the MDS+OST locks for the > file and the read on the client will not invalidate those locks. If there > was other activity on those clients (e.g. lustre locks causing old entries > in the lock LRU to be expired, or memory pressure causing the inodes to be > evicted) then they would get the right atime on the next access. > >> Is this expected? The reason I ask is that we have an application which >> is sensitive to atime changes and it is running really really slowely. > > It sounds like a relatively fragile application model? Could you describe > in a nutshell what the application is trying to do? > > Cheers, Andreas-- Dr Stuart Midgley sdm900@gmail.com