Hi! I wonder about Lustre''s client-side caching strategy for file attributes and directory entries, and whether it suffers from the same problems as NFS in this regard. I seem to be unable to find this question answered on the web, so I''d be happy for any hints. Specifically, I''m interested in caching of negative entries. With /sharedfs shared via NFS, attribute caching might prevent clientA from seeing the newly created file in the final stat call of this test: On clientA: # rm -f /sharedfs/test; ls -l /sharedfs/test; ssh clientB "touch /sharedfs/test"; stat /sharedfs/test How does Lustre behave in this case? Are newly created files immediately visible on all clients, or will they cache (negative) attributes and dentries for some time? Does Posix mandate specific behaviour in this regard? Thanks for any insight, Daniel.
On Apr 24, 2009 22:52 +0200, Daniel Kobras wrote:> I wonder about Lustre''s client-side caching strategy for file attributes > and directory entries, and whether it suffers from the same problems as > NFS in this regard. I seem to be unable to find this question answered > on the web, so I''d be happy for any hints. > > Specifically, I''m interested in caching of negative entries. With > /sharedfs shared via NFS, attribute caching might prevent clientA from > seeing the newly created file in the final stat call of this test: > > On clientA: > > # rm -f /sharedfs/test; ls -l /sharedfs/test; ssh clientB "touch /sharedfs/test"; stat /sharedfs/test > > How does Lustre behave in this case?Correctly :-).> Are newly created files immediately visible on all clients, or will > they cache (negative) attributes and dentries for some time?Lustre will cache negative dentries as long as the directory is not modified. If the directory has a new file created or an existing one deleted then the negative dentries, along with the rest of the readdir data is removed from all client caches, though the individual inodes can still live in the client cache as they have their own locks. This is optimal in the most common use cases, where applications do negitive lookups only in specific directories (e.g. .../bin and .../lib directories) that rarely change.> Does Posix mandate specific behaviour in this regard?No, POSIX is only really applicable to local filesystems, and while there are a number of ways it could be interpreted for distributed filesystems, we try to follow the principle of least surprise when developing Lustre. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Hi! On Mon, Apr 27, 2009 at 01:29:20AM -0600, Andreas Dilger wrote:> Lustre will cache negative dentries as long as the directory is not > modified. If the directory has a new file created or an existing > one deleted then the negative dentries, along with the rest of the > readdir data is removed from all client caches, though the individual > inodes can still live in the client cache as they have their own locks.Interesting. Thanks for the info! This makes me wonder how Lustre copes with temporary network outages in this case. If the client drops off the net, the server cannot invalidate the caches. Does the client notice the communication problem, or could it briefly serve stale data? Regards, Daniel.
On May 18, 2009 12:16 +0200, Daniel Kobras wrote:> On Mon, Apr 27, 2009 at 01:29:20AM -0600, Andreas Dilger wrote: > > Lustre will cache negative dentries as long as the directory is not > > modified. If the directory has a new file created or an existing > > one deleted then the negative dentries, along with the rest of the > > readdir data is removed from all client caches, though the individual > > inodes can still live in the client cache as they have their own locks. > > Interesting. Thanks for the info! This makes me wonder how Lustre copes > with temporary network outages in this case. If the client drops off the > net, the server cannot invalidate the caches. Does the client notice the > communication problem, or could it briefly serve stale data?Lustre is cache coherent even after network failures. If the client had any lock that changed state while it could not communicate with the server, the server will evict the client, which forces the client to discard all of its saved state when it reconnects. If no client locks changed while it was disconnected then the client just reconnects and continues on its way. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.