http://dovecot.org/tools/nfstest.c I wrote a test utility that tests different ways to flush NFS attribute cache and data cache. Please test in your NFS setup and show me the results, so I can make Dovecot v1.1 works well with NFS. Run the test several times (3 should be enough) to make sure the output stays the same, because the attribute cache flushes might incorrectly say "OK". And of course attribute cache must not be disabled for this test to work. If you have other ideas how to flush either cache please tell me and I'll update the test utility. I'd like to have a way to flush the cache without having the NFS client send anything to server. Unfortunately all of the below ways do that. Results for Linux v2.6.15 and v2.6.18: Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, gid): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, gid): OK Info: Attr cache flush chmod(mode): OK Info: Testing data cache.. Info: data cache: Appends weren't noticed (ret = 0) Info: - Attribute cache flush helped Info: data cache (no caching): failed Info: data cache (attr cache): failed Info: data cache (lockf()): OK Info: data cache (flock(shared)): OK Info: data cache (flock(exclusive)): OK -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20070711/d01c7a44/attachment-0002.bin>
On Wed, 11 Jul 2007, Timo Sirainen wrote:> http://dovecot.org/tools/nfstest.cResults for FreeBSD 6.2-RELEASE-p5 clients using a NetApp NFS server: Info: Connected: client Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing data cache.. Info: data cache: Appends weren't noticed (ret = 0) Info: - Attribute cache flush helped Info: data cache (no caching): failed Info: data cache (attr cache): OK Info: data cache (lockf()): failed Info: data cache (flock(shared)): failed Info: data cache (flock(exclusive)): failed Info: data cache (O_DIRECT): failed -Doug
On Wed, 2007-07-11 at 22:21 +0300, Timo Sirainen wrote:> http://dovecot.org/tools/nfstest.cI've done several updates for this. Updated results for Linux 2.6: Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): OK Info: Write flush flock(exclusive): OK Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: data cache (no caching): failed Info: data cache (attr cache): OK Info: data cache (fcntl(shared)): OK Info: data cache (fcntl(exclusive)): OK Info: data cache (flock(shared)): OK Info: data cache (flock(exclusive)): OK Info: data cache (dotlock): failed Info: data cache (O_DIRECT): OK -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20070712/f84c2f58/attachment-0002.bin>
Timo Sirainen wrote:> http://dovecot.org/tools/nfstest.c > > I wrote a test utility that tests different ways to flush NFS attribute > cache and data cache. Please test in your NFS setup and show me the > results, so I can make Dovecot v1.1 works well with NFS. > > Run the test several times (3 should be enough) to make sure the output > stays the same, because the attribute cache flushes might incorrectly > say "OK". And of course attribute cache must not be disabled for this > test to work. > > If you have other ideas how to flush either cache please tell me and > I'll update the test utility. I'd like to have a way to flush the cache > without having the NFS client send anything to server. Unfortunately all > of the below ways do that. > > Results for Linux v2.6.15 and v2.6.18: >What's a good command to compile it? I have a Linux 2.6.20 NFS client connected to a Linux 2.4.34 NFS client. $ gcc ~/tmp/nfstest.c -I lib In file included from lib/lib.h:24, from /home/users/andrewz/tmp/nfstest.c:17: lib/compat.h:31:3: error: invalid preprocessing directive ## In file included from /usr/include/sys/socket.h:35, from lib/compat.h:80, from lib/lib.h:24, from /home/users/andrewz/tmp/nfstest.c:17: /usr/include/bits/socket.h:36: error: conflicting types for ?socklen_t? lib/compat.h:51: error: previous declaration of ?socklen_t? was here In file included from /usr/include/netinet/in.h:23, from lib/compat.h:81, from lib/lib.h:24, from /home/users/andrewz/tmp/nfstest.c:17: /usr/include/stdint.h:111: error: conflicting types for ?uint_fast32_t? lib/compat.h:46: error: previous declaration of ?uint_fast32_t? was here /usr/include/stdint.h:141: error: conflicting types for ?uintmax_t? lib/compat.h:38: error: previous declaration of ?uintmax_t? was here Andrew
On Thu, 12 Jul 2007, Timo Sirainen wrote:> On Wed, 2007-07-11 at 22:21 +0300, Timo Sirainen wrote: >> http://dovecot.org/tools/nfstest.c > I've done several updates for this. Updated results for Linux 2.6:Different result this time: Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Fatal: open(/mnt/nfs/blah) failed: Stale NFS file handle This is on FreeBSD 6.2-RELEASE-p5 and a NetApp NFS server. I'm wondering if it is the same issue with the NFS client cache that I had to workaround in the past with dotlocking (http://www.dovecot.org/list/dovecot/2007-May/022883.html). Before I implement the same workaround, I wanted to check if it would invalidate the test results assuming the workaround worked? -Doug
Timo Sirainen wrote:> http://dovecot.org/tools/nfstest.c > > I wrote a test utility that tests different ways to flush NFS attribute > cache and data cache. Please test in your NFS setup and show me the > results, so I can make Dovecot v1.1 works well with NFS. > > Run the test several times (3 should be enough) to make sure the output > stays the same, because the attribute cache flushes might incorrectly > say "OK". And of course attribute cache must not be disabled for this > test to work. > > If you have other ideas how to flush either cache please tell me and > I'll update the test utility. I'd like to have a way to flush the cache > without having the NFS client send anything to server. Unfortunately all > of the below ways do that. > > Results for Linux v2.6.15 and v2.6.18: >** Test 1 Machine 1: Fedora Core 6, Linux 2.6.20-1.2952.fc6 #1 SMP Wed May 16 17:59:13 EDT 2007 i686 athlon i386 GNU/Linux Machine 2: Fedora Core 7, Linux 2.6.21-1.3228.fc7 #1 SMP Tue Jun 12 15:37:31 EDT 2007 i686 i686 i386 GNU/Linux NFS Server: Trustix Secure Linux 2.2, Linux 2.4.34-1tr #1 Thu Jan 4 11:36:05 GMT 2007 i686 i686 i386 GNU/Linux Because the server is Linux 2.4, that implies the protocol is NFS version 3. Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): OK Info: Write flush flock(exclusive): OK Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): OK Info: Data cache flush flock(exclusive): OK Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK ** Test 2 Machines 1 and 2 switched roles. Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): OK Info: Write flush flock(exclusive): OK Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): OK Info: Data cache flush flock(exclusive): OK Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK Andrew
> Running rpc.lockd and rpc.statd on FreeBSD 6.2-STABLE #17: Sun Jun 24 22:11:00 EDT 2007 > with a NetApp filer as the server, ran 3 times and compared output (same): >Weird, I have the exact same setup and for me this program hangs, and I have to kill lockd. Cor
Timo Sirainen wrote:> http://dovecot.org/tools/nfstest.c > > I wrote a test utility that tests different ways to flush NFS attribute > cache and data cache. Please test in your NFS setup and show me the > results, so I can make Dovecot v1.1 works well with NFS. >Results for Solaris 8 talking to a NetApp filer (OS: ONTAP 7.2.2p3) Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): OK Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): OK Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: OK Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: failed -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin at reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Timo Sirainen wrote:> http://dovecot.org/tools/nfstest.cLatest version used (2007-07-12 ~11am PST) for tests. Machine 1: CentOS 4 (RHEL 4 clone) Linux 2.6.9-42.0.8.ELsmp #1 SMP Tue Jan 30 12:33:47 EST 2007 i686 i686 i386 GNU/Linux Machine 2: CentOS 4 (RHEL 4 clone) Linux 2.6.9-42.0.10.ELsmp #1 SMP Tue Feb 27 10:11:19 EST 2007 i686 i686 i386 GNU/Linux NFS Server: SnapServer 4500 (linux software RAID based appliance) Linux 2.4.19-gos-up #1 Wed Jul 12 23:00:50 PDT 2006 i686 i686 i386 GNU/Linux machine 1: server machine 2: client ================Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: OK Info: Write flush attr cache: failed Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK Info: Data cache flush dup+close: OK machine 1: client machine 2: server ================Info: Connected: client Info: O_EXCL works Info: Testing attribute cache.. Info: Attr cache flush fchown(-1, -1): failed Info: Attr cache flush fchown(uid, -1): OK Info: Attr cache flush fchmod(mode): OK Info: Attr cache flush chown(-1, -1): failed Info: Attr cache flush chown(uid, -1): OK Info: Attr cache flush chmod(mode): OK Info: Attr cache flush dup+close: OK Info: Testing write flushing.. Info: Write flush no caching: failed Info: Write flush fcntl(shared): OK Info: Write flush fcntl(exclusive): OK Info: Write flush flock(shared): failed Info: Write flush flock(exclusive): failed Info: Write flush reopen: OK Info: Write flush dup+close: OK Info: Write flush attr cache: failed Info: Testing data cache.. Info: data cache: Reading EOF requires attribute cache flush Info: Data cache flush no caching: failed Info: Data cache flush attr cache: failed Info: Data cache flush fcntl(shared): OK Info: Data cache flush fcntl(exclusive): OK Info: Data cache flush flock(shared): failed Info: Data cache flush flock(exclusive): failed Info: Data cache flush dotlock: failed Info: Data cache flush O_DIRECT: OK Info: Data cache flush dup+close: OK -- Troy Engel | Systems Engineer Fluid, Inc | http://www.fluid.com
On Wed, 2007-07-11 at 22:21 +0300, Timo Sirainen wrote:> http://dovecot.org/tools/nfstest.cv1.1 code does now NFS cache flushing with mail_nfs_storage=yes and mail_nfs_index=yes. I haven't tried yet how well it works, but at least it finally should work without disabling attribute caches or anything. For read caching I ended up using: #ifdef __linux__ - fcntl() lock + unlock #else - fchown(uid, -1) Would have been nice to avoid the fcntl lock, but I couldn't find any other way (I even checked Linux sources). It also means that either you must use lockd with Linux or you must mount the filesystem with nolock option. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20070713/48706869/attachment-0002.bin>