I ran the same dd tests from KnowYourNFSAlpha-1.pdf and performance is inconsistent and causes the server to become unresponsive. My server freezes every time when I run the following command: dd if=/dev/zero of=garb bs=256k count=64000 I would also like to mount a path like: /volume/some/random/dir # mount host:/gluster/tmp /mnt/test mount: host:/gluster/tmp failed, reason given by server: No such file or directory I can mount it up host:/volume_name and /mnt/test/tmp exists dd if=/dev/zero of=garb bs=64K count=100 100+0 records in 100+0 records out 6553600 bytes (6.6 MB) copied, 0.068906 seconds, 95.1 MB/s dd of=garb if=/dev/zero bs=64K count=100 100+0 records in 100+0 records out 6553600 bytes (6.6 MB) copied, 0.057207 seconds, 115 MB/s dd if=/dev/zero of=garb bs=64K count=1000 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 0.523117 seconds, 125 MB/s dd of=garb if=/dev/zero bs=64K count=1000 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 1.04666 seconds, 62.6 MB/s dd if=/dev/zero of=garb bs=64K count=10000 10000+0 records in 10000+0 records out 655360000 bytes (655 MB) copied, 10.9809 seconds, 59.7 MB/s dd of=garb if=/dev/zero bs=64K count=10000 10000+0 records in 10000+0 records out 655360000 bytes (655 MB) copied, 11.3515 seconds, 57.7 MB/s dd if=/dev/zero of=garb bs=128K count=100 100+0 records in 100+0 records out 13107200 bytes (13 MB) copied, 0.105364 seconds, 124 MB/s dd of=garb if=/dev/zero bs=128K count=100 100+0 records in 100+0 records out 13107200 bytes (13 MB) copied, 0.254225 seconds, 51.6 MB/s dd if=/dev/zero of=garb bs=128K count=1000 1000+0 records in 1000+0 records out 131072000 bytes (131 MB) copied, 60.1008 seconds, 2.2 MB/s dd of=garb if=/dev/zero bs=128K count=1000 1000+0 records in 1000+0 records out 131072000 bytes (131 MB) copied, 1.51868 seconds, 86.3 MB/s dd if=/dev/zero of=garb bs=128K count=10000 10000+0 records in 10000+0 records out 1310720000 bytes (1.3 GB) copied, 18.7755 seconds, 69.8 MB/s dd of=garb if=/dev/zero bs=128K count=10000 10000+0 records in 10000+0 records out 1310720000 bytes (1.3 GB) copied, 18.9837 seconds, 69.0 MB/s dd if=/dev/zero of=garb bs=256k count=64000 My server freezes. Here is the recent nfs log when the server froze: [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: XID: 6f68c85f, WRITE: args: FH: hashcount 2, xlid 0, gen 5458285267163021319, ino 11856898, offset: 1129578496, count: 65536, UNSTABLE [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: RPC XID: 7068c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: XID: 7068c85f, WRITE: args: FH: hashcount 2, xlid 0, gen 5458285267163021319, ino 11856898, offset: 1129644032, count: 65536, UNSTABLE [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: RPC XID: 7168c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: XID: 7168c85f, WRITE: args: FH: hashcount 2, xlid 0, gen 5458285267163021319, ino 11856898, offset: 1129709568, count: 65536, UNSTABLE [2010-04-09 23:38:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: RPC XID: 6268c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 Thanks, Chad Richards
Thanks for the feedback, Chad. We will look at this and get back to you. Some more information, like gluster config and hardware/OS information will be useful. Regards, Tejas. ----- Original Message ----- From: "chadr" <chadr at mail.aspsys.com> To: Gluster-users at gluster.org Sent: Saturday, April 10, 2010 10:04:59 AM Subject: [Gluster-users] nfs-alpha feedback I ran the same dd tests from KnowYourNFSAlpha-1.pdf and performance is inconsistent and causes the server to become unresponsive. My server freezes every time when I run the following command: dd if=/dev/zero of=garb bs=256k count=64000 I would also like to mount a path like: /volume/some/random/dir # mount host:/gluster/tmp /mnt/test mount: host:/gluster/tmp failed, reason given by server: No such file or directory I can mount it up host:/volume_name and /mnt/test/tmp exists dd if=/dev/zero of=garb bs=64K count=100 100+0 records in 100+0 records out 6553600 bytes (6.6 MB) copied, 0.068906 seconds, 95.1 MB/s dd of=garb if=/dev/zero bs=64K count=100 100+0 records in 100+0 records out 6553600 bytes (6.6 MB) copied, 0.057207 seconds, 115 MB/s dd if=/dev/zero of=garb bs=64K count=1000 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 0.523117 seconds, 125 MB/s dd of=garb if=/dev/zero bs=64K count=1000 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 1.04666 seconds, 62.6 MB/s dd if=/dev/zero of=garb bs=64K count=10000 10000+0 records in 10000+0 records out 655360000 bytes (655 MB) copied, 10.9809 seconds, 59.7 MB/s dd of=garb if=/dev/zero bs=64K count=10000 10000+0 records in 10000+0 records out 655360000 bytes (655 MB) copied, 11.3515 seconds, 57.7 MB/s dd if=/dev/zero of=garb bs=128K count=100 100+0 records in 100+0 records out 13107200 bytes (13 MB) copied, 0.105364 seconds, 124 MB/s dd of=garb if=/dev/zero bs=128K count=100 100+0 records in 100+0 records out 13107200 bytes (13 MB) copied, 0.254225 seconds, 51.6 MB/s dd if=/dev/zero of=garb bs=128K count=1000 1000+0 records in 1000+0 records out 131072000 bytes (131 MB) copied, 60.1008 seconds, 2.2 MB/s dd of=garb if=/dev/zero bs=128K count=1000 1000+0 records in 1000+0 records out 131072000 bytes (131 MB) copied, 1.51868 seconds, 86.3 MB/s dd if=/dev/zero of=garb bs=128K count=10000 10000+0 records in 10000+0 records out 1310720000 bytes (1.3 GB) copied, 18.7755 seconds, 69.8 MB/s dd of=garb if=/dev/zero bs=128K count=10000 10000+0 records in 10000+0 records out 1310720000 bytes (1.3 GB) copied, 18.9837 seconds, 69.0 MB/s dd if=/dev/zero of=garb bs=256k count=64000 My server freezes. Here is the recent nfs log when the server froze: [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: XID: 6f68c85f, WRITE: args: FH: hashcount 2, xlid 0, gen 5458285267163021319, ino 11856898, offset: 1129578496, count: 65536, UNSTABLE [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: RPC XID: 7068c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: XID: 7068c85f, WRITE: args: FH: hashcount 2, xlid 0, gen 5458285267163021319, ino 11856898, offset: 1129644032, count: 65536, UNSTABLE [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: RPC XID: 7168c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: NFS3 - WRITE [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: XID: 7168c85f, WRITE: args: FH: hashcount 2, xlid 0, gen 5458285267163021319, ino 11856898, offset: 1129709568, count: 65536, UNSTABLE [2010-04-09 23:38:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: RPC XID: 6268c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 Thanks, Chad Richards _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Hi Chad I've filed a bug at: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=819 Would you please upload the complete log file from NFS server, GlusterFS server and the volume files for the same to this bug report? Thanks -Shehjar chadr wrote:
chadr wrote:> I ran the same dd tests from KnowYourNFSAlpha-1.pdf and performance is > inconsistent and causes the server to become unresponsive. > > My server freezes every time when I run the following command: > dd if=/dev/zero of=garb bs=256k count=64000 >Please confirm whether the server freezes or crashes? I ask because we've had a couple of reports of hanging/freezing servers when actually they had crashed. The way to figure it out is to let us have the complete log file, or at least the last few hundred lines.> I would also like to mount a path like: /volume/some/random/dir > > # mount host:/gluster/tmp /mnt/test > mount: host:/gluster/tmp failed, reason given by server: No such file or > directory > > I can mount it up host:/volume_name and /mnt/test/tmp exists >I see you're trying to mount a directory from a GlusterFS subvolume as an NFS export by itself. This is not supported in the Alpha release. NFS server right now allows each GlusterFS subvolume to be a single NFS export. We are, however, planning to introduce this functionality in the near future. Thanks -Shehjar> dd if=/dev/zero of=garb bs=64K count=100 > 100+0 records in > 100+0 records out > 6553600 bytes (6.6 MB) copied, 0.068906 seconds, 95.1 MB/s > > dd of=garb if=/dev/zero bs=64K count=100 > 100+0 records in > 100+0 records out > 6553600 bytes (6.6 MB) copied, 0.057207 seconds, 115 MB/s > > dd if=/dev/zero of=garb bs=64K count=1000 > 1000+0 records in > 1000+0 records out > 65536000 bytes (66 MB) copied, 0.523117 seconds, 125 MB/s > > dd of=garb if=/dev/zero bs=64K count=1000 > 1000+0 records in > 1000+0 records out > 65536000 bytes (66 MB) copied, 1.04666 seconds, 62.6 MB/s > > dd if=/dev/zero of=garb bs=64K count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes (655 MB) copied, 10.9809 seconds, 59.7 MB/s > > dd of=garb if=/dev/zero bs=64K count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes (655 MB) copied, 11.3515 seconds, 57.7 MB/s > > dd if=/dev/zero of=garb bs=128K count=100 > 100+0 records in > 100+0 records out > 13107200 bytes (13 MB) copied, 0.105364 seconds, 124 MB/s > > dd of=garb if=/dev/zero bs=128K count=100 > 100+0 records in > 100+0 records out > 13107200 bytes (13 MB) copied, 0.254225 seconds, 51.6 MB/s > > dd if=/dev/zero of=garb bs=128K count=1000 > 1000+0 records in > 1000+0 records out > 131072000 bytes (131 MB) copied, 60.1008 seconds, 2.2 MB/s > > dd of=garb if=/dev/zero bs=128K count=1000 > 1000+0 records in > 1000+0 records out > 131072000 bytes (131 MB) copied, 1.51868 seconds, 86.3 MB/s > > dd if=/dev/zero of=garb bs=128K count=10000 > 10000+0 records in > 10000+0 records out > 1310720000 bytes (1.3 GB) copied, 18.7755 seconds, 69.8 MB/s > > dd of=garb if=/dev/zero bs=128K count=10000 > 10000+0 records in > 10000+0 records out > 1310720000 bytes (1.3 GB) copied, 18.9837 seconds, 69.0 MB/s > > dd if=/dev/zero of=garb bs=256k count=64000 > My server freezes. > > > Here is the recent nfs log when the server froze: > > [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: > XID: 6f68c85f, WRITE: args: FH: hashcount 2, xlid 0, gen > 5458285267163021319, ino 11856898, offset: 1129578496, count: 65536, > UNSTABLE > [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: > RPC XID: 7068c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: > XID: 7068c85f, WRITE: args: FH: hashcount 2, xlid 0, gen > 5458285267163021319, ino 11856898, offset: 1129644032, count: 65536, > UNSTABLE > [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: > RPC XID: 7168c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: > Actor found: NFS3 - WRITE > [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3: > XID: 7168c85f, WRITE: args: FH: hashcount 2, xlid 0, gen > 5458285267163021319, ino 11856898, offset: 1129709568, count: 65536, > UNSTABLE > [2010-04-09 23:38:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service: > RPC XID: 6268c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7 > > Thanks, > Chad Richards > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users