Mark Selby
2016-Mar-07 16:11 UTC
[Gluster-users] NFS Client issues with Gluster Server 3.6.9
Here are the logs that you requested Please let me know if I can send you anything else. I really appreciate you taking a look at this - thanks! root at dc1strg001x /root 547# gluster vol info backups Volume Name: backups Type: Replicate Volume ID: 71a26ea6-632d-4a1d-8610-e782ce2a5100 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: dc1strg001x:/zfspool/glusterfs/backups/data Brick2: dc1strg002x:/zfspool/glusterfs/backups/data Options Reconfigured: nfs.disable: off root at dc1strg001x /var/log/glusterfs 551# cat nfs.log [2016-03-07 16:03:14.257919] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/ad38be3bd1baece29e1b672e6659ae60.socket) [2016-03-07 16:03:14.267862] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-03-07 16:03:14.273283] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 [2016-03-07 16:03:14.284154] W [MSGID: 112153] [mount3.c:3929:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled! [2016-03-07 16:03:14.306163] I [rpc-drc.c:694:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF [2016-03-07 16:03:14.306216] I [MSGID: 112110] [nfs.c:1494:init] 0-nfs: NFS service started [2016-03-07 16:03:14.312901] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-03-07 16:03:14.314078] W [graph.c:357:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-03-07 16:03:14.314137] W [graph.c:357:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized [2016-03-07 16:03:14.314185] W [graph.c:357:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized [2016-03-07 16:03:14.314270] I [MSGID: 114020] [client.c:2118:notify] 0-backups-client-0: parent translators are ready, attempting connect on transport [2016-03-07 16:03:14.315341] I [MSGID: 114020] [client.c:2118:notify] 0-backups-client-1: parent translators are ready, attempting connect on transport [2016-03-07 16:03:14.315923] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-backups-client-0: changing port to 49152 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume backups-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host dc1strg001x 5: option remote-subvolume /zfspool/glusterfs/backups/data 6: option transport-type socket 7: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b 8: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3 9: option send-gids true 10: end-volume 11: 12: volume backups-client-1 13: type protocol/client 14: option ping-timeout 42 15: option remote-host dc1strg002x [2016-03-07 16:03:14.317412] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-backups-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) 16: option remote-subvolume /zfspool/glusterfs/backups/data 17: option transport-type socket 18: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b 19: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3 20: option send-gids true 21: end-volume 22: 23: volume backups-replicate-0 24: type cluster/replicate 25: subvolumes backups-client-0 backups-client-1 26: end-volume 27: 28: volume backups-dht 29: type cluster/distribute 30: subvolumes backups-replicate-0 31: end-volume 32: 33: volume backups-write-behind 34: type performance/write-behind 35: subvolumes backups-dht 36: end-volume 37: 38: volume backups 39: type debug/io-stats 40: option latency-measurement off 41: option count-fop-hits off 42: subvolumes backups-write-behind 43: end-volume 44: 45: volume nfs-server 46: type nfs/server 47: option rpc-auth.auth-glusterfs on 48: option rpc-auth.auth-unix on 49: option rpc-auth.auth-null on 50: option rpc-auth.ports.insecure on 51: option rpc-auth-allow-insecure on 52: option transport-type socket 53: option transport.socket.listen-port 2049 54: option nfs.dynamic-volumes on 55: option nfs.nlm on 56: option nfs.drc off 57: option rpc-auth.addr.backups.allow * 58: option nfs3.backups.volume-id 71a26ea6-632d-4a1d-8610-e782ce2a5100 59: option nfs.backups.disable off 60: option nfs.logs.disable off 61: option nfs.users.disable off 62: subvolumes backups 63: end-volume 64: +------------------------------------------------------------------------------+ [2016-03-07 16:03:14.318157] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-0: Connected to backups-client-0, attached to remote volume '/zfspool/glusterfs/backups/data'. [2016-03-07 16:03:14.318276] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-03-07 16:03:14.318400] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-backups-replicate-0: Subvolume 'backups-client-0' came back up; going online. [2016-03-07 16:03:14.318470] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-0: Server lk version = 1 [2016-03-07 16:03:14.496642] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-backups-client-1: changing port to 49152 (from 0) [2016-03-07 16:03:14.498394] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-backups-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-03-07 16:03:14.505580] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-1: Connected to backups-client-1, attached to remote volume '/zfspool/glusterfs/backups/data'. [2016-03-07 16:03:14.505627] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-03-07 16:03:14.506210] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-1: Server lk version = 1 [2016-03-07 16:03:14.507836] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-backups-replicate-0: selecting local read_child backups-client-0 On 3/6/16 9:13 PM, Jiffin Tony Thottan wrote:> > > On 05/03/16 07:12, Mark Selby wrote: >> I am trying to use GlusterFS as a general purpose NFS file server. I >> have tried using the FUSE client but the performance fall off vs NFS >> is quite large >> >> Both the client and the server are Ubuntu 14.04. >> >> I am using Gluster 3.6.9 because of the FUSE performance issues that >> have been reported with 3.7.8 (see >> https://bugzilla.redhat.com/show_bug.cgi?id=1309462) >> >> I am having serious issues with a generic NFS client as shown by the >> issues below. Basically most FOPs are giving me a Remote I/O error. >> >> I would not think I was 1st person to see these issues - but my >> Google Fu is not working. >> >> Any and all help would be much appreciated >> >> BTW - These operation against a plain Linux NFS server work fine. >> >> >> root at dc1strg001x /var/log 448# gluster volume status >> Status of volume: backups >> Gluster process Port Online Pid >> ------------------------------------------------------------------------------ >> >> Brick dc1strg001x:/zfspool/glusterfs/backups/data 49152 Y 6462 >> Brick dc1strg002x:/zfspool/glusterfs/backups/data 49152 Y 6382 >> NFS Server on localhost 2049 Y 6619 >> Self-heal Daemon on localhost N/A Y 6626 >> NFS Server on dc1strg002x 2049 Y 6502 >> Self-heal Daemon on dc1strg002x N/A Y 6509 >> >> >> root at vc1test001 /root 735# mount -o vers=3 -t nfs >> dc1strg001x:/backups /mnt/backups_nfs >> >> root at vc1test001 /mnt/backups_nfs 737# dd if=/dev/zero of=testfile >> bs=16k count=16384 >> 16384+0 records in >> 16384+0 records out >> 268435456 bytes (268 MB) copied, 2.46237 s, 109 MB/s >> >> root at vc1test001 /mnt/backups_nfs 738# rm testfile >> >> root at vc1test001 /mnt/backups_nfs 739# dd if=/dev/zero of=testfile >> bs=16k count=16384 >> dd: failed to open ~testfile~: Remote I/O error >> >> root at vc1test001 /var/tmp 743# rsync -av testfile /mnt/backups_nfs/ >> sending incremental file list >> testfile >> rsync: mkstemp "/mnt/backups_nfs/.testfile.bzg47C" failed: Remote I/O >> error (121) >> >> sent 1,074,004,056 bytes received 121 bytes 165,231,411.85 bytes/sec >> total size is 1,073,741,824 speedup is 1.00 >> rsync error: some files/attrs were not transferred (see previous >> errors) (code 23) at main.c(1183) [sender=3.1.0] >> > > Can you please provide the volume configuration(gluster vol info ) > and log file for nfs server which u mounted (/var/log/glusterfs) > > -- > Jiffin >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >
Soumya Koduri
2016-Mar-08 14:42 UTC
[Gluster-users] NFS Client issues with Gluster Server 3.6.9
The log file didn't have any errors logged. Please check the NFS client logs in '/var/log/messages' or using dmesg and brick logs as well. Probably strace or packet trace could help too. You could use the below command to capture the pkt trace while running the I/Os on the node where gluster-nfs server is running : $ tcpdump -i any -s 0 -w /var/tmp/nfs.pcap tcp and not port 22 Check the file later to see what operation had failed (using filters: nfs, glusterfs). Thanks, Soumya On 03/07/2016 09:41 PM, Mark Selby wrote:> Here are the logs that you requested > > Please let me know if I can send you anything else. > > I really appreciate you taking a look at this - thanks! > > > root at dc1strg001x /root 547# gluster vol info backups > > Volume Name: backups > Type: Replicate > Volume ID: 71a26ea6-632d-4a1d-8610-e782ce2a5100 > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: dc1strg001x:/zfspool/glusterfs/backups/data > Brick2: dc1strg002x:/zfspool/glusterfs/backups/data > Options Reconfigured: > nfs.disable: off > > > root at dc1strg001x /var/log/glusterfs 551# cat nfs.log > [2016-03-07 16:03:14.257919] I [MSGID: 100030] [glusterfsd.c:2318:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 > (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S > /var/run/gluster/ad38be3bd1baece29e1b672e6659ae60.socket) > [2016-03-07 16:03:14.267862] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2016-03-07 16:03:14.273283] I > [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: > Configured rpc.outstanding-rpc-limit with value 16 > [2016-03-07 16:03:14.284154] W [MSGID: 112153] > [mount3.c:3929:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled! > [2016-03-07 16:03:14.306163] I [rpc-drc.c:694:rpcsvc_drc_init] > 0-rpc-service: DRC is turned OFF > [2016-03-07 16:03:14.306216] I [MSGID: 112110] [nfs.c:1494:init] 0-nfs: > NFS service started > [2016-03-07 16:03:14.312901] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 2 > [2016-03-07 16:03:14.314078] W [graph.c:357:_log_if_unknown_option] > 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized > [2016-03-07 16:03:14.314137] W [graph.c:357:_log_if_unknown_option] > 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized > [2016-03-07 16:03:14.314185] W [graph.c:357:_log_if_unknown_option] > 0-nfs-server: option 'transport-type' is not recognized > [2016-03-07 16:03:14.314270] I [MSGID: 114020] [client.c:2118:notify] > 0-backups-client-0: parent translators are ready, attempting connect on > transport > [2016-03-07 16:03:14.315341] I [MSGID: 114020] [client.c:2118:notify] > 0-backups-client-1: parent translators are ready, attempting connect on > transport > [2016-03-07 16:03:14.315923] I [rpc-clnt.c:1847:rpc_clnt_reconfig] > 0-backups-client-0: changing port to 49152 (from 0) > Final graph: > +------------------------------------------------------------------------------+ > > 1: volume backups-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host dc1strg001x > 5: option remote-subvolume /zfspool/glusterfs/backups/data > 6: option transport-type socket > 7: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b > 8: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3 > 9: option send-gids true > 10: end-volume > 11: > 12: volume backups-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host dc1strg002x > [2016-03-07 16:03:14.317412] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-backups-client-0: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > 16: option remote-subvolume /zfspool/glusterfs/backups/data > 17: option transport-type socket > 18: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b > 19: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3 > 20: option send-gids true > 21: end-volume > 22: > 23: volume backups-replicate-0 > 24: type cluster/replicate > 25: subvolumes backups-client-0 backups-client-1 > 26: end-volume > 27: > 28: volume backups-dht > 29: type cluster/distribute > 30: subvolumes backups-replicate-0 > 31: end-volume > 32: > 33: volume backups-write-behind > 34: type performance/write-behind > 35: subvolumes backups-dht > 36: end-volume > 37: > 38: volume backups > 39: type debug/io-stats > 40: option latency-measurement off > 41: option count-fop-hits off > 42: subvolumes backups-write-behind > 43: end-volume > 44: > 45: volume nfs-server > 46: type nfs/server > 47: option rpc-auth.auth-glusterfs on > 48: option rpc-auth.auth-unix on > 49: option rpc-auth.auth-null on > 50: option rpc-auth.ports.insecure on > 51: option rpc-auth-allow-insecure on > 52: option transport-type socket > 53: option transport.socket.listen-port 2049 > 54: option nfs.dynamic-volumes on > 55: option nfs.nlm on > 56: option nfs.drc off > 57: option rpc-auth.addr.backups.allow * > 58: option nfs3.backups.volume-id > 71a26ea6-632d-4a1d-8610-e782ce2a5100 > 59: option nfs.backups.disable off > 60: option nfs.logs.disable off > 61: option nfs.users.disable off > 62: subvolumes backups > 63: end-volume > 64: > +------------------------------------------------------------------------------+ > > [2016-03-07 16:03:14.318157] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-0: > Connected to backups-client-0, attached to remote volume > '/zfspool/glusterfs/backups/data'. > [2016-03-07 16:03:14.318276] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-0: > Server and Client lk-version numbers are not same, reopening the fds > [2016-03-07 16:03:14.318400] I [MSGID: 108005] > [afr-common.c:3841:afr_notify] 0-backups-replicate-0: Subvolume > 'backups-client-0' came back up; going online. > [2016-03-07 16:03:14.318470] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-0: > Server lk version = 1 > [2016-03-07 16:03:14.496642] I [rpc-clnt.c:1847:rpc_clnt_reconfig] > 0-backups-client-1: changing port to 49152 (from 0) > [2016-03-07 16:03:14.498394] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-backups-client-1: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2016-03-07 16:03:14.505580] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-1: > Connected to backups-client-1, attached to remote volume > '/zfspool/glusterfs/backups/data'. > [2016-03-07 16:03:14.505627] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-1: > Server and Client lk-version numbers are not same, reopening the fds > [2016-03-07 16:03:14.506210] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-1: > Server lk version = 1 > [2016-03-07 16:03:14.507836] I [MSGID: 108031] > [afr-common.c:1782:afr_local_discovery_cbk] 0-backups-replicate-0: > selecting local read_child backups-client-0 > > > > > On 3/6/16 9:13 PM, Jiffin Tony Thottan wrote: >> >> >> On 05/03/16 07:12, Mark Selby wrote: >>> I am trying to use GlusterFS as a general purpose NFS file server. I >>> have tried using the FUSE client but the performance fall off vs NFS >>> is quite large >>> >>> Both the client and the server are Ubuntu 14.04. >>> >>> I am using Gluster 3.6.9 because of the FUSE performance issues that >>> have been reported with 3.7.8 (see >>> https://bugzilla.redhat.com/show_bug.cgi?id=1309462) >>> >>> I am having serious issues with a generic NFS client as shown by the >>> issues below. Basically most FOPs are giving me a Remote I/O error. >>> >>> I would not think I was 1st person to see these issues - but my >>> Google Fu is not working. >>> >>> Any and all help would be much appreciated >>> >>> BTW - These operation against a plain Linux NFS server work fine. >>> >>> >>> root at dc1strg001x /var/log 448# gluster volume status >>> Status of volume: backups >>> Gluster process Port Online Pid >>> ------------------------------------------------------------------------------ >>> >>> Brick dc1strg001x:/zfspool/glusterfs/backups/data 49152 Y 6462 >>> Brick dc1strg002x:/zfspool/glusterfs/backups/data 49152 Y 6382 >>> NFS Server on localhost 2049 Y 6619 >>> Self-heal Daemon on localhost N/A Y 6626 >>> NFS Server on dc1strg002x 2049 Y 6502 >>> Self-heal Daemon on dc1strg002x N/A Y 6509 >>> >>> >>> root at vc1test001 /root 735# mount -o vers=3 -t nfs >>> dc1strg001x:/backups /mnt/backups_nfs >>> >>> root at vc1test001 /mnt/backups_nfs 737# dd if=/dev/zero of=testfile >>> bs=16k count=16384 >>> 16384+0 records in >>> 16384+0 records out >>> 268435456 bytes (268 MB) copied, 2.46237 s, 109 MB/s >>> >>> root at vc1test001 /mnt/backups_nfs 738# rm testfile >>> >>> root at vc1test001 /mnt/backups_nfs 739# dd if=/dev/zero of=testfile >>> bs=16k count=16384 >>> dd: failed to open ~testfile~: Remote I/O error >>> >>> root at vc1test001 /var/tmp 743# rsync -av testfile /mnt/backups_nfs/ >>> sending incremental file list >>> testfile >>> rsync: mkstemp "/mnt/backups_nfs/.testfile.bzg47C" failed: Remote I/O >>> error (121) >>> >>> sent 1,074,004,056 bytes received 121 bytes 165,231,411.85 bytes/sec >>> total size is 1,073,741,824 speedup is 1.00 >>> rsync error: some files/attrs were not transferred (see previous >>> errors) (code 23) at main.c(1183) [sender=3.1.0] >>> >> >> Can you please provide the volume configuration(gluster vol info ) >> and log file for nfs server which u mounted (/var/log/glusterfs) >> >> -- >> Jiffin >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Christian Rice
2016-Mar-09 09:25 UTC
[Gluster-users] NFS Client issues with Gluster Server 3.6.9
setting server.manage-gids to on fixed remote i/o error messages for me, accessing over nfs, gluster 3.6.2-2 on Debian Wheezy. worth a shot. From: <gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org>> on behalf of Mark Selby <mselby at unseelie.name<mailto:mselby at unseelie.name>> Date: Monday, March 7, 2016 at 8:11 AM To: Jiffin Tony Thottan <jthottan at redhat.com<mailto:jthottan at redhat.com>>, "gluster-users at gluster.org<mailto:gluster-users at gluster.org>" <gluster-users at gluster.org<mailto:gluster-users at gluster.org>> Subject: Re: [Gluster-users] NFS Client issues with Gluster Server 3.6.9 Here are the logs that you requested Please let me know if I can send you anything else. I really appreciate you taking a look at this - thanks! root at dc1strg001x /root 547# gluster vol info backups Volume Name: backups Type: Replicate Volume ID: 71a26ea6-632d-4a1d-8610-e782ce2a5100 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: dc1strg001x:/zfspool/glusterfs/backups/data Brick2: dc1strg002x:/zfspool/glusterfs/backups/data Options Reconfigured: nfs.disable: off root at dc1strg001x /var/log/glusterfs 551# cat nfs.log [2016-03-07 16:03:14.257919] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/ad38be3bd1baece29e1b672e6659ae60.socket) [2016-03-07 16:03:14.267862] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-03-07 16:03:14.273283] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 [2016-03-07 16:03:14.284154] W [MSGID: 112153] [mount3.c:3929:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled! [2016-03-07 16:03:14.306163] I [rpc-drc.c:694:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF [2016-03-07 16:03:14.306216] I [MSGID: 112110] [nfs.c:1494:init] 0-nfs: NFS service started [2016-03-07 16:03:14.312901] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-03-07 16:03:14.314078] W [graph.c:357:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized [2016-03-07 16:03:14.314137] W [graph.c:357:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized [2016-03-07 16:03:14.314185] W [graph.c:357:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized [2016-03-07 16:03:14.314270] I [MSGID: 114020] [client.c:2118:notify] 0-backups-client-0: parent translators are ready, attempting connect on transport [2016-03-07 16:03:14.315341] I [MSGID: 114020] [client.c:2118:notify] 0-backups-client-1: parent translators are ready, attempting connect on transport [2016-03-07 16:03:14.315923] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-backups-client-0: changing port to 49152 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume backups-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host dc1strg001x 5: option remote-subvolume /zfspool/glusterfs/backups/data 6: option transport-type socket 7: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b 8: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3 9: option send-gids true 10: end-volume 11: 12: volume backups-client-1 13: type protocol/client 14: option ping-timeout 42 15: option remote-host dc1strg002x [2016-03-07 16:03:14.317412] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-backups-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) 16: option remote-subvolume /zfspool/glusterfs/backups/data 17: option transport-type socket 18: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b 19: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3 20: option send-gids true 21: end-volume 22: 23: volume backups-replicate-0 24: type cluster/replicate 25: subvolumes backups-client-0 backups-client-1 26: end-volume 27: 28: volume backups-dht 29: type cluster/distribute 30: subvolumes backups-replicate-0 31: end-volume 32: 33: volume backups-write-behind 34: type performance/write-behind 35: subvolumes backups-dht 36: end-volume 37: 38: volume backups 39: type debug/io-stats 40: option latency-measurement off 41: option count-fop-hits off 42: subvolumes backups-write-behind 43: end-volume 44: 45: volume nfs-server 46: type nfs/server 47: option rpc-auth.auth-glusterfs on 48: option rpc-auth.auth-unix on 49: option rpc-auth.auth-null on 50: option rpc-auth.ports.insecure on 51: option rpc-auth-allow-insecure on 52: option transport-type socket 53: option transport.socket.listen-port 2049 54: option nfs.dynamic-volumes on 55: option nfs.nlm on 56: option nfs.drc off 57: option rpc-auth.addr.backups.allow * 58: option nfs3.backups.volume-id 71a26ea6-632d-4a1d-8610-e782ce2a5100 59: option nfs.backups.disable off 60: option nfs.logs.disable off 61: option nfs.users.disable off 62: subvolumes backups 63: end-volume 64: +------------------------------------------------------------------------------+ [2016-03-07 16:03:14.318157] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-0: Connected to backups-client-0, attached to remote volume '/zfspool/glusterfs/backups/data'. [2016-03-07 16:03:14.318276] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-03-07 16:03:14.318400] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-backups-replicate-0: Subvolume 'backups-client-0' came back up; going online. [2016-03-07 16:03:14.318470] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-0: Server lk version = 1 [2016-03-07 16:03:14.496642] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-backups-client-1: changing port to 49152 (from 0) [2016-03-07 16:03:14.498394] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-backups-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-03-07 16:03:14.505580] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-1: Connected to backups-client-1, attached to remote volume '/zfspool/glusterfs/backups/data'. [2016-03-07 16:03:14.505627] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-03-07 16:03:14.506210] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-1: Server lk version = 1 [2016-03-07 16:03:14.507836] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-backups-replicate-0: selecting local read_child backups-client-0 On 3/6/16 9:13 PM, Jiffin Tony Thottan wrote: On 05/03/16 07:12, Mark Selby wrote: I am trying to use GlusterFS as a general purpose NFS file server. I have tried using the FUSE client but the performance fall off vs NFS is quite large Both the client and the server are Ubuntu 14.04. I am using Gluster 3.6.9 because of the FUSE performance issues that have been reported with 3.7.8 (see https://bugzilla.redhat.com/show_bug.cgi?id=1309462) I am having serious issues with a generic NFS client as shown by the issues below. Basically most FOPs are giving me a Remote I/O error. I would not think I was 1st person to see these issues - but my Google Fu is not working. Any and all help would be much appreciated BTW - These operation against a plain Linux NFS server work fine. root at dc1strg001x /var/log 448# gluster volume status Status of volume: backups Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick dc1strg001x:/zfspool/glusterfs/backups/data 49152 Y 6462 Brick dc1strg002x:/zfspool/glusterfs/backups/data 49152 Y 6382 NFS Server on localhost 2049 Y 6619 Self-heal Daemon on localhost N/A Y 6626 NFS Server on dc1strg002x 2049 Y 6502 Self-heal Daemon on dc1strg002x N/A Y 6509 root at vc1test001 /root 735# mount -o vers=3 -t nfs dc1strg001x:/backups /mnt/backups_nfs root at vc1test001 /mnt/backups_nfs 737# dd if=/dev/zero of=testfile bs=16k count=16384 16384+0 records in 16384+0 records out 268435456 bytes (268 MB) copied, 2.46237 s, 109 MB/s root at vc1test001 /mnt/backups_nfs 738# rm testfile root at vc1test001 /mnt/backups_nfs 739# dd if=/dev/zero of=testfile bs=16k count=16384 dd: failed to open ~testfile~: Remote I/O error root at vc1test001 /var/tmp 743# rsync -av testfile /mnt/backups_nfs/ sending incremental file list testfile rsync: mkstemp "/mnt/backups_nfs/.testfile.bzg47C" failed: Remote I/O error (121) sent 1,074,004,056 bytes received 121 bytes 165,231,411.85 bytes/sec total size is 1,073,741,824 speedup is 1.00 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0] Can you please provide the volume configuration(gluster vol info ) and log file for nfs server which u mounted (/var/log/glusterfs) -- Jiffin _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160309/959eedef/attachment.html>