The RDMA warnings are not relevant if you don't use RDMA. It's simply pointing out that it tried to register and it couldn't, which would be expected if your system doesn't support it. On 03/23/2015 12:29 AM, Mohammed Rafi K C wrote:> > On 03/23/2015 11:28 AM, Jonathan Heese wrote: >> On Mar 23, 2015, at 1:20 AM, "Mohammed Rafi K C" <rkavunga at redhat.com >> <mailto:rkavunga at redhat.com>> wrote: >> >>> >>> On 03/21/2015 07:49 PM, Jonathan Heese wrote: >>>> >>>> Mohamed, >>>> >>>> >>>> I have completed the steps you suggested (unmount all, stop the >>>> volume, set the config.transport to tcp, start the volume, mount, >>>> etc.), and the behavior has indeed changed. >>>> >>>> >>>> [root at duke ~]# gluster volume info >>>> >>>> Volume Name: gluster_disk >>>> Type: Replicate >>>> Volume ID: 2307a5a8-641e-44f4-8eaf-7cc2b704aafd >>>> Status: Started >>>> Number of Bricks: 1 x 2 = 2 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: duke-ib:/bricks/brick1 >>>> Brick2: duchess-ib:/bricks/brick1 >>>> Options Reconfigured: >>>> config.transport: tcp >>>> >>>> >>>> [root at duke ~]# gluster volume status >>>> Status of volume: gluster_disk >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick duke-ib:/bricks/brick1 49152 Y 16362 >>>> Brick duchess-ib:/bricks/brick1 49152 Y 14155 >>>> NFS Server on localhost 2049 >>>> Y 16374 >>>> Self-heal Daemon on localhost N/A Y 16381 >>>> NFS Server on duchess-ib 2049 >>>> Y 14167 >>>> Self-heal Daemon on duchess-ib N/A Y 14174 >>>> >>>> Task Status of Volume gluster_disk >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> I am no longer seeing the I/O errors during prolonged periods of >>>> write I/O that I was seeing when the transport was set to rdma. >>>> However, I am seeing this message on both nodes every 3 seconds >>>> (almost exactly): >>>> >>>> >>>> ==> /var/log/glusterfs/nfs.log <=>>>> [2015-03-21 14:17:40.379719] W >>>> [rdma.c:1076:gf_rdma_cm_event_handler] 0-gluster_disk-client-1: cma >>>> event RDMA_CM_EVENT_REJECTED, error 8 (me:10.10.10.1:1023 >>>> peer:10.10.10.2:49152) >>>> >>>> >>>> Is this something to worry about? >>>> >>> If you are not using nfs to export the volumes, there is nothing to >>> worry. >> >> I'm using the native glusterfs FUSE component to mount the volume >> locally on both servers -- I assume that you're referring to the >> standard NFS protocol stuff, which I'm not using here. >> >> Incidentally, I would like to keep my logs from filling up with junk >> if possible. Is there something I can do to get rid of these >> (useless?) error messages? > > If i understand correctly, you are getting this enormous log message > from nfs log only, all other logs and everything are fine now, right ? > If that is the case, and you are not at all using nfs for exporting > the volume, as a workaround you can disable nfs for your volume or > cluster. (gluster v set nfs.disable on). This will turnoff your > gluster nfs server, and you will no longer get those log messages. > > >>>> Any idea why there are rdma pieces in play when I've set my >>>> transport to tcp? >>>> >>> >>> there should not be any piece of rdma,if possible, can you paste the >>> volfile for nfs server. You can find the volfile in >>> /var/lib/glusterd/nfs/nfs-server.vol or >>> /usr/local/var/lib/glusterd/nfs/nfs-server.vol >> >> I will get this for you when I can. Thanks. > > If you can make it, that will be great help to understand the problem. > > > Rafi KC > >> >> Regards, >> Jon Heese >> >>> Rafi KC >>>> >>>> The actual I/O appears to be handled properly and I've seen no >>>> further errors in the testing I've done so far. >>>> >>>> >>>> Thanks. >>>> >>>> >>>> Regards, >>>> >>>> Jon Heese >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> *From:* gluster-users-bounces at gluster.org >>>> <gluster-users-bounces at gluster.org> on behalf of Jonathan Heese >>>> <jheese at inetu.net> >>>> *Sent:* Friday, March 20, 2015 7:04 AM >>>> *To:* Mohammed Rafi K C >>>> *Cc:* gluster-users >>>> *Subject:* Re: [Gluster-users] I/O error on replicated volume >>>> Mohammed, >>>> >>>> Thanks very much for the reply. I will try that and report back. >>>> >>>> Regards, >>>> Jon Heese >>>> >>>> On Mar 20, 2015, at 3:26 AM, "Mohammed Rafi K C" >>>> <rkavunga at redhat.com <mailto:rkavunga at redhat.com>> wrote: >>>> >>>>> >>>>> On 03/19/2015 10:16 PM, Jonathan Heese wrote: >>>>>> >>>>>> Hello all, >>>>>> >>>>>> Does anyone else have any further suggestions for troubleshooting >>>>>> this? >>>>>> >>>>>> To sum up: I have a 2 node 2 brick replicated volume, which holds >>>>>> a handful of iSCSI image files which are mounted and served up by >>>>>> tgtd (CentOS 6) to a handful of devices on a dedicated iSCSI >>>>>> network. The most important iSCSI clients (initiators) are four >>>>>> VMware ESXi 5.5 hosts that use the iSCSI volumes as backing for >>>>>> their datastores for virtual machine storage. >>>>>> >>>>>> After a few minutes of sustained writing to the volume, I am >>>>>> seeing a massive flood (over 1500 per second at times) of this >>>>>> error in /var/log/glusterfs/mnt-gluster-disk.log: >>>>>> >>>>>> [2015-03-16 02:24:07.582801] W >>>>>> [fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse: 635358: >>>>>> WRITE => -1 (Input/output error) >>>>>> >>>>>> When this happens, the ESXi box fails its write operation and >>>>>> returns an error to the effect of ?Unable to write data to >>>>>> datastore?. I don?t see anything else in the supporting logs to >>>>>> explain the root cause of the i/o errors. >>>>>> >>>>>> Any and all suggestions are appreciated. Thanks. >>>>>> >>>>> >>>>> From the mount logs, i assume that your volume transport type is >>>>> rdma. There are some known issues for rdma in 3.5.3, and the patch >>>>> for to address those issues are already send to upstream [1]. From >>>>> the logs, I'm not sure and it is hard to tell you whether this >>>>> problem is something related to rdma transport or not. To make >>>>> sure that the tcp transport is works well in this scenario, if >>>>> possible can you try to reproduce the same using tcp type volumes. >>>>> You can change the transport type of volume by doing the following >>>>> step ( not recommended in normal use case). >>>>> >>>>> 1) unmount every client >>>>> 2) stop the volume >>>>> 3) run gluster volume set volname config.transport tcp >>>>> 4) start the volume again >>>>> 5) mount the clients >>>>> >>>>> [1] : http://goo.gl/2PTL61 >>>>> >>>>> Regards >>>>> Rafi KC >>>>> >>>>>> /Jon Heese/ >>>>>> /Systems Engineer/ >>>>>> *INetU Managed Hosting* >>>>>> P: 610.266.7441 x 261 >>>>>> F: 610.266.7434 >>>>>> www.inetu.net <https://www.inetu.net/> >>>>>> >>>>>> /** This message contains confidential information, which also >>>>>> may be privileged, and is intended only for the person(s) >>>>>> addressed above. Any unauthorized use, distribution, copying or >>>>>> disclosure of confidential and/or privileged information is >>>>>> strictly prohibited. If you have received this communication in >>>>>> error, please erase all copies of the message and its attachments >>>>>> and notify the sender immediately via reply e-mail. **/ >>>>>> >>>>>> *From:*Jonathan Heese >>>>>> *Sent:* Tuesday, March 17, 2015 12:36 PM >>>>>> *To:* 'Ravishankar N'; gluster-users at gluster.org >>>>>> *Subject:* RE: [Gluster-users] I/O error on replicated volume >>>>>> >>>>>> Ravi, >>>>>> >>>>>> The last lines in the mount log before the massive vomit of I/O >>>>>> errors are from 22 minutes prior, and seem innocuous to me: >>>>>> >>>>>> [2015-03-16 01:37:07.126340] E >>>>>> [client-handshake.c:1760:client_query_portmap_cbk] >>>>>> 0-gluster_disk-client-0: failed to get the port number for remote >>>>>> subvolume. Please run 'gluster volume status' on server to see if >>>>>> brick process is running. >>>>>> >>>>>> [2015-03-16 01:37:07.126587] W [rdma.c:4273:gf_rdma_disconnect] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) >>>>>> [0x7fd9c557bccf] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) >>>>>> [0x7fd9c557a995] >>>>>> (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) >>>>>> [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0: disconnect called >>>>>> (peer:10.10.10.1:24008) >>>>>> >>>>>> [2015-03-16 01:37:07.126687] E >>>>>> [client-handshake.c:1760:client_query_portmap_cbk] >>>>>> 0-gluster_disk-client-1: failed to get the port number for remote >>>>>> subvolume. Please run 'gluster volume status' on server to see if >>>>>> brick process is running. >>>>>> >>>>>> [2015-03-16 01:37:07.126737] W [rdma.c:4273:gf_rdma_disconnect] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) >>>>>> [0x7fd9c557bccf] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) >>>>>> [0x7fd9c557a995] >>>>>> (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) >>>>>> [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1: disconnect called >>>>>> (peer:10.10.10.2:24008) >>>>>> >>>>>> [2015-03-16 01:37:10.730165] I >>>>>> [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gluster_disk-client-0: >>>>>> changing port to 49152 (from 0) >>>>>> >>>>>> [2015-03-16 01:37:10.730276] W [rdma.c:4273:gf_rdma_disconnect] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) >>>>>> [0x7fd9c557bccf] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) >>>>>> [0x7fd9c557a995] >>>>>> (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) >>>>>> [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0: disconnect called >>>>>> (peer:10.10.10.1:24008) >>>>>> >>>>>> [2015-03-16 01:37:10.739500] I >>>>>> [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gluster_disk-client-1: >>>>>> changing port to 49152 (from 0) >>>>>> >>>>>> [2015-03-16 01:37:10.739560] W [rdma.c:4273:gf_rdma_disconnect] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) >>>>>> [0x7fd9c557bccf] >>>>>> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) >>>>>> [0x7fd9c557a995] >>>>>> (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) >>>>>> [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1: disconnect called >>>>>> (peer:10.10.10.2:24008) >>>>>> >>>>>> [2015-03-16 01:37:10.741883] I >>>>>> [client-handshake.c:1677:select_server_supported_programs] >>>>>> 0-gluster_disk-client-0: Using Program GlusterFS 3.3, Num >>>>>> (1298437), Version (330) >>>>>> >>>>>> [2015-03-16 01:37:10.744524] I >>>>>> [client-handshake.c:1462:client_setvolume_cbk] >>>>>> 0-gluster_disk-client-0: Connected to 10.10.10.1:49152, attached >>>>>> to remote volume '/bricks/brick1'. >>>>>> >>>>>> [2015-03-16 01:37:10.744537] I >>>>>> [client-handshake.c:1474:client_setvolume_cbk] >>>>>> 0-gluster_disk-client-0: Server and Client lk-version numbers are >>>>>> not same, reopening the fds >>>>>> >>>>>> [2015-03-16 01:37:10.744566] I [afr-common.c:4267:afr_notify] >>>>>> 0-gluster_disk-replicate-0: Subvolume 'gluster_disk-client-0' >>>>>> came back up; going online. >>>>>> >>>>>> [2015-03-16 01:37:10.744627] I >>>>>> [client-handshake.c:450:client_set_lk_version_cbk] >>>>>> 0-gluster_disk-client-0: Server lk version = 1 >>>>>> >>>>>> [2015-03-16 01:37:10.753037] I >>>>>> [client-handshake.c:1677:select_server_supported_programs] >>>>>> 0-gluster_disk-client-1: Using Program GlusterFS 3.3, Num >>>>>> (1298437), Version (330) >>>>>> >>>>>> [2015-03-16 01:37:10.755657] I >>>>>> [client-handshake.c:1462:client_setvolume_cbk] >>>>>> 0-gluster_disk-client-1: Connected to 10.10.10.2:49152, attached >>>>>> to remote volume '/bricks/brick1'. >>>>>> >>>>>> [2015-03-16 01:37:10.755676] I >>>>>> [client-handshake.c:1474:client_setvolume_cbk] >>>>>> 0-gluster_disk-client-1: Server and Client lk-version numbers are >>>>>> not same, reopening the fds >>>>>> >>>>>> [2015-03-16 01:37:10.761945] I >>>>>> [fuse-bridge.c:5016:fuse_graph_setup] 0-fuse: switched to graph 0 >>>>>> >>>>>> [2015-03-16 01:37:10.762144] I >>>>>> [client-handshake.c:450:client_set_lk_version_cbk] >>>>>> 0-gluster_disk-client-1: Server lk version = 1 >>>>>> >>>>>> [*2015-03-16 01:37:10.762279*] I [fuse-bridge.c:3953:fuse_init] >>>>>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs >>>>>> 7.22 kernel 7.14 >>>>>> >>>>>> [*2015-03-16 01:59:26.098670*] W >>>>>> [fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse: 292084: >>>>>> WRITE => -1 (Input/output error) >>>>>> >>>>>> ? >>>>>> >>>>>> I?ve seen no indication of split-brain on any files at any point >>>>>> in this (ever since downdating from 3.6.2 to 3.5.3, which is when >>>>>> this particular issue started): >>>>>> >>>>>> [root at duke gfapi-module-for-linux-target-driver-]# gluster v heal >>>>>> gluster_disk info >>>>>> >>>>>> Brick duke.jonheese.local:/bricks/brick1/ >>>>>> >>>>>> Number of entries: 0 >>>>>> >>>>>> Brick duchess.jonheese.local:/bricks/brick1/ >>>>>> >>>>>> Number of entries: 0 >>>>>> >>>>>> Thanks. >>>>>> >>>>>> /Jon Heese/ >>>>>> /Systems Engineer/ >>>>>> *INetU Managed Hosting* >>>>>> P: 610.266.7441 x 261 >>>>>> F: 610.266.7434 >>>>>> www.inetu.net <https://www.inetu.net/> >>>>>> >>>>>> /** This message contains confidential information, which also >>>>>> may be privileged, and is intended only for the person(s) >>>>>> addressed above. Any unauthorized use, distribution, copying or >>>>>> disclosure of confidential and/or privileged information is >>>>>> strictly prohibited. If you have received this communication in >>>>>> error, please erase all copies of the message and its attachments >>>>>> and notify the sender immediately via reply e-mail. **/ >>>>>> >>>>>> *From:*Ravishankar N [mailto:ravishankar at redhat.com] >>>>>> *Sent:* Tuesday, March 17, 2015 12:35 AM >>>>>> *To:* Jonathan Heese; gluster-users at gluster.org >>>>>> <mailto:gluster-users at gluster.org> >>>>>> *Subject:* Re: [Gluster-users] I/O error on replicated volume >>>>>> >>>>>> On 03/17/2015 02:14 AM, Jonathan Heese wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> So I resolved my previous issue with split-brains and the >>>>>> lack of self-healing by dropping my installed glusterfs* >>>>>> packages from 3.6.2 to 3.5.3, but now I've picked up a new >>>>>> issue, which actually makes normal use of the volume >>>>>> practically impossible. >>>>>> >>>>>> A little background for those not already paying close attention: >>>>>> I have a 2 node 2 brick replicating volume whose purpose in >>>>>> life is to hold iSCSI target files, primarily for use to >>>>>> provide datastores to a VMware ESXi cluster. The plan is to >>>>>> put a handful of image files on the Gluster volume, mount >>>>>> them locally on both Gluster nodes, and run tgtd on both, >>>>>> pointed to the image files on the mounted gluster volume. >>>>>> Then the ESXi boxes will use multipath (active/passive) iSCSI >>>>>> to connect to the nodes, with automatic failover in case of >>>>>> planned or unplanned downtime of the Gluster nodes. >>>>>> >>>>>> In my most recent round of testing with 3.5.3, I'm seeing a >>>>>> massive failure to write data to the volume after about 5-10 >>>>>> minutes, so I've simplified the scenario a bit (to minimize >>>>>> the variables) to: both Gluster nodes up, only one node >>>>>> (duke) mounted and running tgtd, and just regular (single >>>>>> path) iSCSI from a single ESXi server. >>>>>> >>>>>> About 5-10 minutes into migration a VM onto the test >>>>>> datastore, /var/log/messages on duke gets blasted with a ton >>>>>> of messages exactly like this: >>>>>> >>>>>> Mar 15 22:24:06 duke tgtd: bs_rdwr_request(180) io error >>>>>> 0x1781e00 2a -1 512 22971904, Input/output error >>>>>> >>>>>> And /var/log/glusterfs/mnt-gluster_disk.log gets blased with >>>>>> a ton of messages exactly like this: >>>>>> >>>>>> [2015-03-16 02:24:07.572279] W >>>>>> [fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse: >>>>>> 635299: WRITE => -1 (Input/output error) >>>>>> >>>>>> >>>>>> Are there any messages in the mount log from AFR about >>>>>> split-brain just before the above line appears? >>>>>> Does `gluster v heal <VOLNAME> info` show any files? Performing >>>>>> I/O on files that are in split-brain fail with EIO. >>>>>> >>>>>> -Ravi >>>>>> >>>>>> And the write operation from VMware's side fails as soon as >>>>>> these messages start. >>>>>> >>>>>> I don't see any other errors (in the log files I know of) >>>>>> indicating the root cause of these i/o errors. I'm sure that >>>>>> this is not enough information to tell what's going on, but >>>>>> can anyone help me figure out what to look at next to figure >>>>>> this out? >>>>>> >>>>>> I've also considered using Dan Lambright's libgfapi gluster >>>>>> module for tgtd (or something similar) to avoid going through >>>>>> FUSE, but I'm not sure whether that would be irrelevant to >>>>>> this problem, since I'm not 100% sure if it lies in FUSE or >>>>>> elsewhere. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> /Jon Heese/ >>>>>> /Systems Engineer/ >>>>>> *INetU Managed Hosting* >>>>>> P: 610.266.7441 x 261 >>>>>> F: 610.266.7434 >>>>>> www.inetu.net <https://www.inetu.net/> >>>>>> >>>>>> /** This message contains confidential information, which >>>>>> also may be privileged, and is intended only for the >>>>>> person(s) addressed above. Any unauthorized use, >>>>>> distribution, copying or disclosure of confidential and/or >>>>>> privileged information is strictly prohibited. If you have >>>>>> received this communication in error, please erase all copies >>>>>> of the message and its attachments and notify the sender >>>>>> immediately via reply e-mail. **/ >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>>> Gluster-users mailing list >>>>>> >>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>>>> >>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>> >>> > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150326/96e12a65/attachment.html>
Joe, Hmmm.... But every 3 seconds for all eternity? Seems a bit much for a "warning", doesn't it? Did you see my last reply? My nfs-server.vol file seems to indicate that RDMA is still in use in some capacity... Is this normal? If not, how can I reconcile this? Thanks. Regards, Jon Heese ________________________________ From: gluster-users-bounces at gluster.org <gluster-users-bounces at gluster.org> on behalf of Joe Julian <joe at julianfamily.org> Sent: Thursday, March 26, 2015 4:08 PM To: gluster-users at gluster.org Subject: Re: [Gluster-users] I/O error on replicated volume The RDMA warnings are not relevant if you don't use RDMA. It's simply pointing out that it tried to register and it couldn't, which would be expected if your system doesn't support it. On 03/23/2015 12:29 AM, Mohammed Rafi K C wrote: On 03/23/2015 11:28 AM, Jonathan Heese wrote: On Mar 23, 2015, at 1:20 AM, "Mohammed Rafi K C" <rkavunga at redhat.com<mailto:rkavunga at redhat.com>> wrote: On 03/21/2015 07:49 PM, Jonathan Heese wrote: Mohamed, I have completed the steps you suggested (unmount all, stop the volume, set the config.transport to tcp, start the volume, mount, etc.), and the behavior has indeed changed. [root at duke ~]# gluster volume info Volume Name: gluster_disk Type: Replicate Volume ID: 2307a5a8-641e-44f4-8eaf-7cc2b704aafd Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: duke-ib:/bricks/brick1 Brick2: duchess-ib:/bricks/brick1 Options Reconfigured: config.transport: tcp [root at duke ~]# gluster volume status Status of volume: gluster_disk Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick duke-ib:/bricks/brick1 49152 Y 16362 Brick duchess-ib:/bricks/brick1 49152 Y 14155 NFS Server on localhost 2049 Y 16374 Self-heal Daemon on localhost N/A Y 16381 NFS Server on duchess-ib 2049 Y 14167 Self-heal Daemon on duchess-ib N/A Y 14174 Task Status of Volume gluster_disk ------------------------------------------------------------------------------ There are no active volume tasks I am no longer seeing the I/O errors during prolonged periods of write I/O that I was seeing when the transport was set to rdma. However, I am seeing this message on both nodes every 3 seconds (almost exactly): ==> /var/log/glusterfs/nfs.log <=[2015-03-21 14:17:40.379719] W [rdma.c:1076:gf_rdma_cm_event_handler] 0-gluster_disk-client-1: cma event RDMA_CM_EVENT_REJECTED, error 8 (me:10.10.10.1:1023 peer:10.10.10.2:49152) Is this something to worry about? If you are not using nfs to export the volumes, there is nothing to worry. I'm using the native glusterfs FUSE component to mount the volume locally on both servers -- I assume that you're referring to the standard NFS protocol stuff, which I'm not using here. Incidentally, I would like to keep my logs from filling up with junk if possible. Is there something I can do to get rid of these (useless?) error messages? If i understand correctly, you are getting this enormous log message from nfs log only, all other logs and everything are fine now, right ? If that is the case, and you are not at all using nfs for exporting the volume, as a workaround you can disable nfs for your volume or cluster. (gluster v set nfs.disable on). This will turnoff your gluster nfs server, and you will no longer get those log messages. Any idea why there are rdma pieces in play when I've set my transport to tcp? there should not be any piece of rdma,if possible, can you paste the volfile for nfs server. You can find the volfile in /var/lib/glusterd/nfs/nfs-server.vol or /usr/local/var/lib/glusterd/nfs/nfs-server.vol I will get this for you when I can. Thanks. If you can make it, that will be great help to understand the problem. Rafi KC Regards, Jon Heese Rafi KC The actual I/O appears to be handled properly and I've seen no further errors in the testing I've done so far. Thanks. Regards, Jon Heese ________________________________ From: gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org> <gluster-users-bounces at gluster.org><mailto:gluster-users-bounces at gluster.org> on behalf of Jonathan Heese <jheese at inetu.net><mailto:jheese at inetu.net> Sent: Friday, March 20, 2015 7:04 AM To: Mohammed Rafi K C Cc: gluster-users Subject: Re: [Gluster-users] I/O error on replicated volume Mohammed, Thanks very much for the reply. I will try that and report back. Regards, Jon Heese On Mar 20, 2015, at 3:26 AM, "Mohammed Rafi K C" <rkavunga at redhat.com<mailto:rkavunga at redhat.com>> wrote: On 03/19/2015 10:16 PM, Jonathan Heese wrote: Hello all, Does anyone else have any further suggestions for troubleshooting this? To sum up: I have a 2 node 2 brick replicated volume, which holds a handful of iSCSI image files which are mounted and served up by tgtd (CentOS 6) to a handful of devices on a dedicated iSCSI network. The most important iSCSI clients (initiators) are four VMware ESXi 5.5 hosts that use the iSCSI volumes as backing for their datastores for virtual machine storage. After a few minutes of sustained writing to the volume, I am seeing a massive flood (over 1500 per second at times) of this error in /var/log/glusterfs/mnt-gluster-disk.log: [2015-03-16 02:24:07.582801] W [fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse: 635358: WRITE => -1 (Input/output error) When this happens, the ESXi box fails its write operation and returns an error to the effect of ?Unable to write data to datastore?. I don?t see anything else in the supporting logs to explain the root cause of the i/o errors. Any and all suggestions are appreciated. Thanks.>From the mount logs, i assume that your volume transport type is rdma. There are some known issues for rdma in 3.5.3, and the patch for to address those issues are already send to upstream [1]. From the logs, I'm not sure and it is hard to tell you whether this problem is something related to rdma transport or not. To make sure that the tcp transport is works well in this scenario, if possible can you try to reproduce the same using tcp type volumes. You can change the transport type of volume by doing the following step ( not recommended in normal use case).1) unmount every client 2) stop the volume 3) run gluster volume set volname config.transport tcp 4) start the volume again 5) mount the clients [1] : http://goo.gl/2PTL61 Regards Rafi KC Jon Heese Systems Engineer INetU Managed Hosting P: 610.266.7441 x 261 F: 610.266.7434 www.inetu.net<https://www.inetu.net/> ** This message contains confidential information, which also may be privileged, and is intended only for the person(s) addressed above. Any unauthorized use, distribution, copying or disclosure of confidential and/or privileged information is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify the sender immediately via reply e-mail. ** From: Jonathan Heese Sent: Tuesday, March 17, 2015 12:36 PM To: 'Ravishankar N'; gluster-users at gluster.org<mailto:gluster-users at gluster.org> Subject: RE: [Gluster-users] I/O error on replicated volume Ravi, The last lines in the mount log before the massive vomit of I/O errors are from 22 minutes prior, and seem innocuous to me: [2015-03-16 01:37:07.126340] E [client-handshake.c:1760:client_query_portmap_cbk] 0-gluster_disk-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-03-16 01:37:07.126587] W [rdma.c:4273:gf_rdma_disconnect] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x7fd9c557bccf] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7fd9c557a995] (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0: disconnect called (peer:10.10.10.1:24008) [2015-03-16 01:37:07.126687] E [client-handshake.c:1760:client_query_portmap_cbk] 0-gluster_disk-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-03-16 01:37:07.126737] W [rdma.c:4273:gf_rdma_disconnect] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x7fd9c557bccf] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7fd9c557a995] (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1: disconnect called (peer:10.10.10.2:24008) [2015-03-16 01:37:10.730165] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gluster_disk-client-0: changing port to 49152 (from 0) [2015-03-16 01:37:10.730276] W [rdma.c:4273:gf_rdma_disconnect] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x7fd9c557bccf] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7fd9c557a995] (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-0: disconnect called (peer:10.10.10.1:24008) [2015-03-16 01:37:10.739500] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gluster_disk-client-1: changing port to 49152 (from 0) [2015-03-16 01:37:10.739560] W [rdma.c:4273:gf_rdma_disconnect] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x7fd9c557bccf] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7fd9c557a995] (-->/usr/lib64/glusterfs/3.5.3/xlator/protocol/client.so(client_query_portmap_cbk+0x1ea) [0x7fd9c0d8fb9a]))) 0-gluster_disk-client-1: disconnect called (peer:10.10.10.2:24008) [2015-03-16 01:37:10.741883] I [client-handshake.c:1677:select_server_supported_programs] 0-gluster_disk-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-03-16 01:37:10.744524] I [client-handshake.c:1462:client_setvolume_cbk] 0-gluster_disk-client-0: Connected to 10.10.10.1:49152, attached to remote volume '/bricks/brick1'. [2015-03-16 01:37:10.744537] I [client-handshake.c:1474:client_setvolume_cbk] 0-gluster_disk-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-03-16 01:37:10.744566] I [afr-common.c:4267:afr_notify] 0-gluster_disk-replicate-0: Subvolume 'gluster_disk-client-0' came back up; going online. [2015-03-16 01:37:10.744627] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gluster_disk-client-0: Server lk version = 1 [2015-03-16 01:37:10.753037] I [client-handshake.c:1677:select_server_supported_programs] 0-gluster_disk-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-03-16 01:37:10.755657] I [client-handshake.c:1462:client_setvolume_cbk] 0-gluster_disk-client-1: Connected to 10.10.10.2:49152, attached to remote volume '/bricks/brick1'. [2015-03-16 01:37:10.755676] I [client-handshake.c:1474:client_setvolume_cbk] 0-gluster_disk-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-03-16 01:37:10.761945] I [fuse-bridge.c:5016:fuse_graph_setup] 0-fuse: switched to graph 0 [2015-03-16 01:37:10.762144] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gluster_disk-client-1: Server lk version = 1 [2015-03-16 01:37:10.762279] I [fuse-bridge.c:3953:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.14 [2015-03-16 01:59:26.098670] W [fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse: 292084: WRITE => -1 (Input/output error) ? I?ve seen no indication of split-brain on any files at any point in this (ever since downdating from 3.6.2 to 3.5.3, which is when this particular issue started): [root at duke gfapi-module-for-linux-target-driver-]# gluster v heal gluster_disk info Brick duke.jonheese.local:/bricks/brick1/ Number of entries: 0 Brick duchess.jonheese.local:/bricks/brick1/ Number of entries: 0 Thanks. Jon Heese Systems Engineer INetU Managed Hosting P: 610.266.7441 x 261 F: 610.266.7434 www.inetu.net<https://www.inetu.net/> ** This message contains confidential information, which also may be privileged, and is intended only for the person(s) addressed above. Any unauthorized use, distribution, copying or disclosure of confidential and/or privileged information is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify the sender immediately via reply e-mail. ** From: Ravishankar N [mailto:ravishankar at redhat.com] Sent: Tuesday, March 17, 2015 12:35 AM To: Jonathan Heese; gluster-users at gluster.org<mailto:gluster-users at gluster.org> Subject: Re: [Gluster-users] I/O error on replicated volume On 03/17/2015 02:14 AM, Jonathan Heese wrote: Hello, So I resolved my previous issue with split-brains and the lack of self-healing by dropping my installed glusterfs* packages from 3.6.2 to 3.5.3, but now I've picked up a new issue, which actually makes normal use of the volume practically impossible. A little background for those not already paying close attention: I have a 2 node 2 brick replicating volume whose purpose in life is to hold iSCSI target files, primarily for use to provide datastores to a VMware ESXi cluster. The plan is to put a handful of image files on the Gluster volume, mount them locally on both Gluster nodes, and run tgtd on both, pointed to the image files on the mounted gluster volume. Then the ESXi boxes will use multipath (active/passive) iSCSI to connect to the nodes, with automatic failover in case of planned or unplanned downtime of the Gluster nodes. In my most recent round of testing with 3.5.3, I'm seeing a massive failure to write data to the volume after about 5-10 minutes, so I've simplified the scenario a bit (to minimize the variables) to: both Gluster nodes up, only one node (duke) mounted and running tgtd, and just regular (single path) iSCSI from a single ESXi server. About 5-10 minutes into migration a VM onto the test datastore, /var/log/messages on duke gets blasted with a ton of messages exactly like this: Mar 15 22:24:06 duke tgtd: bs_rdwr_request(180) io error 0x1781e00 2a -1 512 22971904, Input/output error And /var/log/glusterfs/mnt-gluster_disk.log gets blased with a ton of messages exactly like this: [2015-03-16 02:24:07.572279] W [fuse-bridge.c:2242:fuse_writev_cbk] 0-glusterfs-fuse: 635299: WRITE => -1 (Input/output error) Are there any messages in the mount log from AFR about split-brain just before the above line appears? Does `gluster v heal <VOLNAME> info` show any files? Performing I/O on files that are in split-brain fail with EIO. -Ravi And the write operation from VMware's side fails as soon as these messages start. I don't see any other errors (in the log files I know of) indicating the root cause of these i/o errors. I'm sure that this is not enough information to tell what's going on, but can anyone help me figure out what to look at next to figure this out? I've also considered using Dan Lambright's libgfapi gluster module for tgtd (or something similar) to avoid going through FUSE, but I'm not sure whether that would be irrelevant to this problem, since I'm not 100% sure if it lies in FUSE or elsewhere. Thanks! Jon Heese Systems Engineer INetU Managed Hosting P: 610.266.7441 x 261 F: 610.266.7434 www.inetu.net<https://www.inetu.net/> ** This message contains confidential information, which also may be privileged, and is intended only for the person(s) addressed above. Any unauthorized use, distribution, copying or disclosure of confidential and/or privileged information is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify the sender immediately via reply e-mail. ** _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150326/aa9c53c9/attachment.html>