Norman Mähler
2014-Jul-07 13:28 UTC
[Gluster-users] Self-heal Problems with gluster and nfs
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear community, we have got some serious problems with our Gluster installation. Here is the setting: We have got 2 bricks (version 3.4.4) on a debian 7.5, one of them with an nfs export. There are about 120 clients connecting to the exported nfs. These clients are thin clients reading and writing their Linux home directories from the exported nfs. We want to change the access of these clients one by one to access via gluster client. Here are our problems: In the moment we have got two types of error messages which come in burts to our glusterfshd.log [2014-07-07 13:10:21.572487] W [client-rpc-fops.c:1538:client3_3_inodelk_cbk] 0-gluster_dateisystem-client-1: remote operation failed: No such file or directory [2014-07-07 13:10:21.573448] W [client-rpc-fops.c:471:client3_3_open_cbk] 0-gluster_dateisystem-client-1: remote operation failed: No such file or directory. Path: <gfid:b0c4f78a-249f-4db7-9d5b-0902c7d8f6cc> (00000000-0000-0000-0000-000000000000) [2014-07-07 13:10:21.573468] E [afr-self-heal-data.c:1270:afr_sh_data_open_cbk] 0-gluster_dateisystem-replicate-0: open of <gfid:b0c4f78a-249f-4db7-9d5b-0902c7d8f6cc> failed on child gluster_dateisystem-client-1 (No such file or directory) This looks like a missing gfid file on one of the bricks. I looked it up and yes the file is missing on the second brick. We got these messages the other way round, too (missing on client-0 and the first brick). Is it possible to repair this one by copying the gfid file to the brick where it was missing? Or ist there another way to repair it? The second message is [2014-07-07 13:06:35.948738] W [client-rpc-fops.c:2469:client3_3_link_cbk] 0-gluster_dateisystem-client-1: remote operation failed: File exists (00000000-0000-0000-0000-000000000000 -> <gfid:aae47250-8f69-480c-ac75-2da2f4d21d7a>/lock) and I really do not know what to do with this one... I am really looking forward to your help because this is an active system and the system load on the nfs brick is about 25 (!!) Thanks in advance! Norman Maehler -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTuqCNAAoJEB810LSP8y+R+OAH/jMmwbx4mk+39B6/gTJC8iU3 CwqLR0x0spdfYGww2jaRT+DuwlMRFw7LmFjaaOXEIPkTuZ1rs3PFvQ/Jw7OLNKWw e+JqVUC21N7oBFOz/YNpeFwFD3iKdGamaUTSIXr4khL3pf1WeonTK5J78p53dyh8 XjcxkCDhdiCMBo7T4VRp32fh550PRzp5wEhGdn79NGGwGuzwF1g6lGpQrhA3Y2aQ d7Pequw7lD00b8rOOzgiVtgkiC5YjPBmMPBW7WBTCd0D4UtXgY1ve188ki+C/ASj DvRAV7MHl24WKd3oWEKpbgyhqFg2F1DplaIkK6e863NbCUBJQ+HRvOyGeJv8DPQ=SMkr -----END PGP SIGNATURE-----
Pranith Kumar Karampuri
2014-Jul-07 13:39 UTC
[Gluster-users] Self-heal Problems with gluster and nfs
On 07/07/2014 06:58 PM, Norman M?hler wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Dear community, > > we have got some serious problems with our Gluster installation. > > Here is the setting: > > We have got 2 bricks (version 3.4.4) on a debian 7.5, one of them with > an nfs export. > There are about 120 clients connecting to the exported nfs. > These clients are thin clients reading and writing their Linux home > directories from the exported nfs. > > We want to change the access of these clients one by one to access via > gluster client.I did not understand what you meant by this. Are you moving to glusterfs-fuse based mounts?> > > Here are our problems: > > In the moment we have got two types of error messages which come in > burts to our glusterfshd.log > > [2014-07-07 13:10:21.572487] W > [client-rpc-fops.c:1538:client3_3_inodelk_cbk] > 0-gluster_dateisystem-client-1: remote operation failed: No such file > or directory > [2014-07-07 13:10:21.573448] W > [client-rpc-fops.c:471:client3_3_open_cbk] > 0-gluster_dateisystem-client-1: remote operation failed: No such file > or directory. Path: <gfid:b0c4f78a-249f-4db7-9d5b-0902c7d8f6cc> > (00000000-0000-0000-0000-000000000000) > [2014-07-07 13:10:21.573468] E > [afr-self-heal-data.c:1270:afr_sh_data_open_cbk] > 0-gluster_dateisystem-replicate-0: open of > <gfid:b0c4f78a-249f-4db7-9d5b-0902c7d8f6cc> failed on child > gluster_dateisystem-client-1 (No such file or directory) > > > This looks like a missing gfid file on one of the bricks. > I looked it up and yes the file is missing on the second brick. > > We got these messages the other way round, too (missing on client-0 > and the first brick). > > Is it possible to repair this one by copying the gfid file to the > brick where it was missing? Or ist there another way to repair it? > > > The second message is > > [2014-07-07 13:06:35.948738] W > [client-rpc-fops.c:2469:client3_3_link_cbk] > 0-gluster_dateisystem-client-1: remote operation failed: File exists > (00000000-0000-0000-0000-000000000000 -> > <gfid:aae47250-8f69-480c-ac75-2da2f4d21d7a>/lock) > > and I really do not know what to do with this one...Did any of the bricks went offline and came back online? Pranith> > > I am really looking forward to your help because this is an active > system and the system load on the nfs brick is about 25 (!!) > > Thanks in advance! > Norman Maehler > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJTuqCNAAoJEB810LSP8y+R+OAH/jMmwbx4mk+39B6/gTJC8iU3 > CwqLR0x0spdfYGww2jaRT+DuwlMRFw7LmFjaaOXEIPkTuZ1rs3PFvQ/Jw7OLNKWw > e+JqVUC21N7oBFOz/YNpeFwFD3iKdGamaUTSIXr4khL3pf1WeonTK5J78p53dyh8 > XjcxkCDhdiCMBo7T4VRp32fh550PRzp5wEhGdn79NGGwGuzwF1g6lGpQrhA3Y2aQ > d7Pequw7lD00b8rOOzgiVtgkiC5YjPBmMPBW7WBTCd0D4UtXgY1ve188ki+C/ASj > DvRAV7MHl24WKd3oWEKpbgyhqFg2F1DplaIkK6e863NbCUBJQ+HRvOyGeJv8DPQ> =SMkr > -----END PGP SIGNATURE----- > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users