> On Jul 27, 2016, at 4:54 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote: > > > > On Wed, Jul 27, 2016 at 11:26 PM, Gmail <b.s.mikhael at gmail.com <mailto:b.s.mikhael at gmail.com>> wrote: > Hello All, > > I keep getting the following errors accompanied by the bricks in the same replica group going offline when trying to sync files between two volumes located in two different geographic locations using Csync2. > > [2016-07-27 17:07:06.701575] E [MSGID: 113015] [posix.c:1011:posix_opendir] 0-gvol001-posix: opendir failed on /zfsvol/brick02/gvol001/.glusterfs/f2/28/f228f015-2bf4-43a3-bfa6-803f282f13af/man3 [Too many open files] > [2016-07-27 17:07:06.701643] E [MSGID: 115056] [server-rpc-fops.c:690:server_opendir_cbk] 0-gvol001-server: 46179928: OPENDIR <gfid:25c95c97-d5a4-415c-baed-b87b847b2ec1>/perl-5.18.1/cpan/Test-Harness/blib/man3 (10bc989e-395f-44ec-911e-746335f567a0) ==> (Too many open files) [Too many open files] > [2016-07-27 17:07:06.702574] E [MSGID: 113099] [posix-helpers.c:389:_posix_xattr_get_set] 0-gvol001-posix: Opening file /zfsvol/brick02/gvol001/.glusterfs/7f/af/7faf6eff-4036-4603-a974-7e95c8fdd568 failed [Too many open files] > [2016-07-27 17:07:06.709581] E [MSGID: 113099] [posix-helpers.c:389:_posix_xattr_get_set] 0-gvol001-posix: Opening file /zfsvol/brick02/gvol001/.glusterfs/dd/d2/ddd2c86e-c296-463c-9d03-2da596bb9f88 failed [Too many open files] > > 1) May be there is a bug in the brick stack which forgets to close a file. > 2) May be Csync2 doesn't close the files after syncing. > Could you let us know the version of gluster and also statedump of the bricks using https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/ <https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/>I?ve tried to collect the logs as per the documentation provided, but not all of the logs was successfully collected. # gluster volume statedump gvol001 volume statedump: failed: Commit failed on <HOSTNAME>. Please check log file for details.> > > > ?Bishoy > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users <http://www.gluster.org/mailman/listinfo/gluster-users> > > > > -- > Pranith-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160728/fc8290ae/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterdump.glusterd.dump.1469742056 Type: application/octet-stream Size: 16425 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160728/fc8290ae/attachment.obj> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160728/fc8290ae/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterdump.glusterfs.dump.1469742040 Type: application/octet-stream Size: 36510 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160728/fc8290ae/attachment-0001.obj> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160728/fc8290ae/attachment-0002.html>
On Fri, Jul 29, 2016 at 3:50 AM, Gmail <b.s.mikhael at gmail.com> wrote:> > > > On Jul 27, 2016, at 4:54 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> > wrote: > > > > On Wed, Jul 27, 2016 at 11:26 PM, Gmail <b.s.mikhael at gmail.com> wrote: > >> Hello All, >> >> I keep getting the following errors accompanied by the bricks in the same >> replica group going offline when trying to sync files between two volumes >> located in two different geographic locations using Csync2. >> >> [2016-07-27 17:07:06.701575] E [MSGID: 113015] >> [posix.c:1011:posix_opendir] 0-gvol001-posix: opendir failed on >> /zfsvol/brick02/gvol001/.glusterfs/f2/28/f228f015-2bf4-43a3-bfa6-803f282f13af/man3 >> [Too many open files] >> [2016-07-27 17:07:06.701643] E [MSGID: 115056] >> [server-rpc-fops.c:690:server_opendir_cbk] 0-gvol001-server: 46179928: >> OPENDIR >> <gfid:25c95c97-d5a4-415c-baed-b87b847b2ec1>/perl-5.18.1/cpan/Test-Harness/blib/man3 >> (10bc989e-395f-44ec-911e-746335f567a0) ==> (Too many open files) [Too many >> open files] >> [2016-07-27 17:07:06.702574] E [MSGID: 113099] >> [posix-helpers.c:389:_posix_xattr_get_set] 0-gvol001-posix: Opening file >> /zfsvol/brick02/gvol001/.glusterfs/7f/af/7faf6eff-4036-4603-a974-7e95c8fdd568 >> failed [Too many open files] >> [2016-07-27 17:07:06.709581] E [MSGID: 113099] >> [posix-helpers.c:389:_posix_xattr_get_set] 0-gvol001-posix: Opening file >> /zfsvol/brick02/gvol001/.glusterfs/dd/d2/ddd2c86e-c296-463c-9d03-2da596bb9f88 >> failed [Too many open files] >> > > 1) May be there is a bug in the brick stack which forgets to close a file. > 2) May be Csync2 doesn't close the files after syncing. > Could you let us know the version of gluster and also statedump of the > bricks using > https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/ > > I?ve tried to collect the logs as per the documentation provided, but not > all of the logs was successfully collected. > > # gluster volume statedump gvol001 > volume statedump: failed: Commit failed on <HOSTNAME>. Please check log > file for details. >Could you attach the logfiles for us to take a look?> > >> >> *?Bishoy* >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > Pranith > > > >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160729/825af4fa/attachment.html>
I?ve already done that in the previous e-mail. ? Bishoy> On Jul 28, 2016, at 3:20 PM, Gmail <b.s.mikhael at gmail.com> wrote: > > <glusterdump.glusterd.dump.1469742056><glusterdump.glusterfs.dump.1469742040> >> On Jul 27, 2016, at 4:54 PM, Pranith Kumar Karampuri <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote: >> >> >> >> On Wed, Jul 27, 2016 at 11:26 PM, Gmail <b.s.mikhael at gmail.com <mailto:b.s.mikhael at gmail.com>> wrote: >> Hello All, >> >> I keep getting the following errors accompanied by the bricks in the same replica group going offline when trying to sync files between two volumes located in two different geographic locations using Csync2. >> >> [2016-07-27 17:07:06.701575] E [MSGID: 113015] [posix.c:1011:posix_opendir] 0-gvol001-posix: opendir failed on /zfsvol/brick02/gvol001/.glusterfs/f2/28/f228f015-2bf4-43a3-bfa6-803f282f13af/man3 [Too many open files] >> [2016-07-27 17:07:06.701643] E [MSGID: 115056] [server-rpc-fops.c:690:server_opendir_cbk] 0-gvol001-server: 46179928: OPENDIR <gfid:25c95c97-d5a4-415c-baed-b87b847b2ec1>/perl-5.18.1/cpan/Test-Harness/blib/man3 (10bc989e-395f-44ec-911e-746335f567a0) ==> (Too many open files) [Too many open files] >> [2016-07-27 17:07:06.702574] E [MSGID: 113099] [posix-helpers.c:389:_posix_xattr_get_set] 0-gvol001-posix: Opening file /zfsvol/brick02/gvol001/.glusterfs/7f/af/7faf6eff-4036-4603-a974-7e95c8fdd568 failed [Too many open files] >> [2016-07-27 17:07:06.709581] E [MSGID: 113099] [posix-helpers.c:389:_posix_xattr_get_set] 0-gvol001-posix: Opening file /zfsvol/brick02/gvol001/.glusterfs/dd/d2/ddd2c86e-c296-463c-9d03-2da596bb9f88 failed [Too many open files] >> >> 1) May be there is a bug in the brick stack which forgets to close a file. >> 2) May be Csync2 doesn't close the files after syncing. >> Could you let us know the version of gluster and also statedump of the bricks using https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/ <https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/> > I?ve tried to collect the logs as per the documentation provided, but not all of the logs was successfully collected. > > # gluster volume statedump gvol001 > volume statedump: failed: Commit failed on <HOSTNAME>. Please check log file for details. >> >> >> >> ?Bishoy >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> http://www.gluster.org/mailman/listinfo/gluster-users <http://www.gluster.org/mailman/listinfo/gluster-users> >> >> >> >> -- >> Pranith-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160728/f7c40de3/attachment.html>