Hello, We're adding an arbiter node to an existing volume and having an issue. Can anyone help? The root cause error appears to be "00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)", as below. We are running glusterfs 5.6.1. Thanks in advance for any assistance! On existing node gfs1, trying to add new arbiter node gfs3: # gluster volume add-brick gvol0 replica 3 arbiter 1 gfs3:/nodirectwritedata/gluster/gvol0 volume add-brick: failed: Commit failed on gfs3. Please check log file for details. On new node gfs3 in gvol0-add-brick-mount.log: [2019-05-17 01:20:22.689721] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2019-05-17 01:20:22.689778] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0 [2019-05-17 01:20:22.694897] E [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected) [2019-05-17 01:20:22.699770] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected) [2019-05-17 01:20:22.699834] W [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 2: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution failed [2019-05-17 01:20:22.715656] I [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of /tmp/mntQAtu3f [2019-05-17 01:20:22.715865] W [glusterfsd.c:1500:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb223bf6dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560886581e75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560886581ceb] ) 0-: received signum (15), shutting down [2019-05-17 01:20:22.715926] I [fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntQAtu3f'. [2019-05-17 01:20:22.715953] I [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to '/tmp/mntQAtu3f'. Processes running on new node gfs3: # ps -ef | grep gluster root 6832 1 0 20:17 ? 00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO root 15799 1 0 20:17 ? 00:00:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/24c12b09f93eec8e.socket --xlator-option *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 --process-name glustershd root 16856 16735 0 21:21 pts/0 00:00:00 grep --color=auto gluster -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190517/ed2baf66/attachment.html>
On 17/05/19 5:59 AM, David Cunningham wrote:> Hello, > > We're adding an arbiter node to an existing volume and having an > issue. Can anyone help? The root cause error appears to be > "00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected)", as below. >Was your root directory of the replica 2 volume? in metadata or entry split-brain? If yes, you need to resolve it before proceeding with the add-brick. -Ravi> We are running glusterfs 5.6.1. Thanks in advance for any assistance! > > On existing node gfs1, trying to add new arbiter node gfs3: > > # gluster volume add-brick gvol0 replica 3 arbiter 1 > gfs3:/nodirectwritedata/gluster/gvol0 > volume add-brick: failed: Commit failed on gfs3. Please check log file > for details. > > On new node gfs3 in gvol0-add-brick-mount.log: > > [2019-05-17 01:20:22.689721] I [fuse-bridge.c:4267:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 > kernel 7.22 > [2019-05-17 01:20:22.689778] I [fuse-bridge.c:4878:fuse_graph_sync] > 0-fuse: switched to graph 0 > [2019-05-17 01:20:22.694897] E [fuse-bridge.c:4336:fuse_first_lookup] > 0-fuse: first lookup on root failed (Transport endpoint is not connected) > [2019-05-17 01:20:22.699770] W > [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: > 00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected) > [2019-05-17 01:20:22.699834] W > [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 2: > SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) > resolution failed > [2019-05-17 01:20:22.715656] I [fuse-bridge.c:5144:fuse_thread_proc] > 0-fuse: initating unmount of /tmp/mntQAtu3f > [2019-05-17 01:20:22.715865] W [glusterfsd.c:1500:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb223bf6dd5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560886581e75] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560886581ceb] ) 0-: > received signum (15), shutting down > [2019-05-17 01:20:22.715926] I [fuse-bridge.c:5914:fini] 0-fuse: > Unmounting '/tmp/mntQAtu3f'. > [2019-05-17 01:20:22.715953] I [fuse-bridge.c:5919:fini] 0-fuse: > Closing fuse connection to '/tmp/mntQAtu3f'. > > Processes running on new node gfs3: > > # ps -ef | grep gluster > root????? 6832???? 1? 0 20:17 ???????? 00:00:00 /usr/sbin/glusterd -p > /var/run/glusterd.pid --log-level INFO > root???? 15799???? 1? 0 20:17 ???????? 00:00:00 /usr/sbin/glusterfs -s > localhost --volfile-id gluster/glustershd -p > /var/run/gluster/glustershd/glustershd.pid -l > /var/log/glusterfs/glustershd.log -S > /var/run/gluster/24c12b09f93eec8e.socket --xlator-option > *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 > --process-name glustershd > root???? 16856 16735? 0 21:21 pts/0??? 00:00:00 grep --color=auto gluster > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190517/2098d335/attachment.html>
Nithya Balachandran
2019-May-20 04:39 UTC
[Gluster-users] add-brick: failed: Commit failed
On Fri, 17 May 2019 at 06:01, David Cunningham <dcunningham at voisonics.com> wrote:> Hello, > > We're adding an arbiter node to an existing volume and having an issue. > Can anyone help? The root cause error appears to be > "00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected)", as below. > > We are running glusterfs 5.6.1. Thanks in advance for any assistance! > > On existing node gfs1, trying to add new arbiter node gfs3: > > # gluster volume add-brick gvol0 replica 3 arbiter 1 > gfs3:/nodirectwritedata/gluster/gvol0 > volume add-brick: failed: Commit failed on gfs3. Please check log file for > details. >This looks like a glusterd issue. Please check the glusterd logs for more info. Adding the glusterd dev to this thread. Sanju, can you take a look? Regards, Nithya> > On new node gfs3 in gvol0-add-brick-mount.log: > > [2019-05-17 01:20:22.689721] I [fuse-bridge.c:4267:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel > 7.22 > [2019-05-17 01:20:22.689778] I [fuse-bridge.c:4878:fuse_graph_sync] > 0-fuse: switched to graph 0 > [2019-05-17 01:20:22.694897] E [fuse-bridge.c:4336:fuse_first_lookup] > 0-fuse: first lookup on root failed (Transport endpoint is not connected) > [2019-05-17 01:20:22.699770] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] > 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport > endpoint is not connected) > [2019-05-17 01:20:22.699834] W [fuse-bridge.c:3294:fuse_setxattr_resume] > 0-glusterfs-fuse: 2: SETXATTR 00000000-0000-0000-0000-000000000001/1 > (trusted.add-brick) resolution failed > [2019-05-17 01:20:22.715656] I [fuse-bridge.c:5144:fuse_thread_proc] > 0-fuse: initating unmount of /tmp/mntQAtu3f > [2019-05-17 01:20:22.715865] W [glusterfsd.c:1500:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb223bf6dd5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560886581e75] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560886581ceb] ) 0-: > received signum (15), shutting down > [2019-05-17 01:20:22.715926] I [fuse-bridge.c:5914:fini] 0-fuse: > Unmounting '/tmp/mntQAtu3f'. > [2019-05-17 01:20:22.715953] I [fuse-bridge.c:5919:fini] 0-fuse: Closing > fuse connection to '/tmp/mntQAtu3f'. > > Processes running on new node gfs3: > > # ps -ef | grep gluster > root 6832 1 0 20:17 ? 00:00:00 /usr/sbin/glusterd -p > /var/run/glusterd.pid --log-level INFO > root 15799 1 0 20:17 ? 00:00:00 /usr/sbin/glusterfs -s > localhost --volfile-id gluster/glustershd -p > /var/run/gluster/glustershd/glustershd.pid -l > /var/log/glusterfs/glustershd.log -S > /var/run/gluster/24c12b09f93eec8e.socket --xlator-option > *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 --process-name > glustershd > root 16856 16735 0 21:21 pts/0 00:00:00 grep --color=auto gluster > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190520/03fb4d8a/attachment.html>