Alastair Neil
2016-Dec-21 17:38 UTC
[Gluster-users] 3.8.5 replica 3 volumes: I/O error on file on fuse mounts
Would apprecaite any insight into this issue: replica 3 volume, it is showing a number of files on two of the bricks as needing healed, when you examine the files on the fuse mounts they generate I/O errors. No files listed in split brain, but if I look at one of the files it looks to me like they have been updated on gluster-2 and gluster0 but not on gluster1 (see below). I see errors in /va/log/gluster/glustershd.log -Thanks Alastair [2016-12-20 07:25:06.018829] I [MSGID: 101190]> [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2016-12-20 07:25:06.018901] E [socket.c:2309:socket_connect_finish] > 0-glusterfs: connection to ::1:24007 failed (Connection refused) > [2016-12-20 07:25:06.018944] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify] > 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport > endpoint is not connected) > [2016-12-20 07:25:07.187710] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fd93f669dc5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fd940cfbcd5] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fd940cfbb4b] ) 0-: > received signum (15), shutting down > [2016-12-20 07:25:08.197959] I [MSGID: 100030] [glusterfsd.c:2454:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5 > (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p > /var/lib/glusterd/glustershd/run/glustershd.pid -l > /var/log/glusterfs/glustershd.log -S > /var/run/gluster/3fe0b238bd46c38a95636f25cb5b9d8a.socket --xlator-option > *replicate*.node-uuid=bcff5245-ea86-4384-a1bf-9219c8be8001) > [2016-12-20 07:25:08.216336] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2016-12-20 07:25:08.216419] E [socket.c:2309:socket_connect_finish] > 0-glusterfs: connection to ::1:24007 failed (Connection refused) > [2016-12-20 07:25:08.216464] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify] > 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport > endpoint is not connected) > [2016-12-20 07:25:12.208092] I [MSGID: 101173] > [graph.c:269:gf_add_cmdline_options] 0-digitalcorpora-replicate-0: adding > option 'node-uuid' for volume 'digitalcorpora-replicate-0' with value > 'bcff5245-ea86-4384-a1bf-9219c8be8001' > [2016-12-20 07:25:12.208122] I [MSGID: 101173] > [graph.c:269:gf_add_cmdline_options] 0-gluster_shared_storage-replicate-0: > adding option 'node-uuid' for volume 'gluster_shared_storage-replicate-0' > with value 'bcff5245-ea86-4384-a1bf-9219c8be8001' > [2016-12-20 07:25:12.208140] I [MSGID: 101173] > [graph.c:269:gf_add_cmdline_options] 0-homes-replicate-0: adding option > 'node-uuid' for volume 'homes-replicate-0' with value > 'bcff5245-ea86-4384-a1bf-9219c8be8001' > [2016-12-20 07:25:12.208155] I [MSGID: 101173] > [graph.c:269:gf_add_cmdline_options] 0-public-replicate-0: adding option > 'node-uuid' for volume 'public-replicate-0' with value > 'bcff5245-ea86-4384-a1bf-9219c8be8001' > [2016-12-20 07:25:12.208173] I [MSGID: 101173] > [graph.c:269:gf_add_cmdline_options] 0-static-web-replicate-0: adding > option 'node-uuid' for volume 'static-web-replicate-0' with value > 'bcff5245-ea86-4384-a1bf-9219c8be8001' > [2016-12-20 07:25:12.208199] I [MSGID: 101173] > [graph.c:269:gf_add_cmdline_options] 0-tmp-replicate-0: adding option > 'node-uuid' for volume 'tmp-replicate-0' with value > 'bcff5245-ea86-4384-a1bf-9219c8be8001' > [2016-12-20 07:25:12.208215] I [MSGID: 101173] > [graph.c:269:gf_add_cmdline_options] 0-usr-local-replicate-0: adding option > 'node-uuid' for volume 'usr-local-replicate-0' with value > 'bcff5245-ea86-4384-a1bf-9219c8be8001' > [2016-12-20 18:32:06.121734] E [client-common.c:526:client_pre_getxattr] > (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) > [0x7f6bc4ba65d8] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) > [0x7f6bc4bc1ebd] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) > [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0 > [2016-12-20 18:32:06.121809] E [client-common.c:587:client_pre_opendir] > (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) > [0x7f6bc4ba59d5] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) > [0x7f6bc4bc0a65] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) > [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0 > [2016-12-20 18:46:51.764776] E [client-common.c:526:client_pre_getxattr] > (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) > [0x7f6bc4ba65d8] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) > [0x7f6bc4bc1ebd] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) > [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0 > [2016-12-20 18:46:51.764850] E [client-common.c:587:client_pre_opendir] > (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) > [0x7f6bc4ba59d5] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) > [0x7f6bc4bc0a65] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) > [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0 > [2016-12-20 18:49:29.657568] E [client-common.c:526:client_pre_getxattr] > (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) > [0x7f6bc4ba65d8] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) > [0x7f6bc4bc1ebd] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) > [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0 > [2016-12-20 18:49:29.657645] E [client-common.c:587:client_pre_opendir] > (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) > [0x7f6bc4ba59d5] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) > [0x7f6bc4bc0a65] > -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) > [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0 >gluster2: # getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority getfattr: Removing leading '/' from absolute path names # file: export/brick2/home/a/j/ajn/.Xauthority trusted.afr.dirty=0x000000000000000000000000 trusted.afr.homes-client-5=0x000000020000000100000000 trusted.bit-rot.version=0x020000000000000058589e6b0005bdac trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5 gluster1: # getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority getfattr: Removing leading '/' from absolute path names # file: export/brick2/home/a/j/ajn/.Xauthority trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x0200000000000000583f45c20008d152 trusted.gfid=0x6c278b5c94ae436bb669b5f5dd21777e gluster0: # getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority getfattr: Removing leading '/' from absolute path names # file: export/brick2/home/a/j/ajn/.Xauthority trusted.afr.dirty=0x000000000000000000000000 trusted.afr.homes-client-5=0x000000020000000100000000 trusted.bit-rot.version=0x0200000000000000583f3fbb000b5b01 trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5 [root at gluster0 Project3]# glv heal homes info> Brick gluster-2:/export/brick2/home > /s/a/sadams25/pp2.txt > /s/a/sadams25/.viminfo > /a/v/avakil/.Xauthority > /j/m/jmurra17/fork > /c/f/cferris2/.viminfo > /c/s/cs367/bomblab/S001/log-status.txt > /c/s/cs367/bomblab/S001/bomblab-scoreboard.html > /c/s/cs367/bomblab/S001/scores.txt > /c/s/cs367/bomblab/S003/bomblab-scoreboard.html > /c/s/cs367/bomblab/S003/scores.txt > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o > > /j/m/jmurra17/fork/fork.c > /j/m/jmurra17/.viminfo > /a/j/ajn/.Xauthority > /a/v/avakil/source_code/rm_setup/common_setup.tcl > /a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl > /a/v/avakil/source_code/rm_setup/dc_setup.tcl > /j/d/jdenton3/.viminfo > /s/a/sadams25/x.txt > /j/d/jdenton3/Project3/Project3.c > /j/m/jmurra17/fork/fork > /j/d/jdenton3/Project3/p5 > Status: Connected > Number of entries: 27 > > Brick gluster1.vsnet.gmu.edu:/export/brick2/home > Status: Connected > Number of entries: 0 > > Brick gluster0:/export/brick2/home > /s/a/sadams25/pp2.txt > /s/a/sadams25/.viminfo > /c/s/cs367/bomblab/S003/scores.txt > /a/v/avakil/.Xauthority > /c/s/cs367/bomblab/S001/scores.txt > /c/f/cferris2/.viminfo > /c/s/cs367/bomblab/S001/log-status.txt > /c/s/cs367/bomblab/S003/tmpwebpage.14635 > /c/s/cs367/bomblab/S001/bomblab-scoreboard.html > /c/s/cs367/bomblab/S003/bomblab-scoreboard.html > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c > > /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o > > /j/m/jmurra17/fork > <gfid:310211c2-aeec-4906-894f-023d0ad7d5cc>/# > affiliate.nagios.com/settings.sol > /a/v/avakil/source_code/rm_setup/common_setup.tcl > /a/j/ajn/.Xauthority > /j/m/jmurra17/.viminfo > /a/v/avakil/source_code/rm_setup/dc_setup.tcl > /j/m/jmurra17/fork/fork.c > /a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl > /j/d/jdenton3/Project3/Project3.c > /j/d/jdenton3/.viminfo > /s/a/sadams25/x.txt > /j/m/jmurra17/fork/fork > /j/d/jdenton3/Project3/p5 > Status: Connected > Number of entries: 29 > > [ > [root at gluster0 .bad]# cd > /mnt/home/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/ > [root at gluster0 mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh]# ls > -al > ls: cannot access libsupport.a: Input/output error > ls: cannot access Makefile: Input/output error > ls: cannot access memory_system.c: Input/output error > ls: cannot access memory_system.h: Input/output error > ls: cannot access caching.c: Input/output error > ls: cannot access memory_system.o: Input/output error > total 626 > drwxrwxr-x 2 1735 users 4096 Dec 20 11:38 . > drwxr-xr-x 3 root root 4096 Dec 20 13:53 .. > -????????? ? ? ? ? ? caching.c > -rw-rw-r-- 1 1735 users 9056 Dec 20 11:36 caching.o > -rwxrwxr-x 1 1735 users 147855 Dec 20 11:36 lab4 > -rw-r--r-- 1 1735 users 307200 Dec 13 07:04 Lab 4 - 12 > 9_mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh.tar > -rw-rw-r-- 1 1735 users 8254 Dec 20 11:38 lab4_logfile > -rw-r--r-- 1 1735 users 153600 Dec 20 11:32 lab4_mchehreh.tar > -????????? ? ? ? ? ? libsupport.a > -????????? ? ? ? ? ? Makefile > -????????? ? ? ? ? ? memory_system.c > -????????? ? ? ? ? ? memory_system.h > -????????? ? ? ? ? ? memory_system.o > -rw-rw-r-- 1 1735 users 449 Dec 20 11:38 t1 > -rw-rw-r-- 1 1735 users 453 Dec 20 11:38 t2 > -rw-rw-r-- 1 1735 users 2185 Dec 20 11:38 t3 > -rw-rw-r-- 1 1735 users 2195 Dec 20 11:38 t4 >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161221/1451bc66/attachment.html>