Anirban Ghoshal
2014-Oct-17 22:50 UTC
[Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors
Hi everyone, I have this really confusing split-brain here that's bothering me. I am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2 volume 'testvol' that is It seems I cannot read/stat/edit the file in question, and `gluster volume heal testvol info split-brain` shows nothing. Here are the logs from the fuse-mount for the volume: [2014-09-29 07:53:02.867111] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 4560969: FLUSH() ERR => -1 (Input/output error) [2014-09-29 07:54:16.007799] W [page.c:991:__ioc_page_error] 0-testvol-io-cache: page error for page = 0x7fd5c8529d20 & waitq = 0x7fd5c8067d40 [2014-09-29 07:54:16.007854] W [fuse-bridge.c:2089:fuse_readv_cbk] 0-glusterfs-fuse: 4561103: READ => -1 (Input/output error) [2014-09-29 07:54:16.008018] W [page.c:991:__ioc_page_error] 0-testvol-io-cache: page error for page = 0x7fd5c8607ee0 & waitq = 0x7fd5c8067d40 [2014-09-29 07:54:16.008056] W [fuse-bridge.c:2089:fuse_readv_cbk] 0-glusterfs-fuse: 4561104: READ => -1 (Input/output error) [2014-09-29 07:54:16.008233] W [page.c:991:__ioc_page_error] 0-testvol-io-cache: page error for page = 0x7fd5c8066f30 & waitq = 0x7fd5c8067d40 [2014-09-29 07:54:16.008269] W [fuse-bridge.c:2089:fuse_readv_cbk] 0-glusterfs-fuse: 4561105: READ => -1 (Input/output error) [2014-09-29 07:54:16.008800] W [page.c:991:__ioc_page_error] 0-testvol-io-cache: page error for page = 0x7fd5c860bcf0 & waitq = 0x7fd5c863b1f0 [2014-09-29 07:54:16.008839] W [fuse-bridge.c:2089:fuse_readv_cbk] 0-glusterfs-fuse: 4561107: READ => -1 (Input/output error) [2014-09-29 07:54:16.009365] W [page.c:991:__ioc_page_error] 0-testvol-io-cache: page error for page = 0x7fd5c85fd120 & waitq = 0x7fd5c8067d40 [2014-09-29 07:54:16.009413] W [fuse-bridge.c:2089:fuse_readv_cbk] 0-glusterfs-fuse: 4561109: READ => -1 (Input/output error) [2014-09-29 07:54:16.040549] W [afr-open.c:213:afr_open] 0-testvol-replicate-0: failed to open as split brain seen, returning EIO [2014-09-29 07:54:16.040594] W [fuse-bridge.c:915:fuse_fd_cbk] 0-glusterfs-fuse: 4561142: OPEN() /SECLOG/20140908.d/SECLOG_00000000000000427425_00000000000000000000.log => -1 (Input/output error) Could somebody please give me some clue on where to begin? I checked the xattrs on /SECLOG/20140908.d/SECLOG_00000000000000427425_00000000000000000000.log and it seems the changelogs are [0, 0] on both replicas, and the gfid's match. Thank you very much for any help on this. Anirban -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141018/d658b97e/attachment.html>
Pranith Kumar Karampuri
2014-Oct-18 00:26 UTC
[Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors
hi,
Could you see if the size of the file mismatches?
Pranith
On 10/18/2014 04:20 AM, Anirban Ghoshal wrote:> Hi everyone,
>
> I have this really confusing split-brain here that's bothering me. I
> am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2
> volume 'testvol' that is It seems I cannot read/stat/edit the file
in
> question, and `gluster volume heal testvol info split-brain` shows
> nothing. Here are the logs from the fuse-mount for the volume:
>
> [2014-09-29 07:53:02.867111] W [fuse-bridge.c:1172:fuse_err_cbk]
> 0-glusterfs-fuse: 4560969: FLUSH() ERR => -1 (Input/output error)
> [2014-09-29 07:54:16.007799] W [page.c:991:__ioc_page_error]
> 0-testvol-io-cache: page error for page = 0x7fd5c8529d20 & waitq =
> 0x7fd5c8067d40
> [2014-09-29 07:54:16.007854] W [fuse-bridge.c:2089:fuse_readv_cbk]
> 0-glusterfs-fuse: 4561103: READ => -1 (Input/output error)
> [2014-09-29 07:54:16.008018] W [page.c:991:__ioc_page_error]
> 0-testvol-io-cache: page error for page = 0x7fd5c8607ee0 & waitq =
> 0x7fd5c8067d40
> [2014-09-29 07:54:16.008056] W [fuse-bridge.c:2089:fuse_readv_cbk]
> 0-glusterfs-fuse: 4561104: READ => -1 (Input/output error)
> [2014-09-29 07:54:16.008233] W [page.c:991:__ioc_page_error]
> 0-testvol-io-cache: page error for page = 0x7fd5c8066f30 & waitq =
> 0x7fd5c8067d40
> [2014-09-29 07:54:16.008269] W [fuse-bridge.c:2089:fuse_readv_cbk]
> 0-glusterfs-fuse: 4561105: READ => -1 (Input/output error)
> [2014-09-29 07:54:16.008800] W [page.c:991:__ioc_page_error]
> 0-testvol-io-cache: page error for page = 0x7fd5c860bcf0 & waitq =
> 0x7fd5c863b1f0
> [2014-09-29 07:54:16.008839] W [fuse-bridge.c:2089:fuse_readv_cbk]
> 0-glusterfs-fuse: 4561107: READ => -1 (Input/output error)
> [2014-09-29 07:54:16.009365] W [page.c:991:__ioc_page_error]
> 0-testvol-io-cache: page error for page = 0x7fd5c85fd120 & waitq =
> 0x7fd5c8067d40
> [2014-09-29 07:54:16.009413] W [fuse-bridge.c:2089:fuse_readv_cbk]
> 0-glusterfs-fuse: 4561109: READ => -1 (Input/output error)
> [2014-09-29 07:54:16.040549] W [afr-open.c:213:afr_open]
> 0-testvol-replicate-0: failed to open as split brain seen, returning EIO
> [2014-09-29 07:54:16.040594] W [fuse-bridge.c:915:fuse_fd_cbk]
> 0-glusterfs-fuse: 4561142: OPEN()
> /SECLOG/20140908.d/SECLOG_00000000000000427425_00000000000000000000.log
=>
> -1 (Input/output error)
>
> Could somebody please give me some clue on where to begin? I checked
> the xattrs on
> /SECLOG/20140908.d/SECLOG_00000000000000427425_00000000000000000000.log and
> it seems the changelogs are [0, 0] on both replicas, and the gfid's
match.
>
> Thank you very much for any help on this.
> Anirban
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141018/4fd01da9/attachment.html>