[Gluster-users] Assertion failed: ec_get_inode_size

Ashish Pandey
2016-04-15T08:53:40Z

I think this is the statesump of only one brick.
We would required statedump from all the bricks and client process in case of
fuse or nfs process if it is mounted through nfs.

----
Ashish

----- Original Message -----

From: "Serkan ?oban" <cobanserkan at gmail.com>
To: "Ashish Pandey" <aspandey at redhat.com>
Cc: "Gluster Users" <gluster-users at gluster.org>,
"Gluster Devel" <gluster-devel at gluster.org>
Sent: Friday, April 15, 2016 2:11:57 PM
Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size

Sorry for typo, brick state dump file.

On Fri, Apr 15, 2016 at 11:41 AM, Serkan ?oban <cobanserkan at gmail.com>
wrote:
> Hi I reproduce the problem, brick log file is in below link:
>
https://www.dropbox.com/s/iy09j7mm2hrsf03/bricks-02.5677.dump.1460705370.gz?dl=0
>
>
> On Thu, Apr 14, 2016 at 8:07 PM, Ashish Pandey <aspandey at
redhat.com> wrote:
>> Hi Serkan,
>>
>> Could you also provide us the statedump of all the brick processes and
>> clients?
>>
>> Commands to generate statedumps for brick processes/nfs server/quotad
>>
>> For bricks: gluster volume statedump <volname>
>>
>> For nfs server: gluster volume statedump <volname> nfs
>>
>>
>> We can find the directory where statedump files are created using
'gluster
>> --print-statedumpdir'
>> Also, the mount logs would help us to debug the issue.
>>
>> Ashish
>>
>> ________________________________
>> From: "Serkan ?oban" <cobanserkan at gmail.com>
>> To: "Gluster Users" <gluster-users at gluster.org>,
"Gluster Devel"
>> <gluster-devel at gluster.org>
>> Sent: Thursday, April 14, 2016 6:27:10 PM
>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>>
>>
>> Here is the related brick log:
>>
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E
>> [inodelk.c:309:__inode_unlock_lock] 0-v0-locks: Matching lock not
>> found for unlock 0-9223372036854775807, by 94d29e885e7f0000 on
>> 0x7f037413b990
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E
>> [MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk]
>> 0-v0-server: 712984: INODELK
>> /workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605
>> (1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument)
[Invalid
>> argument]
>>
>> On Thu, Apr 14, 2016 at 3:25 PM, Serkan ?oban <cobanserkan at
gmail.com> wrote:
>>> Hi,
>>>
>>> During read/write tests to a 78x(16+4) distributed disperse volume
>>> from 50 clients, One clients hangs on read/write with the following
>>> logs:
>>>
>>> [2016-04-14 11:11:04.728580] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching
xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:11:04.728624] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed
on
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=DFFFF,
>>> bad=20000)
>>> [2016-04-14 11:11:04.736689] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:
>>> name heal successful on FFFFF
>>> [2016-04-14 11:29:26.718036] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching
xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:29:26.718121] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed
on
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=EFFFF,
>>> bad=10000)
>>> [2016-04-14 11:29:42.501760] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:
>>> name heal successful on FFFFF
>>> [2016-04-14 11:31:25.714812] E
[ec-inode-read.c:1612:ec_manager_stat]
>>>
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)
>>> [0x7f5ec9f942b1]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)
>>> [0x7f5ec9f94497]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)
>>> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,
>>> fop->locks[0].lock->loc.inode, &cbk->iatt[0].ia_size)
>>> [2016-04-14 11:31:25.722372] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722411] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722450] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722477] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722503] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722577] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722605] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722742] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722794] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722818] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722840] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722883] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722906] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722958] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722983] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.723037] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.723045] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.725044] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.741338] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.746602] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.746629] W [MSGID: 122015]
>>> [ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode
unlocking
>>> failed (FSTAT) [Invalid argument]
>>> [2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze]
>>>
>>>
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)
>>> [0x7f5ec9fa009e]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)
>>> [0x7f5ec9f97fc0]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0x91)
>>> [0x7f5ec9f975d1] ) 0-: Assertion failed: lock->refs ==
lock->inserted
>>> [2016-04-14 11:31:25.746719] E [ec-common.c:1641:ec_lock_unfreeze]
>>>
>>>
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)
>>> [0x7f5ec9fa009e]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)
>>> [0x7f5ec9f97fc0]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0xf2)
>>> [0x7f5ec9f97632] ) 0-: Assertion failed:
list_empty(&lock->waiting) &&
>>> list_empty(&lock->owners)
>>> [2016-04-14 11:40:54.139418] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-0: Mismatching
xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:40:54.139554] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-0: Operation failed
on
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=2FFFF,
>>> bad=D0000)
>>> [2016-04-14 11:40:54.143400] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-0: /workdir/raw_output2:
>>> name heal successful on FFFFF
>>> The message "W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check]
>>> 0-v0-disperse-0: Mismatching xdata in answers of
'LOOKUP'" repeated 2
>>> times between [2016-04-14 11:40:54.139418] and [2016-04-14
>>> 11:40:54.139547]
>>> [2016-04-14 11:42:31.471317] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-45: Mismatching
>>> xdata in answers of 'LOOKUP'
>>> The message "W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check]
>>> 0-v0-disperse-45: Mismatching xdata in answers of
'LOOKUP'" repeated 8
>>> times between [2016-04-14 11:42:31.471317] and [2016-04-14
>>> 11:42:31.471557]
>>>
>>> Gluster version is 3.7.10
>>> Any thoughts?
>>>
>>> Serkan
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160415/ca42ba83/attachment.html>