Sorry for typo, brick state dump file.
On Fri, Apr 15, 2016 at 11:41 AM, Serkan ?oban <cobanserkan at gmail.com>
wrote:> Hi I reproduce the problem, brick log file is in below link:
>
https://www.dropbox.com/s/iy09j7mm2hrsf03/bricks-02.5677.dump.1460705370.gz?dl=0
>
>
> On Thu, Apr 14, 2016 at 8:07 PM, Ashish Pandey <aspandey at
redhat.com> wrote:
>> Hi Serkan,
>>
>> Could you also provide us the statedump of all the brick processes and
>> clients?
>>
>> Commands to generate statedumps for brick processes/nfs server/quotad
>>
>> For bricks: gluster volume statedump <volname>
>>
>> For nfs server: gluster volume statedump <volname> nfs
>>
>>
>> We can find the directory where statedump files are created using
'gluster
>> --print-statedumpdir'
>> Also, the mount logs would help us to debug the issue.
>>
>> Ashish
>>
>> ________________________________
>> From: "Serkan ?oban" <cobanserkan at gmail.com>
>> To: "Gluster Users" <gluster-users at gluster.org>,
"Gluster Devel"
>> <gluster-devel at gluster.org>
>> Sent: Thursday, April 14, 2016 6:27:10 PM
>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>>
>>
>> Here is the related brick log:
>>
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E
>> [inodelk.c:309:__inode_unlock_lock] 0-v0-locks: Matching lock not
>> found for unlock 0-9223372036854775807, by 94d29e885e7f0000 on
>> 0x7f037413b990
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E
>> [MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk]
>> 0-v0-server: 712984: INODELK
>> /workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605
>> (1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument)
[Invalid
>> argument]
>>
>> On Thu, Apr 14, 2016 at 3:25 PM, Serkan ?oban <cobanserkan at
gmail.com> wrote:
>>> Hi,
>>>
>>> During read/write tests to a 78x(16+4) distributed disperse volume
>>> from 50 clients, One clients hangs on read/write with the following
>>> logs:
>>>
>>> [2016-04-14 11:11:04.728580] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching
xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:11:04.728624] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed
on
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=DFFFF,
>>> bad=20000)
>>> [2016-04-14 11:11:04.736689] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:
>>> name heal successful on FFFFF
>>> [2016-04-14 11:29:26.718036] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching
xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:29:26.718121] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed
on
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=EFFFF,
>>> bad=10000)
>>> [2016-04-14 11:29:42.501760] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:
>>> name heal successful on FFFFF
>>> [2016-04-14 11:31:25.714812] E
[ec-inode-read.c:1612:ec_manager_stat]
>>>
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)
>>> [0x7f5ec9f942b1]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)
>>> [0x7f5ec9f94497]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)
>>> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,
>>> fop->locks[0].lock->loc.inode, &cbk->iatt[0].ia_size)
>>> [2016-04-14 11:31:25.722372] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722411] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722450] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722477] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722503] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722577] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722605] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722742] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722794] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722818] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722840] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722883] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722906] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722958] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722983] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.723037] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.723045] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.725044] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.741338] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.746602] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57:
remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.746629] W [MSGID: 122015]
>>> [ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode
unlocking
>>> failed (FSTAT) [Invalid argument]
>>> [2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze]
>>>
>>>
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)
>>> [0x7f5ec9fa009e]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)
>>> [0x7f5ec9f97fc0]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0x91)
>>> [0x7f5ec9f975d1] ) 0-: Assertion failed: lock->refs ==
lock->inserted
>>> [2016-04-14 11:31:25.746719] E [ec-common.c:1641:ec_lock_unfreeze]
>>>
>>>
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)
>>> [0x7f5ec9fa009e]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)
>>> [0x7f5ec9f97fc0]
>>>
>>>
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0xf2)
>>> [0x7f5ec9f97632] ) 0-: Assertion failed:
list_empty(&lock->waiting) &&
>>> list_empty(&lock->owners)
>>> [2016-04-14 11:40:54.139418] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-0: Mismatching
xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:40:54.139554] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-0: Operation failed
on
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=2FFFF,
>>> bad=D0000)
>>> [2016-04-14 11:40:54.143400] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-0: /workdir/raw_output2:
>>> name heal successful on FFFFF
>>> The message "W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check]
>>> 0-v0-disperse-0: Mismatching xdata in answers of
'LOOKUP'" repeated 2
>>> times between [2016-04-14 11:40:54.139418] and [2016-04-14
>>> 11:40:54.139547]
>>> [2016-04-14 11:42:31.471317] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-45: Mismatching
>>> xdata in answers of 'LOOKUP'
>>> The message "W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check]
>>> 0-v0-disperse-45: Mismatching xdata in answers of
'LOOKUP'" repeated 8
>>> times between [2016-04-14 11:42:31.471317] and [2016-04-14
>>> 11:42:31.471557]
>>>
>>> Gluster version is 3.7.10
>>> Any thoughts?
>>>
>>> Serkan
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>