Junxiao Bi
2016-Jan-13 02:46 UTC
[Ocfs2-devel] ocfs2: A race between refmap setting and clearing
On 01/12/2016 03:16 PM, xuejiufei wrote:> Hi, Junxiao > > On 2016/1/12 12:03, Junxiao Bi wrote: >> Hi Jiufei, >> >> On 01/11/2016 10:46 AM, xuejiufei wrote: >>> Hi all, >>> We have found a race between refmap setting and clearing which >>> will cause the lock resource on master is freed before other nodes >>> purge it. >>> >>> Node 1 Node 2(master) >>> dlm_do_master_request >>> dlm_master_request_handler >>> -> dlm_lockres_set_refmap_bit >>> call dlm_purge_lockres after unlock >>> dlm_deref_handler >>> -> find lock resource is in >>> DLM_LOCK_RES_SETREF_INPROG state, >>> so dispatch a deref work >>> dlm_purge_lockres succeed. >>> >>> dlm_do_master_request >>> dlm_master_request_handler >>> -> dlm_lockres_set_refmap_bit >>> >>> deref work trigger, call >>> dlm_lockres_clear_refmap_bit >>> to clear Node 1 from refmap >>> >>> Now Node 2 can purge the lock resource but the owner of lock resource >>> on Node 1 is still Node 2 which may trigger BUG if the lock resource >>> is $RECOVERY or other problems. >>> >>> We have discussed 2 solutions: >>> 1)The master return error to Node 1 if the DLM_LOCK_RES_SETREF_INPROG >>> is set. Node 1 will not retry and master send another message to Node 1 >>> after clearing the refmap. Node 1 can purge the lock resource after the >>> refmap on master is cleared. >>> 2) The master return error to Node 1 if the DLM_LOCK_RES_SETREF_INPROG >>> is set, and Node 1 will retry to deref the lockres. >>> >>> Does anybody has better ideas? >>> >> dlm_purge_lockres() will wait to drop ref until >> DLM_LOCK_RES_SETREF_INPROG cleared. So if set this flag when find the >> master during doing master request. And then this flag was cleared when >> receiving assert master message, can this fix the issue? >> > I don't think this can fix. Before doing master request, the lock resource is > already purged. The master should clear the refmap before client purge it.inflight_locks is increased in dlm_get_lock_resource() which will stop lockres purged? Set DLM_LOCK_RES_SETREF_INPROG when found lockres owner during master request, then this will stop lockres purged after unlock? Thanks, Junxiao.> > Thanks, > Jiufei > >> Thanks, >> Junxiao. >>> Thanks, >>> --Jiufei >>> >> >> >> . >> >
xuejiufei
2016-Jan-13 06:21 UTC
[Ocfs2-devel] ocfs2: A race between refmap setting and clearing
Hi Junxiao,
I have not describe the issue clearly.
Node 1 Node 2(master)
dlmlock
dlm_do_master_request
dlm_master_request_handler
-> dlm_lockres_set_refmap_bit
dlmlock succeed
dlmunlock succeed
dlm_purge_lockres
dlm_deref_handler
-> find lock resource is in
DLM_LOCK_RES_SETREF_INPROG state,
so dispatch a deref work
dlm_purge_lockres succeed.
call dlmlock again
dlm_do_master_request
dlm_master_request_handler
-> dlm_lockres_set_refmap_bit
deref work trigger, call
dlm_lockres_clear_refmap_bit
to clear Node 1 from refmap
dlm_purge_lockres succeed
dlm_send_remote_lock_request
return DLM_IVLOCKID because
the lockres is not exist
BUG if the lockres is $RECOVERY
On 2016/1/13 10:46, Junxiao Bi wrote:> On 01/12/2016 03:16 PM, xuejiufei wrote:
>> Hi, Junxiao
>>
>> On 2016/1/12 12:03, Junxiao Bi wrote:
>>> Hi Jiufei,
>>>
>>> On 01/11/2016 10:46 AM, xuejiufei wrote:
>>>> Hi all,
>>>> We have found a race between refmap setting and clearing which
>>>> will cause the lock resource on master is freed before other
nodes
>>>> purge it.
>>>>
>>>> Node 1 Node 2(master)
>>>> dlm_do_master_request
>>>> dlm_master_request_handler
>>>> ->
dlm_lockres_set_refmap_bit
>>>> call dlm_purge_lockres after unlock
>>>> dlm_deref_handler
>>>> -> find lock resource is in
>>>> DLM_LOCK_RES_SETREF_INPROG
state,
>>>> so dispatch a deref work
>>>> dlm_purge_lockres succeed.
>>>>
>>>> dlm_do_master_request
>>>> dlm_master_request_handler
>>>> ->
dlm_lockres_set_refmap_bit
>>>>
>>>> deref work trigger, call
>>>> dlm_lockres_clear_refmap_bit
>>>> to clear Node 1 from refmap
>>>>
>>>> Now Node 2 can purge the lock resource but the owner of lock
resource
>>>> on Node 1 is still Node 2 which may trigger BUG if the lock
resource
>>>> is $RECOVERY or other problems.
>>>>
>>>> We have discussed 2 solutions:
>>>> 1)The master return error to Node 1 if the
DLM_LOCK_RES_SETREF_INPROG
>>>> is set. Node 1 will not retry and master send another message
to Node 1
>>>> after clearing the refmap. Node 1 can purge the lock resource
after the
>>>> refmap on master is cleared.
>>>> 2) The master return error to Node 1 if the
DLM_LOCK_RES_SETREF_INPROG
>>>> is set, and Node 1 will retry to deref the lockres.
>>>>
>>>> Does anybody has better ideas?
>>>>
>>> dlm_purge_lockres() will wait to drop ref until
>>> DLM_LOCK_RES_SETREF_INPROG cleared. So if set this flag when find
the
>>> master during doing master request. And then this flag was cleared
when
>>> receiving assert master message, can this fix the issue?
>>>
>> I don't think this can fix. Before doing master request, the lock
resource is
>> already purged. The master should clear the refmap before client purge
it.
> inflight_locks is increased in dlm_get_lock_resource() which will stop
> lockres purged? Set DLM_LOCK_RES_SETREF_INPROG when found lockres owner
> during master request, then this will stop lockres purged after unlock?
>
> Thanks,
> Junxiao.
>
>>
>> Thanks,
>> Jiufei
>>
>>> Thanks,
>>> Junxiao.
>>>> Thanks,
>>>> --Jiufei
>>>>
>>>
>>>
>>> .
>>>
>>
>
>
> .
>