Gluster users - Jun 2013 - held cluster lock blocking volume operations

So, we've got a volume that is mostly functioning fine (its up accessible,
etc etc). However, volume operations fail/don't return on it.


what i mean is

gluster peer status//probe/etc : works
gluster volume info : works
gluster volume status/remove-brick/etc : sit for a long time and return
nothing.

The only things coming up in logs are:

[2013-06-04 10:21:36.398072] I [glusterd-utils.c:285:glusterd_lock]
0-glusterd: Cluster lock held by 757297b4-5648-4e31-88f4-00fc167a43e4
[2013-06-04 10:21:36.398123] I
[glusterd-handler.c:463:glusterd_op_txn_begin] 0-management: Acquired local
lock
[2013-06-04 10:21:36.398424] I
[glusterd-handler.c:502:glusterd_handle_cluster_lock] 0-glusterd: Received
LOCK from uuid: 757297b4-5648-4e31-88f4-00fc167a43e4
[2013-06-04 10:21:36.398448] E [glusterd-utils.c:277:glusterd_lock]
0-glusterd: Unable to get lock for uuid:
757297b4-5648-4e31-88f4-00fc167a43e4, lock held by:
757297b4-5648-4e31-88f4-00fc167a43e4
[2013-06-04 10:21:36.398483] I
[glusterd-handler.c:1322:glusterd_op_lock_send_resp] 0-glusterd: Responded,
ret: 0
[2013-06-04 10:21:36.398498] E [glusterd-op-sm.c:4624:glusterd_op_sm]
0-glusterd: handler returned: -1

If you notice, the UUID holding the lock, and the uuid requesting the lock,
are the same. So it seems like a lock was "forgotten" about?

any thoughts on clearing this?
--
Matthew Nicholson
Research Computing Specialist
Harvard FAS Research Computing
matthew_nicholson at harvard.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130604/301b1919/attachment.html>

On 06/04/2013 07:57 PM, Matthew Nicholson wrote:
> So, we've got a volume that is mostly functioning fine (its up
> accessible, etc etc). However, volume operations fail/don't return on
it.
>
>
> what i mean is
>
> gluster peer status//probe/etc : works
> gluster volume info : works
> gluster volume status/remove-brick/etc : sit for a long time and return
> nothing.
>
> The only things coming up in logs are:
>
> [2013-06-04 10:21:36.398072] I [glusterd-utils.c:285:glusterd_lock]
> 0-glusterd: Cluster lock held by 757297b4-5648-4e31-88f4-00fc167a43e4
> [2013-06-04 10:21:36.398123] I
> [glusterd-handler.c:463:glusterd_op_txn_begin] 0-management: Acquired
> local lock
> [2013-06-04 10:21:36.398424] I
> [glusterd-handler.c:502:glusterd_handle_cluster_lock] 0-glusterd:
> Received LOCK from uuid: 757297b4-5648-4e31-88f4-00fc167a43e4
> [2013-06-04 10:21:36.398448] E [glusterd-utils.c:277:glusterd_lock]
> 0-glusterd: Unable to get lock for uuid:
> 757297b4-5648-4e31-88f4-00fc167a43e4, lock held by:
> 757297b4-5648-4e31-88f4-00fc167a43e4
> [2013-06-04 10:21:36.398483] I
> [glusterd-handler.c:1322:glusterd_op_lock_send_resp] 0-glusterd:
> Responded, ret: 0
> [2013-06-04 10:21:36.398498] E [glusterd-op-sm.c:4624:glusterd_op_sm]
> 0-glusterd: handler returned: -1
>
> If you notice, the UUID holding the lock, and the uuid requesting the
> lock, are the same. So it seems like a lock was "forgotten"
about?
>
> any thoughts on clearing this?
Does gluster peer status list the same UUID more than once?

If not, restarting the glusterd which is the lock owner should address it.

-Vijay