thr3ads.net - Gluster users - [Gluster-users] stuck lock [Dec 2012]

If this information is useful, please help other people find it:
Share via:

Mario Kadastik

2012-Dec-13 13:10 UTC

[Gluster-users] stuck lock

Hi

I still don't know what caused it, wether the failure of one node in gluster
that lost one SATA controller and was rebooted or some user activity, but
gluster became quite unusable. Even basic gluster commands like gluster volume
heal home0 info etc didn't work either hanging or giving operation failed
results. I finally managed to stop the volume after numerous attempts and
restarted gluster on all nodes. However I don't seem to be able to do
anything useful still. Most commands fail and the log shows:

==> etc-glusterfs-glusterd.vol.log <=[2012-12-13 14:59:49.713103] I
[glusterd-volume-ops.c:492:glusterd_handle_cli_heal_volume] 0-management:
Received heal vol req for volume home0
[2012-12-13 14:59:49.713194] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd:
Unable to get lock for uuid: c3ce6b9c-6297-4e77-924c-b44e2c13e58f, lock held by:
c3ce6b9c-6297-4e77-924c-b44e2c13e58f
[2012-12-13 14:59:49.713234] E [glusterd-handler.c:458:glusterd_op_txn_begin]
0-management: Unable to acquire local lock, ret: -1

I've googled and seen people hit with this at times, but never resolutions.
Is there some way to clear this lock? It's been in effect for well over an
hour so one of the googled results that claimed there's a generic lock
timeout of 30 minutes seems not to be at work here.

Any help would be appreciated.

[root at se1 home0]# gluster volume info
 
Volume Name: home0
Type: Distributed-Replicate
Volume ID: 8e594854-16e1-445e-8434-1d597cef1749
Status: Started
Number of Bricks: 4 x 3 = 12
Transport-type: tcp
Bricks:
Brick1: 192.168.1.241:/d35
Brick2: 192.168.1.242:/d35
Brick3: 192.168.1.243:/d35
Brick4: 192.168.1.244:/d35
Brick5: 192.168.1.245:/d35
Brick6: 192.168.1.240:/d35
Brick7: 192.168.1.241:/d36
Brick8: 192.168.1.242:/d36
Brick9: 192.168.1.243:/d36
Brick10: 192.168.1.244:/d36
Brick11: 192.168.1.245:/d36
Brick12: 192.168.1.240:/d36
Options Reconfigured:
cluster.quorum-type: auto
cluster.lookup-unhashed: off
performance.client-io-threads: on
cluster.data-self-heal: on
performance.stat-prefetch

[root at se1 home0]# gluster volume status
Status of volume: home0
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 192.168.1.241:/d35				24009	Y	7137
Brick 192.168.1.242:/d35				24009	Y	6804
Brick 192.168.1.243:/d35				24009	Y	5763
Brick 192.168.1.244:/d35				24009	Y	10378
Brick 192.168.1.245:/d35				24009	Y	3770
Brick 192.168.1.240:/d35				24009	Y	21112
Brick 192.168.1.241:/d36				24010	Y	7143
Brick 192.168.1.242:/d36				24010	Y	6810
Brick 192.168.1.243:/d36				24010	Y	5771
Brick 192.168.1.244:/d36				24010	Y	10384
Brick 192.168.1.245:/d36				24010	Y	3781
Brick 192.168.1.240:/d36				24010	Y	21120
NFS Server on localhost					38467	Y	13552
Self-heal Daemon on localhost				N/A	Y	13792
NFS Server on 192.168.1.242				38467	Y	21254
Self-heal Daemon on 192.168.1.242			N/A	Y	21267
NFS Server on 192.168.1.243				38467	Y	8865
Self-heal Daemon on 192.168.1.243			N/A	Y	8871
NFS Server on 192.168.1.240				38467	Y	18806
Self-heal Daemon on 192.168.1.240			N/A	Y	19045
NFS Server on 192.168.1.244				38467	Y	536
Self-heal Daemon on 192.168.1.244			N/A	Y	745
NFS Server on 192.168.1.245				38467	Y	8689
Self-heal Daemon on 192.168.1.245			N/A	Y	8955
 
[root at se1 home0]# 


[root at se1 home0]# gluster volume heal home0 info

==> cli.log <=[2012-12-13 15:09:33.476616] W
[rpc-transport.c:174:rpc_transport_load] 0-rpc-transport: missing 'option
transport-type'. defaulting to "socket"

==> etc-glusterfs-glusterd.vol.log <=[2012-12-13 15:09:33.565022] I
[glusterd-volume-ops.c:492:glusterd_handle_cli_heal_volume] 0-management:
Received heal vol req for volume home0
[2012-12-13 15:09:33.565122] I [glusterd-utils.c:285:glusterd_lock] 0-glusterd:
Cluster lock held by c3ce6b9c-6297-4e77-924c-b44e2c13e58f
[2012-12-13 15:09:33.565136] I [glusterd-handler.c:463:glusterd_op_txn_begin]
0-management: Acquired local lock
[2012-12-13 15:09:33.565938] I
[glusterd-rpc-ops.c:548:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC
from uuid: 663ecbfb-4209-417e-a955-6c9f72751dbc
[2012-12-13 15:09:33.565999] I
[glusterd-rpc-ops.c:548:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC
from uuid: f1a89ed2-a2f5-49a9-9482-1c6984c37945
[2012-12-13 15:09:33.566024] I
[glusterd-rpc-ops.c:548:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC
from uuid: b1ce84be-de0b-4ae1-a1e8-758d828b8872
[2012-12-13 15:09:33.566047] I
[glusterd-rpc-ops.c:548:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC
from uuid: 0f61d484-0f93-4144-b166-2145f4ea4427
[2012-12-13 15:09:33.566069] I
[glusterd-rpc-ops.c:548:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC
from uuid: d9b48655-4b25-4ad2-be19-c5ec8768a789
[2012-12-13 15:09:33.566224] I
[glusterd-op-sm.c:2039:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req to
5 peers
[2012-12-13 15:09:33.566420] I [glusterd-rpc-ops.c:881:glusterd3_1_stage_op_cbk]
0-glusterd: Received ACC from uuid: b1ce84be-de0b-4ae1-a1e8-758d828b8872
[2012-12-13 15:09:33.566450] I [glusterd-rpc-ops.c:881:glusterd3_1_stage_op_cbk]
0-glusterd: Received ACC from uuid: d9b48655-4b25-4ad2-be19-c5ec8768a789
[2012-12-13 15:09:33.566499] I [glusterd-rpc-ops.c:881:glusterd3_1_stage_op_cbk]
0-glusterd: Received ACC from uuid: f1a89ed2-a2f5-49a9-9482-1c6984c37945
[2012-12-13 15:09:33.566524] I [glusterd-rpc-ops.c:881:glusterd3_1_stage_op_cbk]
0-glusterd: Received ACC from uuid: 0f61d484-0f93-4144-b166-2145f4ea4427
[2012-12-13 15:09:33.566667] I [glusterd-rpc-ops.c:881:glusterd3_1_stage_op_cbk]
0-glusterd: Received ACC from uuid: 663ecbfb-4209-417e-a955-6c9f72751dbc

<hangs here> 
ctrl+C

[root at se1 home0]# gluster volume heal home0
operation failed
[root at se1 home0]# 
==> cli.log <=[2012-12-13 15:10:00.686308] W
[rpc-transport.c:174:rpc_transport_load] 0-rpc-transport: missing 'option
transport-type'. defaulting to "socket"
[2012-12-13 15:10:00.842108] I [cli-rpc-ops.c:5928:gf_cli3_1_heal_volume_cbk]
0-cli: Received resp to heal volume
[2012-12-13 15:10:00.842187] I [input.c:46:cli_batch] 0-: Exiting with: -1

==> etc-glusterfs-glusterd.vol.log <=[2012-12-13 15:10:00.841789] I
[glusterd-volume-ops.c:492:glusterd_handle_cli_heal_volume] 0-management:
Received heal vol req for volume home0
[2012-12-13 15:10:00.841910] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd:
Unable to get lock for uuid: c3ce6b9c-6297-4e77-924c-b44e2c13e58f, lock held by:
c3ce6b9c-6297-4e77-924c-b44e2c13e58f
[2012-12-13 15:10:00.841926] E [glusterd-handler.c:458:glusterd_op_txn_begin]
0-management: Unable to acquire local lock, ret: -1

Mario Kadastik, PhD
Researcher

---
  "Physics is like sex, sure it may have practical reasons, but that's
not why we do it"
     -- Richard P. Feynman

Mario Kadastik

2012-Dec-13 15:27 UTC

head link

[Gluster-users] stuck lock

Hi all,

two updates firstly. All gluster nodes right now are Scientific Linux 5.7 (Linux
se1 2.6.18-308.16.1.el5 #1 SMP Wed Oct 3 00:53:20 EDT 2012 x86_64 x86_64 x86_64
GNU/Linux) with gluster version 3.3.1. The client nodes that mount the volume
over NFS are CentOS 6.3.

Also, finally after all else failed I unmounted (sometimes by force) the volume
from everywhere, stopped glusterd and glusterfsd and after restarting them the
issue had disappeared. However this is NOT a way I'd like to fix things as
it was very very disruptive.

Mario Kadastik, PhD
Researcher

---
  "Physics is like sex, sure it may have practical reasons, but that's
not why we do it"
     -- Richard P. Feynman

Gluster users - Dec 2012 - stuck lock

[Gluster-users] stuck lock

[Gluster-users] stuck lock